Artificial Analysis: Perplexity’s Decider v1.3 scores 70 on decision index, Flash scores 64
Perplexity’s Decider v1.3 scored 70 on the Artificial Analysis Decision Index, and the smaller Decider Flash v1.3 scored 64 at half the cost. According to the chart in the post, the three frontier models at the top scored 73 but cost more than Decider v1.3.
Independent tester Artificial Analysis posted results on X on 14 November for Perplexity’s new decision models, Decider v1.3 and Decider Flash v1.3. Perplexity released the models the previous day, the CASRAI website reported.
A decision model chooses from fixed options
A decision model does not write a free-form answer the way a large language model does. Artificial Analysis describes it as choosing from fixed options and returning a probability for each option in a single call. In an example on the company’s page, the model decides which team a customer message should go to and whether the matter is urgent. The company said this makes the model cheap and fast, and makes its calibration measurable, meaning whether its confidence matches its accuracy.
Decision Index v2.1 combines six benchmarks. AA-Paperclip carries the most weight (30 per cent): in it, the model makes decisions and extracts information from CVs, receipts, invoices and research papers. The other benchmarks measure judgements about workplace documents, tool calls, knowledge and hallucinations, reasoning, and whether a decision stays the same when the question is rephrased.
Index scores are not hit rates. Five of the benchmarks use the Decision Score as their metric: a correct answer earns 50–100 points and a wrong one 0–50. The more confident the model, the further the score moves from 50 in either direction.
Results and costs
Decider v1.3 scored 70 on the index and Flash v1.3 scored 64. Decider v1.3 held sixth place, as its predecessor v1.2 did, Artificial Analysis said. It scored 69 on AA-Paperclip (+2) and 89 on the EvalSafe-O*NET-AA benchmark (+1).
Flash v1.3 scored four points more than v1.2. Artificial Analysis attributed the gain to a 12-point improvement on both AA-Symmetry, which measures whether a decision stays the same, and the When2Call tool-call benchmark.
At current prices, 1 million index tasks cost $68 with Decider v1.3 and $34 with Flash v1.3. Decision models are billed for input tokens only. Artificial Analysis said both models sit on the Pareto frontier: no cheaper model scored higher on the index. The chart in the post shows three models at the top with 73 points, all of them more expensive than Decider v1.3.
The three-point gap to the top is a difference in a weighted average, not a gap of three percentage points in accuracy.
Availability
Artificial Analysis lists the models at 27 billion and 4 billion parameters, with text and image input and an Apache 2.0 licence. They are available through Perplexity’s own API at $0.02 and $0.01 per 1 million input tokens. Perplexity also released the model weights on Hugging Face, CASRAI reported. The sources disagree on the size of the context window: Artificial Analysis gives 256,000 tokens, CASRAI 30,720 tokens.
Read more: Models