Artificial Analysis, which compares AI models, released the second version of its Decision Index on X and its website on 8 November. The first version appeared on 13 October.

The index measures decision models. Instead of writing a free-form answer, these models choose from predefined options and assign a probability to each. In an example on the company’s page, a model routes a customer’s message to the right team and, in a single call, assesses both the urgency of the message and the writer’s emotional state. Such models are cheap and fast, and their calibration can be measured, the company said.

Scores also depend on confidence

The new headline metric, Decision Score, rewards correct answers and calibration. Calibration shows whether a model’s stated confidence matches the share of answers it gets right. A correct answer earns 50–100 points and a wrong one 0–50 points. The more confident the model, the further the score moves from 50 in either direction.

The index is a weighted average of six benchmarks. The largest weight, 30 per cent, goes to the new AA-Paperclip benchmark, whose dataset is private. It contains 1,500 documents, such as CVs, receipts, invoices and AI research papers. The model makes 15 decisions or fact extractions from each document. A document counts as correct only if all 15 are right.

The new AA-Symmetry benchmark changes the order or names of the options in 120 questions, or rephrases a yes/no question in the negative. It checks whether the model then changes its answer. The other benchmarks measure knowledge, hallucination, reasoning, tool calls and judgements about workplace documents. The share of private datasets rose from 25 per cent to 45 per cent. The company said the change makes it harder to tune models purely for the benchmark.

Tie at the top, several-fold differences in cost

In the company’s measurement, Nace AI’s Drex Max v1.1 and Fastino Labs’ GLiDE v1.2 each scored 73, and Base Labs’ Mammoth 27B scored 72. Artificial Analysis describes Drex Max’s lead as narrow.

The table shows index scores and cost per 1 million tasks. Models with the same score can differ several-fold in cost, and the cost does not depend on token price alone.

Measured by Artificial Analysis; costs at current token prices. *Scores read from the chart in the publication.

Most decision models charge for a document’s input once for all of its questions, the company said. Drex Max v1.1 and Decider v1.2, for example, work this way. GLiDE v1.2 and Decider Flash v1.2, by contrast, charge for the input separately for each question. That is why GLiDE costs nearly eight times as much as Drex Max, even though its list prices are only a third higher.

Among language models, Claude Haiku 5.5 (Max) and GPT-6 Luna (Max) scored higher, at 77 and 76. GPT-6 Luna (Max), however, costs 15 times as much as Drex Max and takes 11.34 seconds per task, whereas Drex Max needs 275 milliseconds. Language models are billed for the tokens they generate as well as for the input; decision models only for the input. OpenAI’s separate decision model, GPT-6 Luna (Decisions), scored 62, according to the chart.