Artificial Analysis: Liquid AI’s d1.2-ultra tops decision model index
Liquid AI’s new d1.2-ultra leads evaluation firm Artificial Analysis’s decision model comparison with 75 points, and the cheaper d1.2 scored 69. One million index tasks cost $133 with d1.2-ultra and $63 with d1.2.
Evaluation firm Artificial Analysis (AA) published the results on its X account on 16 November, after Liquid AI released two new decision models. The models, d1.2-ultra and d1.2, update the company’s earlier flagship models, d1-ultra and d1.1. AA says it runs all evaluations itself.
A decision model picks an option and states its confidence
Unlike a language model, a decision model does not write free text. It picks an answer from a fixed set of options and assigns a probability to each. The model handles all questions in one call. In AA’s example, the model chooses which team should handle a customer message and rates how urgent the message is and how annoyed the sender is. AA says such models are cheap and fast, and their stated confidence can be compared with their accuracy.
Decision Index v2.1 combines six evaluations. The most heavily weighted, at 30 per cent, is AA’s own private AA-Paperclip, in which a model makes decisions and extracts information from 1,500 documents: CVs, receipts, invoices and research papers. The other evaluations measure judgement on workplace documents, tool calls, knowledge and hallucinations, and reasoning. One evaluation measures whether a decision stays the same when the order or wording of the options changes.
The scores do not show the share of correct answers. In five of the evaluations, a correct answer earns 50–100 points and a wrong one 0–50 points: the model’s confidence raises the score for a correct answer and lowers it for a wrong one. A model that is always confident and right scores 100.
Results and prices
d1.2-ultra scored 75 on the index, four points more than d1-ultra. d1.2 scored 69, three more than d1.1. AA ranks d1.2 sixth, but the chart in its post shows six models ahead of it. On AA-Paperclip, d1.2-ultra scored 76 and was three points ahead of the second-best model, Drex Max v1.1, AA said.
The top ranking covers the 18 decision models shown by default on AA’s site. Users can add language models to the comparison with a separate option, but AA’s post does not compare d1.2-ultra with them.
One million index tasks cost $133 with d1.2-ultra and $63 with d1.2. The price is the average per-task cost across the evaluations at current token prices, with the evaluations weighted as in the index. AA says both models sit on the capability vs. cost Pareto frontier. Based on the chart in its post, this means no cheaper model scores higher.
Price and availability
The models are available through Liquid AI’s own API. One million input tokens cost $0.042 with d1.2-ultra and $0.02 with d1.2. Decision models are billed only for input tokens, AA says. The context window is 64,000 tokens, and the models accept text and images.
Read more: Models