Google DeepMind and the comparison site Artificial Analysis released the MKA (Multimodal Knowledge Agent) benchmark on 10 November. Experts from ten fields, including architects, electrical engineers, radiologists and lawyers, wrote its 200 tasks.

How the benchmark works

For each task, the agent receives 2–45 files, such as floor plans, wiring diagrams, X-rays or maps. From these it must produce finished work, most often a spreadsheet, presentation, text document or PDF report.

All models ran in Artificial Analysis’s open Stirrup harness, the program that controls the agent, gives the model its tools and runs its steps. For up to 250 turns, the agent can execute code, search the web and view images. Google’s report says this lets the agent fetch some information from the web instead of reading it from the images.

An agent running on OpenAI’s GPT-6 Luna model grades the work. It opens the files, views pages as images and uses an expert’s yes/no checklist of about 70 items to check things such as spreadsheet formulas. In a 50-task comparison, the grader agreed with the experts on 96.2 per cent of criteria, the report said. The comparison covered only the most heavily weighted criteria, so the figure may not hold for formatting checks.

The headline metric, Climb-WSR, is the weighted share of hard criteria met. A criterion counts as hard if at least one of four previous-generation models failed it in two of three runs. Two of these four were older Gemini models. On the remaining criteria, all four models tested in the report scored above 95 per cent.

Results and prices

Google’s technical report compared four models. The table shows their scores and costs per task.

Source: Google DeepMind technical report. List prices at publication, including cache discounts.

The Climb-WSR differences between Fable 5.2, Carbon and Opus 5.5 are not statistically significant, so their order is uncertain. Fable 5.2 and Carbon scored significantly higher than GPT-6 Astra. Each model ran every task once and each result was graded once, so the confidence intervals do not capture run-to-run variation in either the agent or the grader.

The sources give different prices. Artificial Analysis puts Fable 5.2 at $31.97 and Carbon at $6.29 per task. An image in Google DeepMind’s post on X gives Astra’s price as $6.15. No model leads in all ten fields, Artificial Analysis said.

Google both designed the benchmark and makes one of the tested models, Gemini 4 Carbon. The report says the grader was chosen from outside Google so that it would not favour Gemini.

Viewing images affected the results

Google also ran Carbon without the image-viewing tool. On MKA, its Climb-WSR fell by 5.9 percentage points and it used 1.28 times as many turns. On the GDPval benchmark, its score did not change. The agent tried to replace vision with code, for example by printing images as ASCII art. This partly helped with clear drawings but not with photographs or medical images. Google ran this experiment only with Carbon.

The leaderboard is on the Artificial Analysis website. Ten public example tasks are on Hugging Face. The 200 evaluation tasks themselves are kept secret so that they do not end up in training data. Gemini 4 Carbon is available through the Gemini API in Google AI Studio.