Visa’s own benchmark: AI models failed 42.8 per cent of shopping task runs
Written by AI
Visa tested 22 AI models in 15 shopping scenarios. Each model had to uncover the customer’s needs through conversation and make the right purchase decision. The models passed 57.2 per cent of runs, and in more than half of the failed runs they still bought something.
Payment company Visa published its own benchmark on 5 November. It measures how AI models handle shopping on a customer’s behalf. The first round of the Visa Agentic Commerce Benchmark tested models from 12 labs. The models shopped for items such as a television, a laptop or a personal care device.
How the test works
The customer does not reveal every need at once. The model has to ask about things such as budget, image format support and gaming requirements. If no product fits, the model must not buy anything. Another agent played the customer, Fortune reported.
A run passes only if the model discovers every requirement defined for the task and makes the correct purchase decision. That way, Visa said, a lucky guess is not enough. The company says its benchmark is the first to require both. Each model attempted each task three times.
Results and their limits
The table shows pass rates by model group from Visa’s press release.
| Group | Runs passed |
|---|---|
| All models | 57.2% |
| Best models | 82.2% |
| Weakest model | 28.9% |
| Five that found the most hidden requirements | 70.2% |
| Five that found the fewest | 44.4% |
| Five most expensive (average cost per run) | 67.1% |
| Five cheapest | 50.2% |
Source: Visa press release, 5 November 2026; benchmark run by Visa itself.
The differences between groups show correlation, not causation. Visa itself notes that an expensive model does not guarantee success.
Results also varied: all three attempts succeeded in only 34.2 per cent of model–task combinations. Of the failed runs, 54 per cent still ended in a purchase. The model either recommended an unsuitable product or had not uncovered all the requirements.
So far the benchmark covers only needs discovery and the purchase decision. Visa plans to test checkout, payment and ordering later. The company said the results do not mean agents are not ready for commerce. Consistency and reliability, however, remain problems, it said.
Model names come from Fortune
Visa’s press release does not name the models. Fortune reporter Jeff John Roberts, citing Visa’s results site, wrote that agents based on Google’s Gemini and a Meta model shared first place with 82 per cent. A Claude agent came third with 76 per cent and an OpenAI agent sixth with 64 per cent. The sources name the versions differently: Fortune refers to Gemini’s Aragon 4 version, while ForkLog, summarising Fortune, names Gemini 4 Argon and Meta’s Muse Wonder.
Rubail Birwadker, Visa’s Global Head of Growth, told Fortune: ‘We’re not there yet from a trust or technology perspective. But we’ll soon get into a world where consumers can trust and delegate.’
Visa has shared the benchmark with leading model developers for feedback. On the Visa Payments Frontier site, anyone can try the benchmark, watch recordings of the models’ attempts and read the company’s research report.
Read more: Models