AI Aulis

AI NEWS IN ENGLISH · 21 November 2026

The most important AI news every morning. AI writes and checks the stories against the original sources – check important facts in the sources.

← Issue 21 November 2026Paper

Stripe test: 29 per cent of coding agents’ integrations met every required criterion

AI-generated contentWritten by AI

github.com, stripe.dev, i10x.ai, automater.ai, dev.to · 5 sources · Sources published: 19–20 November 2026 · Some source publication dates are unknown

Stripe tested 14 model and harness combinations on 69 payment integration tasks. On average the agents met 78.1 per cent of the evaluation criteria, but only 29.4 per cent of runs met every required criterion.

On 19 November Stripe published a benchmark on its developer blog of Stripe payment integrations built by coding agents. The test used three harnesses, the programs that control an agent by giving the model its tools and running its steps: Claude Code, Codex and OpenCode. The tasks covered one-time payments, subscriptions and Connect platform payments. Each task was run twice without the Stripe agent plugin and twice with it.

The agent does not get a ready-made list of requirements. Another AI model plays a developer with a secret list of needs and asks about them one at a time. Finally, GPT-6.1 Sol assesses the conversation and the files the agent wrote, one criterion at a time. Each criterion is either required (MUST) or recommended (SHOULD).

The gap between the figures comes from how they are calculated. The average shows what share of all criteria a run met. If a task has seven required criteria and one recommended criterion and a run meets six, it scores 75 per cent. Under a strict pass, a single missing required criterion fails the run. Stripe said many models produce a nearly correct implementation but fail on the final details. For example, when an order was held on suspicion of fraud, Opus 5.5 checked only that the payment had succeeded. The order could have been shipped even if the payment had already been refunded or disputed.

The table shows some of Stripe’s results on both metrics.

Model (harness) Average: without → plugin Strict pass: without → plugin
GPT-6 Astra (Codex) 85.5% → 90.0% 41.9% → 52.9%
GPT-6.1 Sol (Codex) 85.6% → 89.6% 39.0% → 50.7%
Opus 5.5 (Claude Code) 83.5% → 86.2% 35.3% → 36.0%
Fable 5.2 (Claude Code) 81.1% → 85.2% 28.4% → 43.2%
Sonnet 5.5 (Claude Code) 78.0% → 82.6% 29.4% → 33.1%
Gemini 3.8 Flash (OpenCode) 64.8% → 72.7% 11.2% → 21.6%

Source: Stripe. Figures are based on 69 tasks and two runs in each setup, with GPT-6.1 Sol as the grader.

Stripe said model comparisons should be read alongside the results for each task category, because the agents that did best on one-time payments and subscriptions were not the best on Connect tasks.

The test also does not run the agent’s code. Stripe acknowledges that a neatly structured integration that does not work can still get full marks. The company’s conclusion that frontier models can build working integrations with harnesses therefore rests on the grader model’s interpretation, not on executed code. Strict pass is also sensitive to grading errors, the company said. When Stripe tried Opus as the grader, Opus gave every agent consistently more lenient scores, but its scores moved in step with Sol’s.

The plugin gives the agent skills and MCP tools written by Stripe, such as documentation search. Stripe said the plugin raised the average for all 14 combinations by roughly 3–8 percentage points. The share of outdated parameters in API calls fell from 7 per cent to 1.3 per cent. This is the company’s own measurement of its own product. Figures from Stripe’s earlier test suite published on GitHub, such as 92 per cent for Opus 4.5, cannot be compared with these because that suite used different tasks and scoring.

Stripe released the task dataset and the Forge evaluation software as open source on GitHub (stripe/forge). A Dev.to post said that running the example test requires Docker, an Anthropic API key for Claude Code and an OpenAI key for the model playing the developer and for the grader, which use an OpenAI model by default. In Stripe’s table, a single run cost between $0.05 and $6.13, depending on the model.

Report an error in this story