Mehmet Murat Huyut of Kuveyt Türk bank and İstanbul Bilgi University published a non-peer-reviewed preprint on arXiv on 4 November. Huyut kept the model fixed and changed the harness: the program that steers the agent, gives the model its tools and runs its steps.

An experiment in a simulated bank

Four open-weight models handled requests in a simulated bank. Its rulebook is modelled on EU rules on consumer credit, payments, investor protection, data protection and anti-money laundering, and in places is stricter. A violation is therefore an action that breaks the rulebook, not necessarily the law. All 156 cases were run on five harnesses. Violations and over-refusals were counted from the log of executed actions and the messages sent, without using a language model.

In the pipeline harness, the bank’s system ran the checks in advance and the model made a single decision. In the other harnesses the model called tools step by step: in the basic loop the rulebook was in the prompt, in the retrieval version the model could only get it by searching, and the memory version included a long customer history. In the reviewer version, the same model approved or blocked every state-changing action and customer message.

The models were mid-sized, with reasoning switched off. Huyut said stronger models, or models using reasoning, may behave differently.

Same model, different ranking

With results pooled across all models, 87.3–92.1 per cent of cases were handled correctly, depending on the harness. For individual models, the gap between the best and worst harness was larger, as the table shows.

Share of correctly handled cases, 156 cases each. Source: Huyut’s arXiv preprint.

A test of whether the harness effect differed between models was not statistically significant (p = 0.07). Huyut regards the model-specific differences as a consistent observation, not a demonstrated effect.

The harness also changed how the models ranked: qwen3.8-flash-next was best with the reviewer and worst in the retrieval version. The confidence intervals partly overlap, so Huyut treats this as a warning about single-harness evaluations. For deepseek-v4.1-flash, no harness effect was observed.

The reviewer helped, but counting days failed

Compared with the basic loop, the reviewer cut violations from 9.6 per cent to 6.4 per cent. Over-refusals did not rise, although Huyut had predicted an increase. The result applies to these models and this rulebook, Huyut said.

In every harness, errors were concentrated in rules that require counting days: the 14-day withdrawal period for credit and the 13-month limit for refunds of unauthorised payments. Between 67 and 71 per cent of withdrawal-period cases were handled correctly, and the reviewer did not improve this. Huyut said the reviewer likely made the same calculation error.

In an exploratory additional analysis, two language-model judges read the conversations and executed actions, and Huyut checked 100 cases blind. Both judges found that in 26 per cent of violations the agent told the customer it had not taken an action it had just carried out. Judged on the conversation alone, such a case looked like a justified refusal.

Huyut considers the share approximate, because the scripted follow-up question assumed a refusal and may have led the agent to deny the action. The model and harness must be validated as a pair, and compliance checked from the action log rather than the conversation, Huyut said.

The cases, harness code, run logs and analysis scripts are on GitHub.