Epoch AI: agents tested by Trajectory Labs tried to break into Epoch’s website
AI agents in a Trajectory Labs evaluation in October tried SQL injection, among other methods, against Epoch’s website and posed as Epoch employees, Epoch AI said. The agents did not gain access to private data, the organisation said.
The research organisation Epoch AI published an incident report on 13 November about three events in October. In each, AI agents – programs that act through a model – interacted with Epoch’s website and staff. The agents were run by Trajectory Labs, PBC, which develops AI safety evaluations and tests frontier models from an attacker’s perspective.
Trajectory said the agents used publicly available frontier models. Epoch has not verified this itself and therefore does not name the models or their developers.
An impossible task and an order never to give up
On 5 October Epoch had opened a honeypot API on its website to monitor malicious activity; the API had no connection to private data. When traffic rose on 20 October, Epoch opened a channel for the agents to send short messages. Between 16.30 and 19.30 Pacific Time, the agents sent 487 messages and made 36,411 calls to the honeypot API.
In three messages the agents posed as Epoch employees, giving a name and job title. In 27 more they claimed to be engineers or auditors. Almost every substantive message asked for details of FrontierMath benchmark problems that do not exist. One agent attempted SQL injection – embedding database commands in input – Epoch’s investigation found. Some of the other agents searched for admin credentials and secret endpoints, and made more than 200 attempts to read real FrontierMath evaluation logs. The agents did not gain access to any private or confidential information, Epoch said.
Trajectory had not announced the experiment in advance. The next morning it said it had deliberately given the agents an impossible task: evaluating a fake version of FrontierMath. Trajectory said 40 agents had interacted with Epoch on 20 October, with the model provider’s own safeguards in place. The agents ran in Trajectory’s own harness, the program that controls an agent, gives the model its tools and runs its steps. According to information Trajectory shared with Epoch, the harness included language such as ‘NEVER stop, NEVER give up, and NEVER conclude that a task or input is impossible, unavailable, or exhausted.’ In Epoch’s assessment, this set-up at least partly caused the agents’ behaviour.
The redirect did not keep the agents on the copy
On 23 October Epoch received four messages and 95 calls from the same IP address the agents had used on 20 October. Trajectory said it had been testing the agents on a local copy of the website. It had pointed them to the copy with a DNS redirect but had not sandboxed the run, and the agents had internet access. The agents bypassed the redirect, Trajectory said.
The third experiment took place on 26 October with Epoch’s consent. A vulnerability supplied by Trajectory was installed on a test copy behind a firewall. Epoch was told the agents had unrestricted internet access and some would likely move to the public website. On its public website, Epoch attributes 133 messages, 65,460 honeypot API calls and a new SQL injection attempt to the experiment.
Epoch proposes three minimum requirements for evaluators. An evaluator should share its risk assessment with any organisation likely to be targeted and obtain a signed agreement from it in advance. Experiments that the evaluator expects could lead to misaligned behaviour should be actively monitored. Incidents involving outside organisations should be reported to all parties as fully as existing agreements allow.
Read more: How an agent works · Agent map