In a talk published by the AI Engineer channel on 10 November, Zenity engineer Roey Ben Chaim explains why nobody can judge the safety of an AI agent’s skill by reading its instructions. A skill is a package of instructions, often with scripts or code, that the agent loads when needed.

In Ben Chaim’s example, a skill read cloud and Kubernetes configuration, contacted an authentication service and probed the cloud server’s metadata service. Many infostealers behave the same way, but this was an official Azure diagnostic skill.

The malicious part can load from elsewhere

The SKILL.md file is often just the entry point. It can fetch the rest of its instructions from the web. An install script can download packages, and the skill’s other code can download more. A scanner that reads only the file sees none of this. A scanner that uses a large language model can also be misled. Ben Chaim cited Trail of Bits research in which a skill changed a project’s package registry; a small comment saying the change was approved was enough to fool the scanner.

Zenity therefore began running skills with an agent in an isolated container, much like a detonation chamber in malware research. The chamber sets up the services the skill expects. A separate orchestrator writes tasks that exercise the skill’s different execution paths. Because agents often refused to unpack or run code, the orchestrator persuades them.

Besides the agent’s own log, the chamber records process launches, file events and network traffic. A post-install script in an npm package, for example, can start a process that reads files and contacts a server without appearing in the agent’s log.

A hundred malicious skills

Ben Chaim said the chamber found 100 malicious skills among thousands, and they were removed from the skills.sh directory. The video description says the examples do not show that any single scanner classifies every skill correctly.

One skill installed an npm package whose name resembled Polymarket’s client library. The package pointed to an unofficial registry, and its code was an infostealer. Other techniques included an instruction added to the agent’s instruction file to reinstall the skill; an unofficial skill creator that silently removed Anthropic’s own and installed itself in its place; and instructions fetched from the web. Legitimate skills also fetch instructions from the web, but the content can change later. Hooks, actions tied to events, can also trigger a skill on every user message.

In the largest case, the attacker imitated Paperclip, an open-source project for orchestrating agents, with a lookalike GitHub organisation and domain. A logging function added to the cloned code fetched further code from the attacker’s server, ran it with Node and deleted it after 60 seconds. The downloaded code collected SSH, AWS and Azure credentials and sent them to the attacker.

PyPI packages that used the same command-and-control server were removed in under two hours. Skills using the same server had been downloaded 1.7 million times in total, Ben Chaim said. Zenity reported the case to Vercel and Microsoft, which took everything down in under 12 hours. These details are based on the talk’s automatic captions, which may contain recognition errors.

Ben Chaim advises treating skills like software dependencies: checking what they download, granting only the permissions they need and monitoring what actually runs. Users can test their own skills in Zenity’s chamber, Ben Chaim said.