Top AI tools including Claude, Codex, and Hermes installed suspicious code inside corporate networks | usagoldmines.com

• Researchers found unclaimed llms.txt references on 120 domains, exploitable by cybercriminals
• AI agents could install malware if they execute hallucinated or outdated documentation commands
• Fixes: clean documentation and restrict AI agents from treating docs as executable instructions

Cybercriminals are able to now abuse hallucinated, outdated, and outright incorrect website documentation to deliver malware to unsuspecting victims through AI agents, new research has claimed.

An increasing number of websites now contain two documents: llms.txt, and llms-full.txt. These are conventions that allow AI agents to properly read the contents of the websites. If an AI agent is looking to install software or add code to a project, they can search through these documents across the web until they find a fitting solution.

Researchers have analyzed 6,214 live domains belonging to defense contractors, Fortune 500 organizations, as well as big tech. On these domains they found 8,265 of these .txt files and among them 120 (all on a different site) pointing to one or more code packages and domain names that weren’t registered at all.

Claiming packages and domains

There can be a myriad of reasons why they’re not registered. It can be due to human error, renamed or abandoned packages, copy/paste errors, or hallucinated documentation.

Now, for the purpose of the experiment, the researchers registered some of these unclaimed names and hosted packages that would phone home when installed. It took less than an hour for a Fortune 500 company to start pinging, and the numbers soon grew to “a few dozen more”.

This means that if the researchers can do it, so can cybercriminals. In theory, a cybercriminal could find these unclaimed packages and register malware. If an AI agent has permission to execute shell/package-manager commands and stumbles upon this documentation, it can end up infecting the device.

Claude, OpenAI’s Codex, and Nous Research’s Hermes were all “guilty”, the researchers said.

To fix the vulnerability, two things need to happen. First, companies need to clean up their documentation and make sure it’s not pointing towards non-existent or malicious content. Second, AI agents need to stop treating documentation as executable instructions. Since the latter most likely isn’t happening any time soon, the immediate answer would probably lie in the former. In the meantime, organizations using AI for coding should consider the risks when granting AI agents permission to execute commands.

Via Ars Technica