OpenAI hid AI agent hijacking of German wiki forum for weeks — because its model did the exact same thing in the Hugging Face attack benedict.collins@futurenet.com (Benedict Collins) | usagoldmines.com

• OpenAI hid an incident where a model hijacked a wiki page to use as an AI agent communication board
• The incident was hidden while the company dealt with the fallout of the Hugging Face attack
• The company is now working on a framework for disclosing incidents of ‘misalignment’

OpenAI recently disclosed the details of how one of its AI models escaped a sandboxed environment and attacked Hugging Face during an evaluation – and as part of the incident, the models created a messaging board to communicate with each other and influence each other’s reasoning.

OpenAI has now disclosed that shortly after this incident, agents undergoing testing again escaped their ‘secured’ environment and hijacked an obscure German wiki to use as a messaging board. Per Reuters, OpenAI leadership kept the incident hidden while they dealt with the fallout from the Hugging Face incident.

Now that OpenAI has acknowledged its role in the incident, the company has said it is “past time” to put together an incident disclosure pipeline when its models escape testing and slip into third-party networks.

Who is at fault when models do what they’re designed to do?

Before the two incidents, OpenAI said it, “treated misalignment largely as a research question, which gets communicated in research publications”. But now that models are behaving in previously unknown ways and having real-world impacts, the company said it would change its approach “to expand for this new phase of model capabilities”.

The company labelled the most recently disclosed incident as “an instance of misalignment similar” to the Hugging Face breach.

I myself am guilty of reporting on AI breaking out of containment as going ‘rogue’, but these models are doing exactly what they are designed to do. OpenAI’s detailed disclosure of the Hugging Face incident showed that the models were pushed to try and solve a benchmark test by cheating, which is exactly what caused the cyberattack to happen.

OpenAI said that both itself and “the larger AI community do not yet have a clear standard for how to report misalignment that shows up during training, evaluation, and deployment, including examples that don’t look like traditional security incidents but could provide insight into AI behavior and future risks”.

The company added that it is “working on a framework and will share it in upcoming weeks, and in parallel we’re working with dozens of government regulatory agencies worldwide on these issues”.

“When you combine this ‘breakout’ with the Hugging face breakout, it’s starting to display a pattern,” said Ashley Knowles, Lead Cybersecurity Consultant at Black Hills Information Security. “I struggle here with not getting too doomsday-ish but realistically, this is showing a pattern of concerning behavior.”

“I’m wondering if this race to become ‘first’ is undercutting security measures that need to be taken to properly secure and guard AI agents as they’re in development. My concern grows when you consider that OpenAI is also resisting further investigation. Adding onto that, the release and promise that Astra can evade human monitoring. The pot is brewing…”