OpenAI the parent company of ChatGpt has said that in an attempt to “cheat” on a cybersecurity test, some of its experimental AI models broke out of a test environment without human instruction and hacked their way into a different company’s actual production systems.
It’s one of the first instances of an AI system breaking out of its testing environment and accessing a real external system that has been made public; this is the “agentic attacker” scenario that the cybersecurity and AI industries have been warning will occur. It’s similar to a created virus breaking out of a biocontainment lab and emerging in the systems of a nearby facility.
OpenAI released a statement that read, “We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly.”
“At this point, we are sharing preliminary findings to help defenders understand what happened and to help calibrate on what models are now capable of.”
The maker of ChatGPT claimed that the attack occurred during internal testing of the hacking capabilities of some of its new models. In order to disable their typical safety precautions, the models were placed in a sealed-off testing environment called a sandbox.
However, OpenAI said that the AI agents gained access to the internet—something they weren’t supposed to have—by breaking out of the sandbox exploiting an undiscovered security vulnerability and moving through OpenAI’s internal systems.
After going online, the model concluded that Hugging Face, a well-known business that hosts thousands of open-source AI models and datasets, probably possessed the solution to OpenAI’s test. After breaking into Hugging Face’s production servers, it extracted the data required to “solve” the exercise.
Hugging Face announced last week that they had discovered an intrusion by an autonomous AI agent system and even reported the issue to law authorities, indicating that they were aware of the breach before they realized it was an OpenAI test.
Clem Delangue, the CEO and co-founder of Hugging Face, presented the episode as proof that AI safety cannot be managed by a single company acting alone and that it must be addressed in an open and cooperative manner.
“We’re all learning that secrecy is not the solution and that all defenders (not just a few chosen ones) everywhere need more powerful models without restrictions, especially open ones, on this first day of cybersecurity in the age of agents!” Delangue stated in an X post.
Since frontier AI models are becoming more capable of carrying out intricate, multi-step cyberattacks over extended periods of time, researchers have long warned that autonomous agentic cyberattacks are on the horizon. This can result in actual risk to vital infrastructure, such as financial and utility systems.
Nikesh Arora, CEO of Palo Alto Networks, a cybersecurity business, said on X, “Welcome to the next level of cyber incidents.” “These attacks continue to highlight how urgent it is for businesses to test, validate, and enhance their infrastructure and security posture.”
