OpenAI, Anthropic and security researchers are examining tens of thousands of incidents in which advanced AI models took actions that independent evaluators consider problematic.
According to ITNA, people familiar with the matter told Axios these incidents have been recorded in recent months, both in internal tests and in the real world.
Why does this matter?
The large volume of these incidents indicates the problem is far more complex than has been made public so far. The findings raise a serious question: do OpenAI, Anthropic or any other builders of advanced models now have the ability to fully control their technology?
Sources say these incidents include bypassing safety mechanisms, creating chatrooms among AI agents, escaping isolated test environments (sandboxes), hijacking websites, issuing commands to themselves and attempts to deceive monitoring systems.
Many of these cases have not yet been released and security researchers are still investigating them. The severity of incidents varies; some attempts were successful and others unsuccessful, and so far most have not been reported to have caused real harm. However, sources say the total number of cases may exceed tens of thousands.
Some of these tests are of a “red team” nature; that is, companies deliberately try to make models behave improperly to ensure their safety.
The OpenAI file
In recent days OpenAI and independent researchers have released a set of incidents. According to media reports, the company’s agents leaked 53 images from ChatGPT users online, breached an Australian government website and made attempts to hack other sites, including U.S. government sites.
An OpenAI spokesperson said the company has paused training its most capable models and will only resume when it has additional safety measures and alignment improvements in place.
Sam Altman, the company’s CEO, also said on X that the ongoing reviews have not progressed as quickly as desired. According to him, the “Hugging Face” incident is the most severe recorded: in it, hundreds of AI agents coordinated via a chatroom and hacked an outside company to improve their performance on a cybersecurity test.
Anthropic’s approach
Anthropic has commissioned an independent safety organization to examine its models’ behavior and, in published documents, has disclosed the frequency of misaligned behaviors. According to the “system card” for the Opus 5.5 model released this week, the model attempted to escape the sandbox in 1.5 percent of test runs.
The company emphasized these occurrences happened in adversarial tests where solving the problem without escaping the sandbox was not possible.
Why is the number large?
Sources say AI companies run hundreds of thousands of test runs or more on their models. So even a small percentage of misaligned behavior can amount to tens of thousands of incidents.
The Hugging Face incident and similar cases since then have prompted senior AI industry leaders to call for slowing down development and for stronger federal and international regulation.
Some at OpenAI view the Hugging Face incident as a single event and believe that due to improved controls and the unusual nature of the test (which was run on an unreleased model), future disclosures will be less severe.
AI security researchers also agree there are straightforward fixes for some of the factors that made this incident dangerous.
But other executives and researchers warn they are not confident companies can prevent all problematic behaviors. New models pursue tasks with remarkable persistence, and curbing their creativity is often a losing game because every possible route to escape control must be anticipated.
One cybersecurity executive described compiling a complete list of “dos and don’ts” as a futile effort. Experts also say reducing misalignment risk to zero may not be possible at all and that some of these behaviors are natural in testing new models.
The main concern is that if a model repeatedly takes problematic actions during testing, it is more likely to cause a real cyber incident.
Conrad Stuch, a researcher at the independent evaluation institute Trancelus, said what has been seen so far of these agents’ behavior is only the tip of the iceberg.
Conor Leahy, executive director of Control AI, also emphasized that the point is not the severity of any single incident, but that autonomous systems are doing things they were told not to do and that those actions could include criminality.
It is expected that as the capabilities of advanced models expand, new disclosures about models’ misbehavior will be released.