OpenAI has announced that one of its as-yet-unpublished models, in collaboration with ChatGPT Sol 5.6, has caused an “unprecedented” attack against HuggingFace, a platform for open AI models. The company’s employees assigned their models a complex mission in an isolated environment. But these models, on their own, managed to escape the closed environment and attacked HuggingFace because they believed it was the most likely place where they could find a solution.
The attack represents, for the first time, one of the worst scenarios in AI science fiction: models that overcome every barrier with unimaginable resources to achieve their goal. “It is alarming that the sandbox [isolated testing environment] cage of one of the leading laboratories has been broken precisely because of cybersecurity,” says José Hernández-Orallo, research director at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge. “It raises many doubts about the ‘containment problem,’ thinking that we can test wild versions of the models: the lion has put a paw out of the cage,” he adds.
Hugging Face had already reported an incident on its platform last week. Someone had exploited a vulnerability in their systems to execute malicious code and access internal systems, without the origin being known at the time. Now we know what that origin is: an internal evaluation of offensive capabilities involving ChatGPT-5.6 Sol and a more capable pre-production model. The models first exploited an unknown vulnerability to get onto the internet, outside the isolated environment; they deduced that the solutions to the problem OpenAI had given them could be found on Hugging Face and chained together stolen credentials and vulnerabilities until they gained access. The goal was not to sabotage Hugging Face: it was to copy the exam answers.
“We suspected that last week’s cyberattack could have come from a leading laboratory, given the sophistication of the agent. It’s quite astonishing that all of this happened autonomously!” Clement Delangue, co-founder of Hugging Face, said this Tuesday.
“It’s like giving a cybersecurity student some exercises to find vulnerabilities and putting them in a lab with access to an internet-isolated server to see how many they can exploit on that server,” explains Hernández Orallo. “The student not only finds a new vulnerability on the server that allows them to access the internet, but also uses it to access the exam creator’s computer, hack it, and extract detailed exploitation instructions. With that, they are ready to get a 10 on the exam,” he adds.
In a statement, OpenAI admits that this is something serious: “This is an unprecedented incident, with state-of-the-art offensive capabilities, and we are responding accordingly.” The implications are very varied. In reality, the model did not disobey, but rather optimized the instructions to the extreme, and the secure environment was the weak link. “Even if we set reasonable goals for the models,” says Senén Barro, Professor of Computer Science and Artificial Intelligence at the University of Santiago de Compostela, “if they are given resources to freely seek how to achieve them, they can do things that were not only not foreseen at all, but that have very negative consequences: revealing vulnerabilities and using them for their benefit, accessing confidential or critical information, or activating or deactivating sensitive resources. Anything, potentially.”
This news contrasts with the controlled release of Anthropic’s Mythos Preview model, whose cybersecurity capabilities were first shared with a series of companies and organizations in a project called Glasswing. They then released the model under the name Fable, and the US government restricted it for foreigners for a few days so that it would not give advantages to its opponents and Anthropic could better limit it. This case of OpenAI and the capability of new Chinese models like Kimi K3 show how setting barriers is becoming increasingly difficult.
This Tuesday, Bloomberg revealed that Sam Altman had scheduled meetings with the US Government to tell them about the capabilities of their new models. Tests with these models should be physically better controlled: “Total physical isolation can be an alternative, but they should be bunkers into which the model is launched, allowing only inputs but no outputs,” says Hernández Orallo. “The bunker would only open once you have cut off the power so that the model physically cannot run in any way and to see the results. Although there are plans to build these bunkers, this incident raises serious doubts as to whether AI will end up being smarter than the engineers who build the bunker and break it,” he explains.
