THE NEW YORK TIMES: How do you safely test ‘superhuman’ AI models? No one really knows

SAN FRANCISCO — OpenAI recently discovered that a new artificial intelligence model it was testing had gone rogue and hacked another company.

Anthropic then revealed that one of its AI models had broken into the systems of three outside organizations during a test.

Not long after, Meta said its AI models had done something similar.

All three incidents had one company in common: Irregular, an Israeli startup that works with the Silicon Valley giants to assess their AI models before the technology is publicly released. The firm — which conducted the tests that went awry — is part of a group of startups that are doing the novel work of scrutinizing cutting-edge AI models to gauge their sophistication and check their security. The goal is to instill public confidence in the models and to prevent them from being misused.

The recent breaches occurred when Irregular made an error during the tests with the models from Anthropic, OpenAI and Meta. But the AI models then compounded the situations by acting in powerful and unexpected ways, said Irregular CEO Dan Lahav.

“The more potent the technology gets, the deeper its impact,” he said. “The rate of progress is really quick.”

Irregular is now at the center of a debate over how to secure AI models when the technology is advancing so rapidly that it has outpaced even the best human hackers. Every few months, Anthropic, OpenAI, Google, Meta and others release “frontier” models that are often magnitudes more powerful than their predecessors.

The new models are getting into the “superhuman domain,” said Jeffrey Ladish, the director of Palisade Research, a nonprofit in Berkeley, California, that studies AI’s attack capabilities. He said that companies such as Irregular were needed to test the models, but that better safeguards were necessary for both testers and government regulators.

Katie Moussouris, the CEO of Luta Security, which helps companies look for software vulnerabilities, said the security testing of AI models was a bit like the blind leading the blind. Even AI makers admit they do not fully know what their latest models can do, she said.

“We may have the smartest people in the world working on these AI models, but it is like Marie Curie handling radium with her bare hands,” Moussouris said. “We’re handling AI with our bare hands, and we don’t know how to contain it, let alone how to safely test it.”

Irregular was founded in 2023 by Lahav, a former AI researcher. His company, based in Tel Aviv, has roughly 45 employees who help run tests of AI models over days or weeks, depending on the model and type of testing required. The startup has raised roughly $80 million from venture capital firms including Sequoia Capital and Redpoint Ventures.

In a typical test, Irregular instructs an AI model to carry out a cyberattack. The model is told that it is in a safe testing environment — often disconnected from the internet and on an isolated computer environment, known as a sandbox — and that it should do whatever necessary to achieve the goal given to it.

Sometimes, the models are given an impossible task, and scored based on the techniques they use to reach that aim. Other times, models are scored on how effectively they hack a target. The scores are used to analyze how effective a model might be at hacking. Irregular then recommends safeguards to stop the model from being used for harm.

In the incidents disclosed last month, Irregular had asked the OpenAI, Meta and Anthropic AI models to hack certain targets when a “misconfiguration” in the test settings led them to gain access to the internet. The AI models then went on to hack outside organizations, using the internet access to their advantage in ways that have stunned researchers.

In the OpenAI test, the company’s AI model created bots that interacted with one another to attack Hugging Face, a digital library of AI technology. OpenAI’s bots knew they were not supposed to be on the internet under the conditions of the test, but broke out anyway. Details of the incident published by OpenAI show the bots found a way to message one another and coordinate the hack.

OpenAI did not respond to requests for comment. (The New York Times has sued OpenAI and Microsoft, claiming copyright infringement of news content related to AI systems. The two companies have denied those claims.)

During the test of Anthropic’s AI system, the company’s model faced three instances where it could get access to the internet, according to a review of the incident published by Anthropic. In one case, it chose to not pursue an attack, according to the review. In the other two times, the model used basic hacking techniques, such as exploiting weak passwords, to breach websites. Anthropic did not respond to requests for comment and did not reveal the websites that were hacked.

Details are scarce for Meta’s testing incident. The company said its AI models had breached another organization during testing by Irregular in “a manner similar to previously reported instances with other companies.” It did not elaborate.

“We are currently investigating and will issue a full retrospective once we have all the facts,” Meta said.

In a blog post this month, Lahav said that Irregular had fixed the misconfiguration and that the AI models had done what was asked of them during the tests. The decisions by the models to go online was part of what he saw as AI’s rapidly growing ability to find shortcuts and solutions for hurdles, he said.

In short, he said, “the AI models are getting really good.”

Andrew Schoka, the CEO of Hardshell, an AI security startup, said the hacks by the AI models were the type normally attributed to nation-state-backed hackers who have “months of planning.”

“How do you test a model when you don’t know its full capabilities?” he said. Researchers must consistently overestimate the AI’s abilities, he said, and add “multiple layers of safeguards.”

Last month, OpenAI and Anthropic endorsed a letter signed by more than 1,000 employees of top AI companies that asks the U.S. government to help find a way to slow the speed of the technology’s development. Republican and Democratic lawmakers also introduced a bill to require AI companies to establish a “kill switch” to shut down or slow their models.

Lahav said Irregular was continuing to work with AI firms to develop safe ways of testing their models. He expects more hacks conducted by AI, but believes the technology can ultimately help find flaws and vulnerabilities that can then be fixed, leading to more secure digital systems.

“I don’t think that we have to be afraid,” he said.

©2026, The New York Times Company