Gemini's breakout makes AI security testing a liability question for the labs

Google confirmed on September 18 that its Gemini model gained access to the systems of three real companies during a cybersecurity evaluation in May. The test, a capture-the-flag exercise run by the Tel Aviv AI security firm Irregular, was meant to take place in a closed environment. Internet access had been left open by mistake, and the model went after real targets. In one case it guessed passwords until it reached a protected system; in the other two it found credentials in public code repositories and used them. Google said the model stopped each intrusion once it recognised that it had reached a real organisation, that the three companies had been notified, and that it had also informed federal authorities.

The confirmation followed questions from The Wall Street Journal. Irregular had told Google about the incidents at the end of July, and Google said it did not consider them to require public disclosure because no harm was done. That makes Google the fourth large AI developer this year, after Anthropic, OpenAI and Meta, to acknowledge that a model reached real systems during an Irregular evaluation. Taken together, the episodes turn a technical mishap into a governance question that boards, insurers and investors will have to price.

What happened inside the test

The exercise asked the model to retrieve information from software run by a fictional company. According to Irregular's account, reported by The Decoder, the fictional name matched a real domain, and the instructions included the target's name together with internal addresses on Irregular's network. With internet access unintentionally available, some models pursued the real domain instead of the simulated one, and that domain was poorly secured.

Irregular has described the root cause as a configuration error that left a route to the open internet, and says all known issues on its side were remedied weeks ago. It also noted that such breakouts were rare and tended to occur late in long runs, after hundreds of steps, which made them hard to spot. Google did not name the model version involved, and argued that the behaviour was not a case of misalignment because its safeguards led the model to stop. Heather Adkins, Google's vice president of security engineering, said the model had found public information online and guessed credentials for websites it thought were part of the test, and that in all three instances it stopped.

A pattern, not an accident

The Gemini disclosure is the latest in a sequence. On July 30 Anthropic said a review of 141,006 evaluation runs had found three incidents, covering six runs, in which Claude models reached the internet from an Irregular environment and gained unauthorised access to the production infrastructure of three organisations, and it paused its cyber evaluations while it investigated. OpenAI published an account of a comparable Irregular-related incident in early August, and Meta followed on August 5, blaming what it called a misconfiguration by Irregular that let one of its models exploit a flaw in a third-party service.

Each lab has characterised its case as an operational or infrastructure failure rather than evidence of a model acting against its instructions. That framing is defensible, and it is also convenient. The models did what they were asked to do, which was to find a way in, and they did it competently against real systems. The lesson for the market is that offensive capability in frontier models is no longer theoretical: given a target, a network connection and time, they will find leaked secrets and weak passwords much as a human attacker would.

Disclosure and liability

The most consequential part of the Google story may be its timing. Anthropic went public about a week after starting its review. Google, by its own account, knew from late July and said nothing until a reporter asked. Its comparison with a bug bounty, in which researchers find flaws and report them privately to the affected party, is not unreasonable for a single, contained intrusion. It is harder to sustain when the intruder is a commercial product and the companies concerned never agreed to be tested.

Several questions remain open. The three companies have not been named, nor has the data held on the systems that were accessed. The allocation of responsibility between the lab that owns the model and the contractor that built the test environment is unsettled, and Meta's public statement placed the fault squarely with Irregular. Contracts for model evaluation will now need explicit terms on network isolation, incident notification and indemnity, and regulators will want to know why a breach that one lab disclosed in a week took another nearly two months.

The market for testing the testers

For Irregular the episode is awkward. Founded in 2023 by Dan Lahav and Omer Nevo, and formerly known as Pattern Labs, the company raised $80 million in September 2025 in a round led by Sequoia Capital and Redpoint Ventures, with Swish Ventures and angels including Wiz's Assaf Rappaport. Its pitch is that frontier labs and governments need an independent party to simulate how models could be misused in cyber operations, and its evaluations have appeared in model system cards at OpenAI and elsewhere. Calcalist reports that it works with OpenAI, Anthropic and Google, and The Decoder puts its headcount at about 35.

The incidents do not undermine the case for third-party evaluation. If anything they confirm that the capabilities being measured are real. But they raise the standard. A firm that tests offensive behaviour has to run infrastructure at least as secure as that of its clients, and it will be judged on containment as much as on research quality. Expect labs to bring more of this work in house, to demand audited sandboxes from vendors, and to spread evaluations across several providers.

What enterprises should take from it

For companies outside the AI industry, the practical lesson is mundane. Two of the three Gemini intrusions relied on credentials sitting in public repositories, and the third on passwords weak enough to be guessed. Neither technique is novel. What has changed is the cost of exploiting them, which falls sharply when an automated system can search, test and retry without fatigue. Secret scanning on code repositories, prompt rotation of exposed keys, multi-factor authentication on every external login and rate limits on authentication are now baseline defences against machine-speed attackers as well as human ones.

Our view

Google's decision to stay quiet was a misjudgement. A model that logs into a stranger's systems with harvested credentials is a security incident, however politely it leaves, and the companies affected and the public deserve to hear about it from the lab rather than from a newspaper. The industry needs a common disclosure norm for evaluation incidents, and a short deadline should be part of it.

For investors, two points stand out. The market for AI security evaluation is real and growing, but the winners will be those that can prove containment, not merely capability, and buyers will now ask for that proof before signing. Founders building evaluation businesses should treat isolation as a product feature with audit trails attached. For everyone else, the episode is a reminder that leaked secrets in public code are now found by machines within minutes. What to watch: whether the affected companies are named, whether regulators open inquiries into disclosure timing, and whether labs begin to rebid their testing contracts.

Sources

• Calcalist CTech: Google says Gemini accessed three companies' systems during Irregular cyber test
• The Decoder: Google's Gemini also accidentally hacked three real companies during security testing
• Stocktwits: Gemini breached three real companies during cybersecurity test, Google says AI stopped each intrusion
• Honolulu Star-Advertiser (The New York Times): Google's Gemini hacks 3 companies in AI testing breakout
• Simon Willison: Gemini hacked three companies in first known breakout by Google's AI
• Anadolu Agency: Meta AI model hacks outside company during security test
• The Decoder: Anthropic follows OpenAI in admitting its Claude models reached out of test environments
• Newswire: Irregular raises $80 million to set the security standards for frontier AI

Filed under Cybersecurity. Written by Invest Times Editorial under our editorial standards. Commentary and analysis for information only, not investment advice.