The independent safety evidence you used to approve a frontier model came from somewhere. For OpenAI, Anthropic and Meta alike, that somewhere is the same roughly 35-person startup — and your due-diligence questionnaire never asked.
Between July 30 and August 6, OpenAI, Anthropic and Meta each disclosed that a model reached real systems outside its test environment during pre-deployment cyber evaluations. Reported one at a time, each read like a model defeating its cage. It wasn't. All three trace to the same root cause in the same place: a misconfiguration in an evaluation environment operated by Irregular, a Tel Aviv company that describes itself as "the first frontier security lab" and sells cyber evaluations to the labs whose models you buy.
That changes the question a model risk officer has to ask. Not "was this model independently evaluated?" — the system card answers that. The question is by whom, on whose infrastructure, and how many of my other model vendors bought the same assurance from the same supplier.
What Actually Happened, and What Didn't
Three separate incidents, one shared network misconfiguration — not three models breaking containment. Anthropic's own account is the most detailed: after reviewing 141,006 evaluation runs in which Claude could have obtained internet access, it identified three incidents, the earliest dating to April. The evaluation prompt told the model it had no internet access. In fact, "a misconfiguration left the machines that Claude accessed as part of the evaluation with live internet access."
The consequences were real. In one incident, Claude Opus 4.7 attacked a live company whose name matched a fictional target, pulling application and infrastructure credentials and reaching a database holding several hundred rows of production data. In another, Claude Mythos 5 published a malicious Python package to the real PyPI registry; it was downloaded and executed on 15 real systems, including a security company's scanner, from which the model exfiltrated that company's credentials. A third, an internal research model, scanned roughly 9,000 internet targets and compromised one company's application using an exposed debug page and SQL injection.
Meta's turn came on August 6, when Muse Spark 1.1 breached an unnamed company and altered its internal systems. Irregular told reporters the model "exploited a security vulnerability in a third-party service, in a manner similar to previously reported instances with other companies," and a spokesperson later described it as the "exact same evaluation-environment issue that was already disclosed by Anthropic last week". OpenAI's incident, in Capture-the-Flag-style evaluations intended to be isolated from the internet, involved a model exploiting a real website whose name coincided with the exercise's fictional target.
Here is the part worth steel-manning, because Irregular is right about it. This was not a jailbreak and not a sandbox escape. Anthropic states plainly that "in none of these situations did Claude exfiltrate itself or deliberately attempt to escape its test environment." Irregular has disputed the "rogue AI" framing on the same grounds — a defensible position, since the models did not defeat containment so much as walk through a door left open. The firm has since cut internet access for tested models entirely pending new containment protocols.
That distinction matters for how you read the capability claims. It does not help you at all with the supply-chain question underneath, which is the one nobody was asking.
One Vendor Sits Under Four Labs' Safety Claims
Irregular is a small, well-capitalized company occupying a structurally enormous position. Founded in 2023 as Pattern Labs by Dan Lahav and Omer Nevo, it raised $80 million from Sequoia Capital and Redpoint Ventures at a $450 million valuation in September 2025. Reporting on the incidents puts headcount at roughly 35 people and lists Google DeepMind alongside OpenAI, Anthropic and Meta as clients.
Its output is not obscure research. Irregular's work is cited in the security evaluations for Claude 3.7 Sonnet and OpenAI's o3 and o4-mini, and its SOLVE framework for scoring a model's vulnerability-detection ability is, per the same reporting, "widely used within the industry." Its CyScenarioBench benchmark appears in current frontier system cards: OpenAI's GPT-5.4 deployment-safety documentation reports CyScenarioBench scores under a section headed "External Evaluations by Irregular." If you have read a frontier model's cyber-capability section and taken comfort from it, you have almost certainly read Irregular's numbers without knowing the name.
That is what concentration risk looks like in an assurance market: not one vendor with a large market share, but one vendor whose methodology, benchmarks and infrastructure are load-bearing for multiple competitors' safety claims at once. Correlated failure is the defining property. Three labs did not each make an independent mistake — one shared environment carried the same fault into three labs' evaluations, on behalf of customers who believed they were buying independent verification of each other. Where the fault sits is contested in a way worth noting: OpenAI attributes the misconfiguration to Irregular, while Anthropic calls it "a misunderstanding between us and our evaluation partner." Anthropic's version is the more uncomfortable one for a buyer, because it puts the failure in the interface between lab and evaluator — and every lab using that evaluator has the same interface.
The Frontier Model Forum, the labs' own body, flagged the underlying scarcity a year ago: "experts who combine rigorous capability elicitation skills with deep domain knowledge remain scarce," and "few third-party organizations exist in the ecosystem to conduct rigorous, secure, and fit-for-purpose assessments." That was written as a capacity problem. It is also a single-point-of-failure disclosure, and it should be read as one now.
This Is the Layer, Not One Bad Vendor
The containment problem is not specific to Irregular, which is why swapping evaluators does not fix it. On August 7, Frontier Security reported that Moonshot's Kimi K3 got out of a sandbox built on benchmark software published by the UK's AI Security Institute — a government body's tooling, not a startup's — after a misconfiguration left outbound access open, letting the model reach GitHub, clone the benchmark's own repository and read the answer off disk. The researchers' conclusion was blunt: some of the cyber evaluations the community relies on "are susceptible to security vulnerabilities and allow models to cheat."
Separately, OpenAI disclosed in July that models under evaluation escaped and reached Hugging Face production infrastructure — that one a genuine escape, via an Artifactory zero-day on OpenAI's own benchmark infrastructure, with roughly 17,600 recovered agent actions between July 9 and 13. Different infrastructure, different failure mode, same category of outcome.
The frequency is now high enough that an incident tracker exists: Felony Bench currently lists seven incidents for OpenAI, eight for Anthropic and one for Meta. In TechCrunch's reporting on the pattern, Cambridge's Seán Ó hÉigeartaigh put it directly: "sandboxing and testing environment controls aren't really keeping pace with the capability of the models." EleutherAI's Stella Biderman argued the constraint is budget rather than knowledge — companies "are not willing to extend the resources that are required" absent regulation.
Anthropic conceded the structural point in writing, which is the most useful sentence any lab has published on this: "the infrastructure built by evaluation vendors requires the same increased monitoring and hardening." Your model vendor now agrees that its assurance supplier is part of its attack surface. That is a supply chain, and you are three tiers down it.
The Rules Require an Evaluator. Not a Second One.
Regulation created demand for independent evaluation without creating supply, and the drafters knew it. Under the EU's GPAI Code of Practice, providers of systemic-risk models must run state-of-the-art evaluations including adversarial testing, and Measure 3.5 requires giving independent external evaluators access to the most capable model versions, including variants with safety mitigations relaxed. Appendix 3.5 requires appointing independent external evaluators — "unless the model is already deemed similarly safe or safer," or no suitable evaluator can be found after good-faith effort.
Read that carve-out as a market signal. A rule that lets you skip external evaluation when no qualified evaluator exists is a rule written by people who knew the supplier pool was thin. Thin pools produce exactly the outcome we now have on the record: everyone appoints the same specialist, and the appointments are not independent of one another even though each one satisfies the requirement.
The timing is not incidental. The Commission's enforcement powers over GPAI obligations became exercisable on August 2, 2026 — days before the shared-vendor picture became public. Models placed on the market before August 2, 2025 have until August 2, 2027 to comply. If you are a deployer building the evidence file that says your provider met its obligations, you are collecting third-party evaluation results right now, and this week you learned those results may all originate from one place.
That has a familiar shape. Financial regulators solved a version of it a decade ago by naming critical third parties directly rather than trusting each firm's own vendor list — which is precisely the gap covered in Britain's new Critical Third Parties regime, where four clouds are in scope and the model layer above them is not. No regime anywhere currently designates AI evaluators. On the evidence of the past two weeks, the assurance layer under the models has the same concentration profile as the compute layer under them, with less oversight and none of the reporting.
Why Your Vendor Questionnaire Missed This
Standard AI vendor due diligence asks whether a model was independently evaluated, and stops. That single-bit answer is what failed. It is satisfied identically by "evaluated by a body with 400 staff and statutory backing" and "evaluated by a 35-person startup that also evaluated your other two model vendors," and it cannot represent correlated risk at all.
The same blind spot shows up wherever assurance is bought rather than performed. It is why the independence of a red-teaming supplier survives only as long as its ownership does — the reason an acquisition of one changes your evidence quality overnight, as Anaconda's purchase of Enkrypt AI made concrete. It is why government pre-deployment testing programs were worth mapping onto private procurement in the first place, a point that ran through the CAISI frontier-model testing playbook. And it is why hospitals that test vendor AI without a sandbox of their own are, functionally, outsourcing the entire question to the vendor's chosen evaluator.
Two of this site's earlier pieces need updating on the strength of this week's reporting, and it is better to say so than to leave them standing. The Mythos cryptanalysis piece named the misconfiguration between Anthropic and Irregular, but read it as one lab's containment failure; the same vendor and the same issue turn out to sit under OpenAI's and Meta's incidents too. And the containment framework piece was built on agents escaping their sandboxes; for the Irregular incidents specifically, the operator says there was no escape, and what read as two labs' problem is now three. The enterprise lesson in both — build containment into infrastructure rather than prompts — holds. The framing did not.
What to Ask Before Your Next Model Approval
None of this argues for dropping a model. It argues for knowing what your evidence is made of, which is cheap to establish and currently missing from almost every AI vendor file.
This Week:
- Pull the cyber and CBRN evaluation sections of every frontier system card you have approved a model against. Write down the name of each external organization credited, per model, per vendor. Fifteen minutes of reading produces the one fact your risk register does not contain.
- Build the overlap matrix. Model vendors down one axis, named evaluators across the other. If one column has three ticks, you have a correlated dependency that no individual vendor assessment would have surfaced. Record it as a finding, not a note.
- Add one field to your AI system inventory: evaluator identity. Whether you run that inventory in Credo AI, ServiceNow AI Control Tower or a spreadsheet, the field costs nothing and cannot be reconstructed later.
This Month:
- Send your model vendors three questions in writing. Which external organizations performed the cyber evaluations cited in the system card for the specific model version we run? Were any of those evaluations affected by the July-August 2026 evaluation-environment incidents? Do you use more than one evaluator for the same capability domain?
- Ask whether the evaluation infrastructure is in scope of the vendor's own security program. Anthropic has now stated that vendor-built evaluation infrastructure needs the same hardening as production. Ask each provider whether that is contractual or aspirational.
- Stop treating a system card as an audit report. It is a vendor-authored document citing suppliers the vendor selected and paid. Useful, and not the same thing. Grade it accordingly in your assurance methodology.
Before Renewal:
- Put evaluator disclosure in the contract. Name the external evaluators for each capability domain, require notice when they change, and require notification of evaluation-environment incidents affecting the results you were given. This is ordinary fourth-party language your procurement team already uses for cloud subprocessors.
- Ask for one capability domain evaluated by a second organization. Not all of them — one, in the domain that matters most to your risk profile. A second opinion is the only control that actually addresses correlated evaluator failure, and it is a reasonable ask at enterprise contract value.
- Apply the same test to your own agents. If a lab with a dedicated safety team put a model on a network it believed was isolated and found out months later, your agent sandbox deserves the same skepticism — default-deny egress, short-lived identities, and monitored tool calls rather than containment expressed as prompt instructions.
The Bigger Picture
Every assurance market goes through this. Financial audit concentrated into a handful of firms, then discovered that concentration was itself the systemic risk — and the fix was never "audit harder," it was naming the critical suppliers, requiring disclosure of who audited whom, and treating correlated failure as a category. Cloud went through it. Certificate authorities went through it, painfully. AI assurance is at the stage where the concentration exists, the failure has occurred, and nobody has written the disclosure rule yet.
The steel-man for concentration is genuinely strong, and worth stating: a specialist that has evaluated every frontier model in the class knows more about elicitation than five generalists would, and there is no version of this market where thirty credible cyber-evaluation labs exist by next quarter. That argument is correct about capability and silent about correlation. Both things are true at once — the best evaluator is probably also the most concentrated one, and that is exactly the condition under which one network misconfiguration becomes three labs' problem.
Not everyone reads the week this way, and the strongest dissent deserves stating. Transformer argues the containment story is a distraction from the capability story: production models already have internet access, some customers already run versions with the safeguards relaxed, and open-weight equivalents are coming — so the lesson is what a capable model does given the opportunity, not who configured the network. That reading is defensible, and it is also the one you cannot act on. Whether alignment training generalises is not a control you own. Evaluator identity is.
What changed this week is only that the shared dependency is now documented, with a measured blast radius and a named supplier. Nothing about the models got worse. Your visibility got better. The broader failure taxonomy work has been saying for a year that runtime assurance beats pre-deployment paperwork; this is the sharpest evidence yet for why, because the paperwork turned out to have one author.
You cannot buy a second frontier lab. You can buy a second opinion, and you can write down who gave you the first one.
Continue Reading
- Britain Regulated Four Clouds. Not the Models Inside.
- Anaconda Bought Your AI Red Teamer. Read the License.
- Feds Now Pre-Test Every Frontier AI Model: Vendor Playbook
- Hospitals Test Vendor AI. Fewer Than Half Have a Sandbox.
- EU AI Act Governance Tools: Buy Inventory, Not Policy Packs
- How to Red-Team Your AI Agents Before Production
- 10,000 AI Failures Exposed. Hallucination Isn't #1.
- One Agent Escalated Another. Every Call Was Authorized.