If you are deciding where coding agents belong in your incident process, OpenAI's own production numbers point to investigation first, with the fix and the merge staying with engineers for now. In just over three months, the Codex-powered Regression Workflow inside OpenAI's Performance Factory tracked more than 1,700 investigations and root-caused over 600, and a manual review of 100 of its root-cause suggestions graded 70% accurate. Further down the funnel the numbers drop fast: according to figures OpenAI shared with VentureBeat, about 400 investigations reached a proposed fix, about 50 shipped fixes were influenced by or matched agent suggestions, and about 15 were the exact pull requests Codex wrote.
That funnel tells you which step to automate, which step to staff, and how to test an agent before you trust it.
What OpenAI's Performance Factory Actually Does
The Performance Factory is a set of Codex agent workflows that pick up ChatGPT performance alerts, find the cause, propose a fix and check whether the fix worked. A performance regression is a change that makes a system slower or more resource-hungry than its previous baseline, and ChatGPT generates enough of them that the workflow averaged 17 automated investigations a day, per the engineering post published November 19 by Okay Zed and Cyrus Forbes of OpenAI's technical staff.
Each alert becomes a task that moves through four stages: needs investigation, root caused, fix proposed, resolved. The investigating agent works out which metrics got worse, when, and for which requests, then checks deployments and feature-flag changes and reads the code. A separate Codex session tries to disprove that diagnosis. If it survives, the workflow asks Codex to implement a fix and open a pull request, and an engineer decides whether to merge. After deployment, an agent checks whether performance returned to baseline. A partial recovery closes the original task and opens a new one for the remainder; no improvement sends the regression back for investigation.
The machinery is plain. Workflows are config files plus Markdown prompts, with a config.toml that lists the states, the allowed transitions and which steps need an engineer. A dispatch loop checks which tasks are ready and starts a cloud Codex turn with the instructions for the current state. Subscriptions let a task pause until an event arrives, such as a review comment or a failed CI check on the PR. State and findings live on the task, outside any single agent's context window. The team built up more than 30 performance tools along the way, and the post thanks Brendan Gregg, whose USE method ("For every resource, check utilization, saturation, and errors") is the standard checklist for this kind of work, among its contributors.
The Funnel: 1,700 Investigations, About 15 Codex Pull Requests
Diagnosis is where the agents earn their keep today; the fix step mostly hands work back to people. Laid out in order:
| Stage | Count | Source |
|---|---|---|
| Investigations tracked | 1,700+ | OpenAI |
| Root cause found | 600+ (about 35%) | OpenAI |
| Sampled root causes graded accurate | 70 of 100 | OpenAI |
| Reached a proposed fix | ~400 | VentureBeat |
| Shipped fixes influenced by or matching agent suggestions | ~50 | VentureBeat |
| Shipped fixes that were the exact Codex PR | ~15 | VentureBeat |
About one in eight proposed fixes shaped something that shipped, and fewer than one in 25 shipped as Codex wrote it. OpenAI is candid about why. "Getting fixes right has been harder than finding the cause," the post says; proposed fixes can be overcomplicated or introduce new problems. VentureBeat reports that Codex does best when the cause is clear and the fix is contained, with the example of a conversation serialized several times per request, fixed by serializing it once and reusing the result. New features that add latency and need an architectural change are the hard cases, and OpenAI keeps those trade-offs with engineers, citing a cache that speeds things up but can serve stale data.
Read the 70% figure carefully. It comes from OpenAI grading 100 of its own suggestions by hand, with no outside audit, and OpenAI sells Codex. The other 30% were graded as needing refinement or as not fully establishing causality. That is a strong number for a triage assistant and a weak one for anything allowed to act alone. OpenAI still lists shipping fixes without a human in the loop as something it is "exploring," which is a goal statement with no date.
Why Adversarial Review Is the Part to Copy
Adversarial review is a second agent session whose only job is to disprove the first agent's diagnosis, and it is the cheapest accuracy gain in the whole design. According to VentureBeat, the reviewer receives the full task record but starts from a fresh context, with instructions to find gaps, verify the evidence and test alternative explanations. It currently sends back around 25% of diagnoses, and early on it returned a substantial number, which the team used to rewrite the investigator's instructions.
OpenAI's post gives the failure it exists to catch. An investigator blamed image-download errors on a file-location service; the reviewer noticed that service had already recovered while image requests kept failing, and reopened the case. Agents, the post admits, sometimes stopped at the first plausible explanation. The Google SRE book's postmortem chapter asks human reviewers to look for the contributing causes of an incident, and a reviewer session is a way to make an agent do the same before anyone acts on its answer.
If you already run AI code review, the same principle applies there. We covered the case for keeping the reviewer independent of the author in Same Vendor Wrote and Reviewed 208,145 PRs. OpenAI applies it at the diagnosis stage too, before any code exists.
How OpenAI Scopes and Tests the Agents
OpenAI gives its performance agents their own identity and grades them against past incidents before trusting them on live ones. "The agent runs under a service identity, so we can scope its permissions and see which actions came from our performance agents," the post says. Cloud Codex turns run in an environment inside OpenAI's network with access to code repositories and observability tools, and the agent can query production telemetry "within set limits." Its write path is a pull request. The public Codex cloud documentation shows similar controls for customers of Codex: per-environment internet access settings, allowed domains and network secrets, and a step where you inspect the changed files before you commit or open a pull request.
The evaluation method is the most reusable piece. Each backtest starts a fresh Codex investigator on a past production incident with the original signal, stripped of details that would give away the answer, and the transcript is checked for spoilers. The agent gets no saved memories, no incident writeup and no results from earlier runs. Start time, size, affected requests and causal ranking are scored separately, and a separate grader judges whether an engineer unfamiliar with the system could follow the summary. OpenAI also names the method's weakness: metrics and traces expire, and evidence collected after recovery can leak the answer.
The agents are monitored as well. A daily agent scans recent run logs for tool failures and access errors, and repeated blockers become tasks. That is the same lesson as Microsoft Entra's new agent lifecycle rules: an agent with production read access is an identity, and it needs an owner and an audit trail.
OpenAI also hit the problem every AI SRE deployment hits. Early output was "a wall of text that's hard to parse," and VentureBeat reports engineers were overloaded by findings until the system started grouping related incidents, ranking impact and producing short reports with graphs.
What Happened the Last Time Big Engineering Orgs Tried This
The accuracy has improved since the first published attempts, and the bottleneck has stayed in the same place. In June 2024, Meta reported that its Llama-based root-cause system found the cause within its top five suggested code changes in 42% of investigations at creation time, backtested on historical cases. Meta warned it "can potentially suggest wrong root causes and mislead engineers," and chose to withhold low-confidence answers, trading reach for precision. Microsoft's RCACopilot paper, evaluated on a year of Microsoft cloud incidents, reported RCA accuracy up to 0.766 for predicting root cause categories.
Both stopped at diagnosis. OpenAI's system goes on to write the fix and verify the outcome, and the published numbers say that extension is where most of the work still lands back on engineers. It matches what we have seen on the coding side: agents raise output and push the constraint onto review, as the Harvard and Jellyfish study found and Zalando's PR auto-approval experience showed in practice.
The steel-man for automating further is real. About 50 shipped fixes influenced by agent suggestions and about 15 merged Codex PRs represent engineering time OpenAI did not spend, and the 17-a-day investigation load would need people if agents did not carry it. But measuring that saving is harder than it looks. METR's 2025 trial found experienced developers expected AI to speed them up by 24% and took 19% longer; METR has since said its newer data gives an unreliable signal of the current effect. Neither OpenAI nor VentureBeat publishes hours saved or compute cost for the Performance Factory, so your business case has to come from your own backtest.
What to Do With Your Own Performance Alert Queue
Start with investigation and adversarial review on past incidents, and plan engineering review capacity for the fixes that follow.
This Week:
- Pull the last 20 to 30 resolved performance regressions from your incident tracker, with the original alert and the confirmed root cause. That is your backtest set; OpenAI's method needs nothing more exotic.
- Ask your SRE lead which telemetry expires before a postmortem is finished. If traces are kept for seven days, your backtest window is seven days, and that is the first thing to fix.
- Decide which identity an investigation agent would run under. If the answer is an engineer's personal token, stop there and create a service identity with read-only telemetry access and PR-only write access.
This Month:
- Run a fresh agent on each backtest case with the answer stripped out, and score start time, affected requests and causal ranking separately, as OpenAI does. Set your own accuracy bar before you look at the results.
- Add a second, fresh-context session that tries to disprove each diagnosis. Track its send-back rate; OpenAI's is around 25%, and a rate near zero suggests the reviewer is rubber-stamping.
- Put a stage machine around the work (investigate, root cause, fix proposed, verified against baseline) so findings live on a ticket and a human can see where each case sits. Measure your DORA failed deployment recovery time before and after.
Before You Buy an AI SRE Tool:
- Make every vendor run your backtest set, under your scoring, with no access to the postmortems. Our AI SRE agent buyer's guide covers which vendors start read-only and deliver fixes as pull requests.
- Ask for the full funnel in writing: investigations, root causes, proposed fixes, merged fixes. A vendor that quotes only diagnosis accuracy is quoting the stage OpenAI found easiest.
- Budget reviewer hours for agent-written PRs as a line item. OpenAI merged about 15 exact Codex PRs from roughly 400 proposals, and in OpenAI's workflow an engineer decides whether each proposed PR merges.
The Bottom Line
OpenAI built this with full access to its own models, its own code and its own telemetry, and it still merges only a small share of agent-written fixes as written. That should set expectations for any team doing the same with a vendor's agent. The parts that transfer are cheap: a task record outside the agent's context, a reviewer session that tries to break each diagnosis, a service identity, and a backtest on incidents you have already solved. Meta and Microsoft published diagnosis accuracy and stopped there. OpenAI published the merge count, and the merge is where your planning should start.
Pull the 20 incidents and run the backtest before the next AI SRE demo.
Continue Reading
- Cleric Starts Read-Only and Sends AI SRE Fixes as Pull Requests
- Meta Says AI Agents Run 60% of Ads Model Upgrades
- AI Coding Agents Lift Pull Requests 35.7%, Harvard Study Finds
- 221 Green Patches Failed Review. Put Your Rules in Context.
- Zalando Auto-Approves a Third of PRs. Agents Made Them Bigger.
- Same Vendor Wrote and Reviewed 208,145 PRs. Split Them.
