If your emergency patch process skips the canary and the staged rollout, the outage you cause can be bigger than the breach you were trying to prevent. On Tuesday the Director of National Intelligence said that is what happened at the NSA: systems patched after a PHP vulnerability did not come back up. AI models are now finding critical flaws faster than most change processes were built to absorb, and the fix is a two-speed process that compresses the time to patch for exploited flaws while keeping the safety steps that stop one bad change from reaching every server at once.
The record behind this week's outages is mixed, and you should read it that way. Only one of the major failures between November 12 and 17 has an official tie to rushed security patching. The others have different stated causes, or none yet: United Airlines has not named a cause for its November 16 outage. But the one that does, plus Microsoft's own account of why its testing missed a bug for four months, gives you enough to fix your change process before your board asks.
What the DNI Said About the NSA Outage
Jay Clayton, the Director of National Intelligence, tied the NSA network failure to patching a PHP vulnerability and ruled out an attack. Speaking on Fox Business on November 17, he said "the patching that has gone on following [a PHP vulnerability] has caused some of these systems to not come back up when you patch them as expected," and that this "is what has happened at the NSA today." He added that "some systems have to hand off to other systems, and those handoffs are not coming as expected," and that the FBI and NSA "were appropriately saying, take it down, get the patch in place." On the question of an attack, he was direct: "We do not see the indicia of a coordinated cyber attack."
Clayton did not name a CVE. Implicator's roundup of the outages reports that he referred to PHP flaws found the previous month, and the NSA network that failed was NSANet, the agency's classified communications network. The NSA itself gave a different account. Per CNBC, it cited "scheduled routine maintenance."
A broader claim comes with weaker sourcing. The Washington Sun reported, citing three anonymous sources, that bank executives and NSA officials told members of Congress the week's outages came from efforts to close gaps AI models had identified, and that the fixes temporarily overwhelmed systems. A congressional official briefed on the matter told the paper: "These models are identifying massive vulnerabilities, and that increases the urgency to patch right away." No named executive has said this on the record.
What Caused Four of the Other Outages
The companies involved blame bugs, hardware and an internal update, and none of them blames a security patch. Here is each stated cause, as of November 18:
- Microsoft 365 (Nov. 12-13): a bug introduced in a July update to networking gateway servers, which Microsoft says lay dormant for months, per BleepingComputer's summary of the preliminary report.
- FAA (Nov. 13): a router at a remote radio facility in El Dorado, Arkansas, flooded the network with connection requests during copper-to-fiber work, and communications and radar failed at 17 of 22 en route centers, per Implicator. FAA Administrator Bryan Bedford said there was "zero evidence of anything AI-related," and Transportation Secretary Sean Duffy said the failure has been recreated but its root cause is not yet clear.
- JPMorgan Chase (Nov. 16): a software bug in on-premise hardware systems, per CNBC.
- Apple (Nov. 17): "an internal update" that "caused some outages," per the same CNBC report.
Each of these was a change that reached production and failed at scale: a hardware install at the FAA, an update at Apple, a months-old code change at Microsoft. Your change process has to survive all of them, whatever turns out to have started this week's run.
Why PHP2Shell Had to Be Patched Fast
PHP2Shell is a pair of PHP flaws that allow unauthenticated remote code execution, and attackers were using them within days of disclosure. Wiz Research disclosed them on October 21 as CVE-2026-109618 and CVE-2026-109619, rated CVSS 9.2. Nearly every PHP version back to 5.4 is affected, and Wiz demonstrated code execution against the then-current WordPress, Drupal and Joomla container images. Wiz says it used Gemini 4 Argon and GPT-5.6 Sol throughout the research, and that a single researcher did in a week what previously would have taken a team months.
The clock after that was short. PHP shipped fixes in 8.5.12, 8.4.27, 8.3.36 and 8.2.35 on October 21. Perplexity's Adel Karimi confirmed working exploitation against honeypots on October 24, CISA added both flaws to its Known Exploited Vulnerabilities catalog on October 26, and federal civilian agencies had until October 29 to fix them. Censys counted more than 32.51 million PHP services still running an affected version. Amazon Threat Intelligence said attackers began targeting the flaws four days before the patches shipped, per BleepingComputer.
So the argument for patching fast is strong, and you should state it plainly to anyone who wants to slow things down. CISA allowed three days from catalog listing to federal deadline here, as it did for Ray's KEV listing in August. And attackers are getting faster at turning a fix into an exploit: Google's threat group has documented attackers building working exploits from patch diffs. Slowing your exploited-flaw patches back to a monthly window is not an option.
Where Fast Changes Break Production
Microsoft's preliminary report shows the three ways a correct change still takes down a fleet: test hardware that differs from production, thin capacity headroom, and a rollback that is slow when you need it. The Microsoft change was an ordinary update, and each of its failure modes applies to an emergency patch, which gives you less time to catch it.
First, the lab did not match production. Microsoft said its "normal load testing did not detect the latent issue because the traffic mix and SKUs used in our labs" differed from production, and that "the contention issue is worsened on machines with more cores and newer hardware," per BleepingComputer. A load test on older, smaller machines passed a change that broke the newer, larger ones.
Second, there was little headroom. About 14% of gateway capacity was already offline for an unrelated reason, and Teams had shifted its traffic onto Exchange Online's network about 15 hours earlier. When the bug hit, "in less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive." More than 80% of servers stopped responding at the low point, and restarted servers were overwhelmed again by backlogged traffic.
Third, rollback was slow. Microsoft's Charles Lamanna said normal rollback could not quickly revert the defect, per Implicator. The outage ran from 11:20 a.m. ET on November 12 to 11:10 a.m. ET the next day.
Clayton's description of the NSA failure, systems that did not come back up and handoffs that did not arrive, is the same pattern you get when many interdependent systems are patched and restarted at once. You have seen a smaller version of it before in GitHub's August outage, when Copilot's retries added load to a failing service.
How to Run a Two-Speed Patch Process
A two-speed patch process has two separate service levels: one for flaws that are known to be exploited or have a public working exploit, measured in days, and one for everything else, kept on your normal cycle. The fast lane stays safe by running a shortened version of every safety step.
The core safety step is a canary. Google's SRE workbook defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation," and says the canary "should be sizeable and last long enough to be representative of the overall deployment." It also warns that you must compare canary signals against a control group, because aggregate metrics can hide errors in a small slice of traffic. In the fast lane the canary runs for hours rather than days.
The fast lane needs four things written down before the next PHP2Shell:
- A written trigger for the fast lane: a KEV listing, a vendor advisory confirming exploitation, or a working public exploit. A high CVSS score alone does not qualify, because AI-assisted research is going to produce a lot of high scores.
- Compensating controls applied before the patch. Wiz notes that Cloudflare, Fastly, Google, Vercel and others shipped mitigations for systems that could not patch at once. A WAF rule or a disabled feature buys you the hours a canary needs.
- Waves ordered by dependency. Patch one tier at a time and confirm its downstream handoffs work before the next tier starts. Never restart every interdependent system in one window.
- A tested rollback for each wave. If a patch cannot be reverted in minutes, the wave size has to shrink to match.
Plan for more of these. Anthropic's Glasswing project raised the gap between AI-speed discovery and human-speed patching in April, and this month a Linux kernel flaw that Claude exploited had no Ubuntu fix yet. A process sized for one emergency a quarter will struggle with one a week.
What to Do Before the Next Critical CVE
This Week:
- Pull the change records for every PHP2Shell patch you applied since October 21. Check how many went out without a canary or staged wave, and whether any caused an incident.
- Ask your SRE lead one question: does our load-test hardware match production core counts and SKUs? If the answer is no, write down which services that applies to.
- Check current capacity headroom on the systems most likely to receive an emergency patch. If a 14% capacity loss would push you near the edge, you are carrying Microsoft's risk.
This Month:
- Write the two-speed policy: the fast-lane trigger, the minimum canary duration, the wave order by dependency, and who can approve skipping a step. Get your CISO and head of infrastructure to sign the same page.
- Run a game day where one emergency patch fails in the canary. Time how long the rollback takes.
- List the compensating controls (WAF rules, feature flags, network blocks) you can apply within an hour for your top internet-facing stacks.
Before Your Next Board Meeting:
- Report time-to-patch for exploited flaws and change failure rate for emergency changes side by side. A board that sees only the first number will push you toward the NSA's outcome.
The Bigger Picture
The industry learned this lesson once already, from a security vendor. On July 19, 2024, a CrowdStrike content update affected an estimated 8.5 million Windows devices, by Microsoft's count. CrowdStrike's response, per its remediation hub, was to commit that updates would be "gradually deployed to larger portions of the sensor base, starting with a canary deployment," and to give customers more control over when updates land. That update was a security change pushed fast to every machine, and the remedy CrowdStrike committed to was staging.
AI-assisted vulnerability research will raise the number of emergency changes you make, and each one needs the same staging. Set the fast-lane SLA in days, keep the canary, and start reporting both numbers this quarter.
Continue Reading
- Google Finds Attackers Turning Patch Diffs Into Working Exploits
- CISA Gave 3 Days to Patch Ray. Auth Is Still Off by Default.
- Linux CVE-2026-98432 Has No Ubuntu Fix After Claude's Root Exploit
- Copilot Retried Into GitHub's Outage. Cap Your Agents.
- Amazon AI Code Outages: The 'High Blast Radius' Problem
- GPT-6 Cyber Tests Found No Wins on Fully Hardened Targets
