The thing that kept GitHub's worst outage of the year going was a retry loop in the client for GitHub's own AI assistant. On August 17 the platform degraded for 7 hours and 47 minutes across github.com, authentication, Actions, the APIs, pull requests, issues and Copilot. GitHub CTO Vlad Fedorov's post-incident write-up is unusually direct about the amplifier: errors in those services "triggered a client-side retry loop that increased traffic during recovery," and GitHub had to mitigate that behavior before it could safely restore traffic.
Read that again as a statement about your own estate. The most sophisticated engineering organization in developer tooling shipped an AI client that, on encountering errors from a struggling backend, hit it harder. Your coding agents, your CI jobs, your MCP servers and your LLM gateway are all doing the same thing right now, on defaults you did not choose.
What Actually Failed on August 17
The trigger was capacity, not a bad deploy. GitHub's investigation "found that the outage began when traffic reached a new peak, and a critical infrastructure component in our Central US data center failed to scale with it." Per GitHub's status-page incident report, that component was an Istio sidecar that hit its concurrency limit and failed to autoscale because the scaling policy watched the host service and not the sidecar — a monitoring gap rather than a shortage of machines. Four HAProxy nodes then exhausted their flow limits, which degraded the gateway auth path, and from authentication the pressure spread into everything that needs it — which on GitHub is everything.
The blast radius was measured in error rates, not a hard down. Web and API traffic ran at roughly a 20% error rate, while archive downloads and raw repository content sat near 50%. That second number is the one that broke builds: a coin-flip failure rate on raw.githubusercontent.com takes out Docker builds, Go module fetches and any install script that curls a file from a repo. GitHub marked the incident resolved at 9:15 PM UTC, with authentication-dependent services — Copilot among them — recovering last.
The scale numbers in the postmortem explain why the ceiling moved. Monthly commits grew from 1.4 billion in April 2026 to 2.9 billion. GitHub now sees roughly 130 million merged pull requests and 24 million new repositories a month, and Actions runs climbed past 100 million to finish near 115.4 million. Azure now serves about 58% of platform load and half of all Git operations, up from 12% of platform load in May. GitHub added more than 3 million CPU cores and 120 petabytes of high-speed storage. Traffic doubled in four months and the autoscaling did not.
Doubling in four months is a demand curve few capacity models survive. But the interesting failure is not that a component ran out of headroom — it is what the clients did next.
What they did next is measurable, and it is the most useful number GitHub published. The status-page report attributes the amplification to a latent retry bug in VS Code — specifically, the Copilot client re-requesting authentication tokens without pausing when the token service answered slowly or failed. Traffic to the Copilot Token Service went from a normal 7,000-9,000 requests per second to 70,000-100,000, roughly a tenfold amplification, and it was that surge, not the original saturation, that held recovery open. GitHub's escape was blunt: reduce gateway retries by shipping a code change, and block inbound Copilot token requests at the load balancers with a 403. The company had to refuse its own AI assistant's traffic to get the platform back.
A Retry Storm Is an Old Failure With a New Client
A retry storm is what happens when a service starts returning errors and its clients respond by sending more requests, converting a partial brownout into a total outage. This is not a novel failure mode. It is chapter-and-verse in Google's SRE book, which is a decade old and prescribes two distinct controls that most teams conflate into one.
The first is a per-request retry budget: "If a request has already failed three times, we let the failure bubble up to the caller." Three attempts, then stop. Almost everyone has this, because it is the knob every HTTP library exposes.
The second is a per-client retry budget, and almost nobody has it: "Each client keeps track of the ratio of requests that correspond to retries. A request will only be retried as long as this ratio is below 10%." The difference is not academic. Google's own arithmetic puts worst-case request growth — a datacenter accepting a trickle and rejecting most of what it is sent — at "somewhere just below 3X" without the per-client cap, and "just 1.1x in the general case" with it. Three times the load arriving at a component that just failed to scale is precisely the shape of August 17.
The book is also explicit about stacking: "If multiple layers retried, we'd have a combinatorial explosion." A failed request should be retried by exactly one layer — the one immediately above the failure.
The infrastructure world has been converging on the per-client version for a while. Envoy implements a retry budget as a circuit-breaker threshold, with budget_percent defaulting to 20% of the sum of active and pending requests and min_retry_concurrency defaulting to 3. Kubernetes' Gateway API GEP-3388, currently experimental, proposes the same thing as a first-class policy because "too many client-side retries can also exacerbate consistent failures and slow down recovery, quickly overwhelming a failing system and leading to cascading failures such as retry storms."
The control exists. It has existed for years. It sits in your service mesh, and it does not sit in your agent harness.
Your Agents Have a Per-Request Budget, Not a Per-Client One
Every mainstream LLM SDK ships a per-request retry budget on by default and no per-client budget at all. That is the gap the August 17 postmortem should make you go and look at.
The OpenAI Python SDK states it plainly: "Certain errors are automatically retried 2 times by default, with a short exponential backoff," covering "Connection errors, 408 Request Timeout, 409 Conflict, 429 Rate Limit, and >=500 Internal errors." Default request timeout is ten minutes. Anthropic's documented behaviour is the same shape: the official SDKs "automatically retry transient failures (such as connection errors, rate limits, and 5xx server errors) with exponential backoff, twice by default."
Those are reasonable defaults for one client. They are not a fleet control. Nothing in either SDK knows what fraction of your organization's traffic is currently retries, and nothing stops a hundred agent sessions from each independently deciding, at the same moment, that the 429 they just got deserves another go. Then stack the layers: the agent loop retries the tool call, the tool retries the HTTP request, the SDK retries the API call, the gateway retries upstream, and the runner retries the whole job. Four layers of three attempts is eighty-one requests for one user action.
GitHub's own API documentation has been telling integrators this for years. Its REST API best practices say "to avoid exceeding secondary rate limits, you should make requests serially instead of concurrently," that if a retry-after header is present "you should not retry your request until after that many seconds has elapsed," and otherwise to "wait for at least one minute before retrying." The closing line is the one to put in your agent's system prompt: "Continuing to make requests while you are rate limited may result in the banning of your integration."
The limits are published and they are lower than agent workloads assume. A personal access token gets 5,000 requests per hour flat; the higher 15,000 ceiling applies to a GitHub App owned by an Enterprise Cloud organization, not to a token in a developer's shell. Secondary limits cap you at 100 concurrent requests, 900 points per minute for REST endpoints, and 90 seconds of CPU time per 60 seconds of real time. The one that catches CI fleets is GITHUB_TOKEN: 1,000 requests per hour per repository, rising to 15,000 for resources belonging to an Enterprise Cloud account. A handful of agents opening, commenting on and updating pull requests in one busy monorepo will find that ceiling, and the naive response to the resulting 429 is another request.
We have already seen what agent volume does to a repo. Linear's agent teams pushed 65 pull requests a week without anyone getting time back, and Zalando now auto-approves roughly a third of its PRs while agent-written changes got measurably larger. That is API traffic, and it is growing faster than anyone's rate-limit budget.
The SLA Excludes the Service That Broke Longest
Here is the commercial sting: the AI assistant that amplified the outage is not covered by the uptime commitment you pay for. GitHub's Online Services SLA applies to exactly three things — GitHub Actions, GitHub Enterprise Cloud and GitHub Packages. Copilot is not among them.
The rest of the document is worth reading before your next renewal. The commitment is "at least 99.9% Uptime" measured over a calendar quarter, not a month. A quarter is about 2,208 hours, so 99.9% leaves roughly two hours and twelve minutes of Downtime. August 17 alone was three and a half times that. For Enterprise Cloud, Downtime means "a period of time where either (i) the error rate exceeds five percent (5%) in a given minute for a Service feature or (ii) the Service was unavailable" — so a 20% error rate does count, which is better than many enterprise SLAs.
What you get for it is small. Falling into the 99.5%-to-99.9% band earns a 5% service credit; below 99.0% earns 25%. Enterprise Cloud lists at $21 per user per month, so a 500-seat estate is $31,500 a quarter and a 5% credit is $1,575. That is not a remedy, it is a gesture — and you have to request it in writing within 30 days or it evaporates. Meanwhile the SLA explicitly excludes "rate limits, performance degradation, or other latency issues without actual service unavailability," which is a large carve-out in a world where outages present as elevated error rates rather than hard downs.
Price the downtime, not the credit. This is the same lesson as the 19-day vendor outage that forced enterprises to write a real resilience playbook: the contract will not make you whole, so the degraded-mode plan has to.
July Had Eight Incidents. This Is a Trend, Not an Event.
August 17 was the largest of a run, not an outlier. GitHub's own July 2026 availability report documents eight separate incidents: a 7-hour 4-minute event on July 8 that peaked at a 96% error rate, a 9-hour 18-minute Actions degradation on July 9 caused by an overloaded shard, a DNS control-plane failure on July 19, an expired internal certificate that delayed 78.99% of larger-hosted Actions jobs, and a Vitess backfill on July 24 that dropped a backing table and failed 113,930 pull request attempts across 50,904 users.
The May 2026 report listed nine incidents and named the cause of the pressure in GitHub's own words: "traffic is growing rapidly, driven in large part by AI-assisted and agentic development workflows."
That is GitHub's account, and it is worth noting who benefits from it. A vendor explaining a run of outages has an obvious interest in an external, industry-wide cause, and credible analysts do not accept the framing. The Pragmatic Engineer's read is that the growth is not actually extraordinary — Fedorov's own figure is roughly 3.5x over two years — and that peers absorbing comparable AI-driven demand, among them Vercel, Linear, Railway and Sentry, have kept up. On that reading GitHub's problem is substantially self-inflicted: eighteen years of accumulated architectural debt, capacity planning that started late, and a data-center-to-Azure migration running concurrently with the load spike. Take that seriously, because it changes what you should copy: if agent traffic were an irresistible force, nothing here would be actionable. The point of the counter-argument is that the load was survivable and the controls were missing.
For your purposes the distinction matters less than it looks, because the amplifier is not in dispute. Whatever you conclude about GitHub's capacity planning, the thing that turned a regional saturation event into an eight-hour platform outage was a client retry loop — and that client was an AI coding assistant.
Third-party tracking puts the trend in context. IncidentHub, which monitors the public status page, counted 257 GitHub incidents between May 2025 and April 2026, of which 48 were major, totalling 112 hours and 18 minutes with an average time-to-restore of 6 hours 7 minutes. Actions was the worst-affected component at 57 outages; Copilot was second at 44.
Steel-man the other side, because it is strong: retries are correct. Transient failures are the overwhelming majority of failures, an agent run is long and expensive, and failing a 40-minute autonomous task on one 503 is a genuinely worse product. Nobody sensible is arguing for zero retries. The argument is that a per-request cap is a client-level control masquerading as a fleet-level one, and at agent volumes the difference between those two is an outage.
What to Do Before Your Next Brownout
This Week:
- Inventory the retry layers on one path. Pick your busiest agent workflow and count how many independent components will retry a single failed call — agent loop, tool wrapper, HTTP client, SDK, gateway, CI runner. Multiply the attempt counts. If the product is above about ten, you have a combinatorial explosion waiting for a bad afternoon. Collapse it to one retrying layer, immediately above the failure.
- Cap agent concurrency in CI. GitHub Actions has shipped the control: the
concurrencykey ensures "only a single job or workflow using the same concurrency group will run at a time," withqueue: maxallowing up to 100 pending runs. Group agent-triggered workflows so one repository cannot self-DoS its ownGITHUB_TOKENallowance. - Find out what your agent harness does on a 429. Read the config, not the docs. If the answer is "exponential backoff, unbounded," you have GitHub's bug.
This Month:
- Add a per-client retry budget at the gateway. If you already run a service mesh, this is configuration, not code — Envoy's 20%
budget_percentis a defensible starting point, and Google's 10% is the conservative one. If your LLM traffic goes through a gateway, put the budget there so every agent inherits it without a code change. - Emit a retry-ratio metric and alert on it. You cannot budget what you do not measure. The number to graph is retries as a share of total outbound requests per service, per minute. It should be near zero in steady state; a sustained climb is the earliest signal of a brownout you will get. Agent-specific observability platforms like Langfuse will show you the agent loop, but the retry ratio lives in your HTTP layer — instrument it there. If you are still choosing, our monitoring comparison starts from the kill switch rather than the dashboard.
- Write the degraded-mode runbook for an SCM control-plane failure. Specifically: what happens when github.com, Actions and the assistant are all impaired for eight hours. Which deployments can proceed from a mirror? Which agents should be paused rather than left retrying? Who has authority to pause them? The answer "we waited" is what most organizations did on August 17.
Before Renewal:
- Read your SLA against what actually happened. Confirm which services are in scope, whether the measurement window is monthly or quarterly, and what the degradation carve-out excludes. Then calculate the credit for an eight-hour outage and compare it to the loaded cost of your engineering organization sitting idle for a day. Negotiate against that gap, and note that the assistant is likely outside the agreement entirely — as our seat-level comparison of coding assistants found, the AI layer and the platform layer are priced and governed as separate things.
- Make retry behaviour a procurement question. Ask every agent vendor: what is your per-request retry limit, do you implement a per-client retry budget, and do you honour
retry-after? Get the answer in the contract or the security questionnaire. It is a fair question — GitHub just published the consequences of the wrong answer.
The Bottom Line
The remediation list GitHub published is the actual news. Its first commitment is "applying consistent retry limits, retry budgets, and variable timeouts across service-to-service interactions to prevent retry storms and cascading load," alongside a review of lower-priority CPU and memory alerts to find components that fail under sudden spikes. That is a company with world-class infrastructure engineers announcing, in August 2026, that it is only now standardizing a control the SRE literature has recommended for a decade.
Note the scope of that commitment, though: it covers service-to-service interactions, which is not where this outage's amplifier lived. The client-side half appears only in the status-page follow-ups, as "reviewing retry limits and backoff behavior across gateways and clients" and addressing the VS Code retry behavior itself. The blog post commits to fixing the plumbing; the thing that actually held recovery open was an editor extension.
Whether that amounts to negligence is genuinely contested, and the case that it partly does is a real one. But the mechanism is not in doubt: the client population changed underneath the architecture. Human developers retry a few times and then go get coffee — they are a natural rate limiter. Agents do not get bored, do not context-switch, and do not notice that the thing they are hammering is on fire. Every capacity assumption built when the caller was a person needs re-deriving now that the caller is a loop, which is the same structural point the UK's critical-third-parties regime keeps missing and the same reason agent permissions have to be policy rather than prompts.
GitHub got the postmortem and the fix. You got the warning for free.
Your agents will not stop retrying because you asked them nicely. Give them a budget.
Continue Reading
- Agent Teams Hit 65 PRs a Week. Nobody Got Time Back.
- Zalando Auto-Approves a Third of PRs. Agents Made Them Bigger.
- Best AI Agent Monitoring: Langfuse, Then a Real Kill Switch
- An AI SRE Agent Fixed 35,000 Incidents While Engineers Slept
- 19 Days Dark: How a Shutdown Broke Enterprise AI's Vendor Myth
- Claude Code Stops Asking Aug 14. Prompts Aren't Policy.
- Best AI Coding Assistant at 500 Seats: Buy Copilot Business