Removing the Cause Doesn’t Fix It

At 2:19 on a Sunday morning in September 2015, a network link inside Amazon’s US-East region misbehaved for a moment. Just a moment. Amazon’s own write-up calls it a brief network disruption, and it was over almost as soon as it started.

DynamoDB in that region was in trouble until 7:10am.

The part that should bother you is in the middle. A few hours in, the engineers tried to add capacity to the overloaded metadata service, the obvious move and the one your incident channel would suggest. They couldn’t. The service was so busy answering retries that it wouldn’t accept the administrative request that would have made it bigger. So they did the opposite instead: at 5:06am they paused requests to the metadata service entirely. The retries stopped, the capacity change went through, and by 7:10 it was over (AWS).

So the cause lasted seconds. The outage lasted most of a morning. Adding capacity failed, and the thing that worked was taking all the load away.

That isn’t bad luck and it isn’t an Amazon problem. It’s a named failure mode with a small, good literature behind it, and the first thing that literature does is stop describing your service as having one capacity.

Two stable states, not one

The framing comes from a short 2021 paper by Nathan Bronson and three co-authors, built from years of failures observed in production at Facebook. It’s seven pages and worth all of them (Bronson et al.).

Their claim is that a system like this has three regions, not two. Stable, where any trigger you throw at it heals by itself. Metastable, where the system is overloaded and stays overloaded after the trigger is gone, because something inside it is now generating the overload. And in between, vulnerable: perfectly healthy, serving everything, meeting its SLO, and one bad thirty seconds from the second one.

Here’s the uncomfortable part: most production systems run in the vulnerable region deliberately, because it’s much cheaper than the stable one.

Take their worked example, which I’ll use for the rest of this post. A database tops out at about 300 queries a second. Put a look-aside cache in front at a 90% hit rate and the service handles 3,000 requests a second, because only one in ten reaches the database. Ten times the throughput for the price of some memory.

Nobody would call that a mistake. I wouldn’t, and I’ve made it more than once.

But look at what you built. Your capacity now rests on a property of the environment rather than a property of the machines, and that property can go away in one second. It’s supercooled water: liquid at minus five, right up until you tap the glass, and then warming it back to minus one won’t make it liquid again.

Which raises the only question that matters. If minus five isn’t safe, what is?

The number that breaks you and the number that fixes you

A year later a larger group went looking for this pattern in public incident reports, and along the way they wrote down the formula (Huang et al.). Their Theorem 2 says the load you can hold and still recover from any trigger is:

                      capacity
    C_stable  =  ─────────────────────
                 worst-case work amplification

That’s a division, not a subtraction, and the difference between those two operations is the entire post. Capacity planning is subtraction: we can do 3,000, let’s run at 2,400 and keep 20% headroom. Metastability is division: we can do 3,000, our cold-cache amplification is 10×, so the level we can actually recover from anything at is 300.

Three hundred. A tenth of what the service does on a boring Tuesday, and a number nobody wrote down, because nobody computes it.

So the shape to hold in your head isn’t a cliff, it’s two thresholds with a gap between them. Push the slider past 3,000 and the system tips. Pull it back to 2,900, where it was entirely happy ninety seconds ago, and watch it stay down.

The load that breaks you isn’t the load that fixes you

Interactive

Offered load2,400 req/s

Triggernone yet

2,400req/s servedVulnerable, which is where almost everything in production lives, because it’s the efficient place to live. Nothing is wrong. A 300 req/s database is serving 2,400 req/s thanks to a 90% hit rate. Now press the blip.

The two thresholds are computed, not drawn by eye: the tipping point is the capacity a 90% hit rate buys you (300 × 10 = 3,000), and the recovery point is that capacity divided by the cold-cache amplification (3,000 ÷ 10 = 300), which is Theorem 2 of Huang et al. Source: author’s illustration, using the worked example shared by Bronson et al. (HotOS 2021) and Huang et al. (OSDI 2022). The 300 req/s served while broken is generous to the system: in the real failure timeouts stop the application writing results back, so goodput is lower than this.

The blip button is the more interesting control. It doesn’t need you anywhere near 3,000: anywhere above 300 a thirty-second cache wipe is fatal, and 301 requests a second is a tenth of what the service was serving without complaint. The whole vulnerable band, which is to say everything you have ever run in production, is somewhere a brief and completely ordinary event is unrecoverable.

Once you’ve seen those two numbers separately, a sentence you have definitely read in an incident channel stops making sense.

“But the network is healthy now”

On 21 April 2011, during a routine capacity upgrade, AWS shifted traffic off one of the redundant routers on the primary EBS network. The shift went to the wrong place: instead of the other primary router, the traffic landed on the low-capacity replication network, which couldn’t carry it. A large number of EBS nodes lost contact with their replicas.

Then somebody noticed and rolled the change back. Read what happens next in Amazon’s words: when the traffic shift was rolled back and network connectivity was restored, the nodes rapidly began searching the cluster for space to re-mirror their data, exhausted the cluster’s free capacity, and got stuck in a loop, continuously searching (AWS).

The re-mirroring storm didn’t start when the mistake was made. It started when the mistake was fixed.

By the end, roughly 13% of the volumes in that zone were stuck, and getting capacity back into the cluster meant physically relocating spare servers from elsewhere in US-East and installing them. The survey above clocks the whole event at 66.7 hours. The trigger lasted a few minutes.¹That 66.7 hours is Huang et al.’s figure. Amazon’s own timeline is staged and harder to reduce to one number, but the first stabilising action, disabling create-volume calls in the affected zone, came at 2:40am.

Ten years later it happened again in a different shape. On 7 December 2021 an automated scaling activity provoked a surge of connection attempts inside AWS’s internal network, which congested the devices joining it to the main one, which raised latency and errors, which produced more attempts and retries. By 9:28am the team had moved internal DNS traffic off the congested paths and DNS errors had fully recovered. Amazon is blunt about what that bought them: it improved several services by reducing load, but did not fully resolve the impact or eliminate the congestion (AWS). The devices didn’t fully recover until 2:22pm, five hours later.

Bronson’s paper names the difficulty exactly: the outage gets blamed on the trigger, but the real root cause is the sustaining effect. The trigger is the thing you can see, name and roll back, so it’s the thing you fix. The sustaining effect is invisible, has no owner, and is holding the door shut.

Which raises the obvious question: what is holding the door shut, usually?

The loop is nearly always retries

Huang and his co-authors went through hundreds of public incident reports looking for the pattern. They confirmed twenty-one across eleven organisations, plus one at Twitter they could see from the inside. Two findings are worth carrying around.

The first: at least four of the fifteen major AWS outages of the previous decade were metastable failures. Not a bug that got fixed or hardware that got replaced. The system’s own response to a problem that had already gone away.

The second: retries were the sustaining effect in eleven of the twenty-one. More than half.

And here I want to be careful, because the obvious reaction is “fine, we have exponential backoff”. So did Amazon. From the December 2021 postmortem: their networking clients have well tested request back-off behaviour designed for exactly this, and a latent issue prevented those clients from backing off adequately. The code path had been in production for many years. The scaling activity triggered a behaviour nobody had seen before.

That’s the detail I’d take to a design review. The backoff existed. It was tested. It had worked every day for years. The one time it was load-bearing, a path nobody had exercised stopped it firing, and there was no alarm for “backoff is not backing off” because why would there be.

Meanwhile Microsoft’s architecture guidance ships the antipattern as a code sample: a while(true) loop that retries immediately and for ever, with the note that retrying indefinitely is pointless because requests only stay valid for a short window (Microsoft). It’s in the docs because people write it.

The reason to care isn’t hygiene, it’s arithmetic. Your retry policy is a dial on the recovery threshold, and it’s the one dial you fully control:

Your retry cap is your recovery threshold

Interactive

Retry policy3× worst case

1,000req/s recovery pointA 3,000 req/s service with this policy has to come back down to 1,000 before it can recover on its own. Not 2,900. Not a 20% haircut. 1,000.

One division, 3,000 ÷ attempts, which is Theorem 2 of Huang et al. applied to the retry policy alone. The 3-second timeout and the 4-retry cap are the ones in their reproduction, which used a three-replica MongoDB cluster on EC2. Source: author’s illustration of the arithmetic in Huang et al. (OSDI 2022); the observation that an uncapped policy leaves no stable region at all is theirs.

Two retries means three attempts means you can never amplify past 3×, so a third of your capacity is ground you can stand on. No cap at all means there is no such ground anywhere, at any load. Huang et al. say it plainly: an uncapped retry policy leaves the system with no stable region.

None of this is an argument against retries, and it isn’t the argument about queues and goodput, which I’ve made elsewhere and won’t repeat. It’s narrower: retries are a work amplification factor, work amplification is a divisor, and you should know what yours is.

Sometimes, though, the loop has nothing to do with retries at all.

When the loop isn’t retries

Go back to the cache. There are no retries in this story and it still can’t climb out.

With a look-aside cache the application writes results back. So when the database slows, the application times out, and a request that times out never populates the cache. The hit rate stays at zero because the database is slow, and the database stays slow because the hit rate is zero. Nothing is retrying and nothing is amplifying anything. The loop is two facts holding each other up.

Bronson’s paper has the sharpest observation I’ve read about this, and it’s architectural. In that state, filling the cache matters more than serving clients, and a look-aside cache cannot express that preference: the only component that can fill it is the one that has given up. A read-through cache can, trivially. Give the cache a permissive timeout of its own, and the application still gives up on the user’s request while the cache still gets filled, the hit rate climbs, and the system walks itself out.

Same data, same database, same failure. One of them can recover and the other can’t, and the difference is which component owns the write.

Garbage collection does it too, and the best-documented case is the Twitter one in the survey, because the authors had the internal metrics. During a routine peak load test on a mature, well-tuned service, the success rate started falling. Operators shed load at the 83-minute mark and again at 106. In the window after that second shed the load was more than 20% below where it had been at 40 minutes, when everything was fine, the queue was more than 50% longer, GC was busier, and the success rate was still dropping. The service came back when they restarted it and not before.

Load below the level it had been healthy at. Still dying. That’s hysteresis, measured, in production, by people watching it happen.

The general rule I’d write down is this: the sustaining effect is whatever your system does more of when it’s unhappy. Retries, error logging, stack traces, DNS lookups on the failure path, leader elections, rebalancing, garbage collection, connection churn. All of those are cheap in steady state, and that’s exactly why nobody budgets for them.

And it gets worse, because the most dangerous version of this isn’t a loop you failed to notice. It’s one you built on purpose.

The fix for the last one causes the next one

In June 2014 a power loss took out several SimpleDB storage nodes at once. Survivable. The knock-on wasn’t: the lock service got hammered de-registering them, handshake latencies rose, and healthy nodes started blowing through their handshake timeout. After a few failed handshakes, storage and metadata nodes removed themselves from the cluster, and then couldn’t rejoin, because rejoining needs authorisation from the metadata nodes, which had removed themselves for the same reason. Recovery meant manually raising the timeout and restarting things (AWS).

Now the aftermath, which Huang et al. recount in a section they call fix to break. The engineers concluded that giving up was what had botched the recovery, so servers should keep retrying the lock service rather than backing out. A completely reasonable reading of that incident, and also one that makes the amplification factor unbounded.

About a year later, the DynamoDB event this post opened with: storage nodes that did not back out of retrying for their membership data, sustaining exactly the overload they were waiting on.

The other example is gentler and I find it more human. Spotify had a load-spike incident in April 2013 sustained by retries. To understand it, the engineers added significant logging to the error path. In the June incident that followed, the extra logging made each retried request more expensive, and the cost of failing became part of why it kept failing.

They instrumented the failure path so they’d understand the next failure. The instrumentation was the next failure.

None of those decisions were careless. Every one made the system better at the thing that had just gone wrong. What none of them did was compute what the change had done to the divisor, because the divisor isn’t a thing most teams have a name for.

So let’s give it one and go and measure it.

Measure the second number

Start with a multiplication you can do at a desk in ten minutes.

List every mechanism that produces more work when things go badly and write down its worst-case factor. Retries: attempts per request, including the first. Cache: one over one minus the hit rate, because a cold cache sends every request to the back end. Failover fanout: replicas tried before giving up. Client-side re-issues, if your callers are browsers or mobile apps, which they are.

Multiply them, then divide your capacity by the result.

Work out the number you’d have to shed to

Interactive

Cache hit rate90% → 10× on a cold cache

Attempts per request1 try + 2 retries

Documented cases30× combined

Comfortable3,000

Recovers below100

3% of yesterdayA 90% hit rate and 3 attempts multiply out to 30× worst-case work amplification, so a service comfortable at 3,000 req/s has to get down to 100 before it recovers on its own. That is the number your runbook needs and almost certainly doesn’t have.

The two amplifications multiply rather than add, which is why a good cache and a reasonable retry policy together are worse than either sounds. Source: author’s illustration of Theorem 2 in Huang et al. (OSDI 2022), applied to the 3,000 req/s service from their worked example. The 99% preset is theirs too: they note that losing a cache at that hit rate is a 100× amplification.

Multiply, not add, is the bit that gets people. A 90% cache is 10×. A modest three-attempt retry policy is 3×. Together they’re 30×, which turns a service comfortable at 3,000 into one that recovers below a hundred. If your incident plan says “shed load until it recovers”, this is the number it should be quoting, and “shed 20%” is not in the same postcode.

The honest caveat: that multiplication is an upper bound, not a measurement. The real threshold depends on the size of the trigger and how long it lasts, and those interact in ways arithmetic won’t give you.

Which is why the survey’s authors open-sourced their reproductions, and why their experiment is the most useful thing in the paper. Three MongoDB replicas on EC2, a client holding a steady 6,200 successful requests a second, a 3-second timeout and up to four retries. They squeezed the primary’s CPU and watched:

Trigger	Result
78% CPU cut, 10 seconds	Brief latency bump, full recovery
80% CPU cut, 10 seconds	Never recovers. 20,000 req/s attempted, goodput down ~90% to 600
80% CPU cut, 9 seconds	Brief latency bump, full recovery
80% CPU cut, 10 seconds, at 4,200 req/s	Shrugs it off

Two percentage points of CPU, or one second of duration, is the whole difference between a blip and an outage. And the last row is the one to take home: the trigger that killed the system at 6,200 was a non-event at 4,200. No amount of reasoning would have produced that number. They had to go and hit it.

The last practical thing is what to watch. Both papers converge on the idea of a characteristic metric: one that moves when the trigger hits and only comes back after the failure resolves. Cache hit rate. Queueing delay. Timeout rate. Consumer lag, if you live in a streaming pipeline, which I mostly have.

That’s a different alert from the ones you have. Your dashboards tell you when something started. This one tells you whether you’ve come back, which during a metastable failure is the only question anyone in the channel actually wants answered.

What I’d do about it

Write down the multiplication. Retry attempts, times one over one minus your hit rate, times failover fanout. One number, in the design doc, beside the capacity it divides.
Put the quotient in the runbook. In the same units as your dashboard, so at 3am somebody reads “shed to 300” instead of arguing about percentages.
Cap every retry, and know what the cap multiplies to. Uncapped anywhere means no stable region anywhere.
Prefer the architecture that can prioritise its own recovery. Read-through over look-aside is one word in a design doc and the difference between a system that climbs out and one that doesn’t.
Load-test the way back, not just the way in. Push past the knee, take the load away, and record where it actually recovers. That number is the deliverable.
Call the sustaining effect the root cause. There are many triggers and usually one loop, so fixing the loop is the only work that prevents the next one.

There’s a line in Bronson’s paper, quoted as a vice president’s plea to an engineer, that I haven’t been able to shake: can you predict the next one of these, rather than explain the last one?

Mostly we can’t, and the reason is that it’s the wrong first question. You don’t have to predict which blip will get you. You have to know, in advance and in writing, the number you’d have to shed to if one did.

Your monitoring will tell you the moment the trigger is gone. Nothing you own will tell you the moment you’re coming back.

Sources

Papers

Bronson, N., Aghayev, A., Charapko, A. and Zhu, T. Metastable Failures in Distributed Systems, HotOS, 2021.
Huang, L., Magnusson, M., Bangalore Muralikrishna, A., Estyak, S., Isaacs, R., Aghayev, A., Zhu, T. and Charapko, A. Metastable Failures in the Wild, OSDI, 2022. Reproductions at github.com/lexiangh/Metastability.

Incident reports and documentation

Amazon Web Services. Summary of the Amazon EC2 and Amazon RDS Service Disruption in the US East Region, April 2011.
Amazon Web Services. Summary of the Amazon SimpleDB Service Disruption, June 2014.
Amazon Web Services. Summary of the Amazon DynamoDB Service Disruption and Related Impacts in the US-East Region, September 2015.
Amazon Web Services. Summary of the AWS Service Event in the Northern Virginia (US-EAST-1) Region, December 2021.
Microsoft. Retry Storm antipattern, Azure Architecture Center.

Distributed systemsReliability

Cite this post

@article{ghosh2025removing,
  title = {Removing the Cause Doesn’t Fix It},
  author = {Ghosh, Krish},
  journal = {krishghosh.com},
  year = {2025},
  month = {March},
  url = "https://krishghosh.com/writing/removing-the-cause"
}