Software Development

Metastable Failures and Retry Storms: When Your Resilience Code Is the Outage

A short slowdown should end when the cause ends. Sometimes it does not, because the safety net keeps the system down.

When Safety Code Becomes the Problem

Retries look harmless. A call fails, so the client tries again, and most of the time that works. However, under heavy load the same habit can turn a five-second hiccup into an hour-long outage. In this article, we will see how that happens, why it is so common, and which small changes stop it. Moreover, we will test the idea with a simulation of our own.

What a Metastable Failure Is

A metastable failure is an outage that keeps itself alive. First, something triggers trouble, such as a load spike or a capacity drop. Then a sustaining effect takes over, so the system stays broken even after the trigger is gone. Bronson and colleagues introduced this framework in 2021, and the OSDI 2022 follow-up gave it a wider test. In other words, fixing the trigger does not fix the outage.

How Common Is It?

The follow-up study looked at public incident reports. The authors found 22 metastable failures from 11 organizations, and at least 4 of 15 major AWS outages in the last decade were of this kind. Most importantly for us, the retry policy was the most common sustaining effect, affecting more than half of the studied incidents.

Finding (Huang et al., OSDI 2022)Value
Metastable failures studied22, from 11 organizations
Major AWS outages in the last decade that were metastableAt least 4 of 15
Incidents sustained by the retry policyMore than 50%

Why Retries Fail When You Need Them

Marc Brooker writes in the Amazon Builders’ Library that retries are “selfish”: a client spends more of the server’s time to raise its own chance of success. Nobody coordinates the total. Consequently, when the server is slow, every client adds work at the worst moment.

Layers make it worse. The same article notes that distributed systems often have several layers, and each layer can retry. With three retries per layer and three layers, one user request can become up to 4 x 4 x 4 = 64 attempts at the bottom. That is simple arithmetic, yet it surprises many teams.

See It in a Simulation

To make this concrete, we wrote a small Python simulation. A server normally handles 1,000 requests per second, and clients send 700. For ten seconds, capacity drops to 400. Clients give up after one second. The server works through its queue in order, and it cannot tell that a client has already left. Finally, capacity returns to normal at second 20. We compared three clients.

 own simulation, a simplified model (single server, FIFO queue, fixed arrivals). It illustrates the mechanism and is not a benchmark of any real system.

The chart tells the story clearly. The capacity dip hits all three clients. After the trigger ends, however, only the client with unlimited retries stays at zero for the rest of the run.

same simulation. “Load” counts requests arriving at the server per second, including retries.

Here is the reason. Retries pushed the arriving load from 700 to 2,800 requests per second, which is four times more. The server then spends all its time on requests whose clients have already given up. As a result, it never catches up.

Client policyPeak loadBack to 690+/sAverage goodput
No retries700/sSecond 28583/s
3 retries, no limit2,800/sNot within 90 s89/s
3 retries, 10% budget849/sSecond 33546/s

How to Stop the Spiral

Limit Retries With a Budget

The Builders’ Library describes a token bucket for retries: retries run while tokens last, then slow to a fixed rate. It also notes that AWS added this to its SDK in 2016. Our 10% budget works the same way, and it recovered at second 33.

Add Backoff and Jitter

Next, spread retries out. Exponential backoff with a cap increases the wait between attempts, and jitter randomizes it so clients do not retry together. Still, backoff alone does not cap total volume, so use it together with a budget.

Shed Load and Retry in One Place

The study’s own recovery advice is to end the overload through load shedding, more capacity, or lower amplification. Likewise, retry at one layer only, and retry only safe calls. The Builders’ Library says calls with side effects are not safe to retry unless they are idempotent. It also warns that circuit breakers add modal behavior that is hard to test, which is why Amazon prefers local retry limits.

Quick checklist

  • Do your retries have a budget, not just a maximum count?
  • Do you use capped exponential backoff with jitter?
  • Does only one layer retry?
  • Can the server drop requests whose clients already left?
  • Are retried calls idempotent?

What We Have Learned

In short, a metastable failure keeps going after its trigger ends, and retries are its most common fuel, appearing in more than half of the incidents in the OSDI study. In our simulation, unlimited retries quadrupled the load and kept goodput at zero, while a simple 10% budget let the system recover. Therefore, treat retry code as load you are adding, not as free safety. Budget it, spread it, and let the server shed what nobody is waiting for.

Eleftheria Drosopoulou

Eleftheria is an Experienced Business Analyst with a robust background in the computer software industry. Proficient in Computer Software Training, Digital Marketing, HTML Scripting, and Microsoft Office, they bring a wealth of technical skills to the table. Additionally, she has a love for writing articles on various tech subjects, showcasing a talent for translating complex concepts into accessible content.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Back to top button