What is Exponential Backoff and Jitter ?

Five services deep. Three retries at every layer. When the database at the bottom starts failing, it sees 243 times its normal load. That number comes from Marc Brooker's Amazon Builders' Library paper on timeouts and retries. It's also the shortest explanation I know for why an outage that should clear in ninety seconds can still be running four hours later.

Retry logic looks like the safest code you'll ever write. It's usually the code that turns a blip into an incident.

A retry is one client deciding its request matters more than everybody else's. Brooker's word for this is "selfish", and that's the right frame. The client spends more of the server's time to buy itself a better chance of success.

When failures are rare, nobody notices. A dropped packet, a pod restarting, the second attempt works and the extra load rounds to nothing.

The trouble is that failures and load are correlated. Servers rarely break at random. They break because they're saturated: thread pool full, connections exhausted, GC thrashing. Every client sees an error in the same second, and every client answers by sending the request again. Your safety mechanism just became the traffic spike.

Timeouts sharpen it. Set a 200ms timeout on a service whose p99 latency, the slowest 1 percent of calls, has drifted to 400ms, and you turn slow successes into fresh requests. The original work keeps running, burning capacity nobody is waiting on anymore.

There's a name for what happens next. A metastable failure is one where a trigger pushes the system into overload, and a feedback loop then holds it there after the trigger is gone. Remove the cause and the system stays down.

Researchers from Penn State, the University of New Hampshire and Twitter studied this in Metastable Failures in the Wild, published at USENIX OSDI in July 2022. They pulled 22 metastable failures out of public incident reports across 11 organizations. Three of their findings should bother anyone who writes a retry loop.

At least 4 of the 15 major AWS outages in the decade before the paper were metastable. And retries are the single most common sustaining effect, showing up in more than half the incidents they studied. Outages in their sample ran from 1.5 hours to 73.53 hours.

The AWS US-EAST-1 event on 7 December 2021 is the textbook version. An automated capacity scaling activity caused a surge of connections that overwhelmed the networking devices between two internal networks. In Amazon's own words, the resulting delays led to "even more connection attempts and retries" and "persistent congestion and performance issues".

The clients had backoff logic. A latent bug stopped it from kicking in. Recovery took roughly seven hours.

Exponential backoff is the obvious fix and every SDK ships it: wait 100ms, then 200ms, then 400ms, doubling until you hit a cap. The cap matters, because 2^12 seconds is more than an hour of waiting and no user is sticking around for that.

What it doesn't fix is the shape of the traffic. If a thousand clients fail in the same millisecond, they all wait 100ms, and they all wake up in the same millisecond. Backoff spaced the waves apart. It did nothing about the waves.

Brooker put it plainly in the AWS Architecture Blog post Exponential Backoff And Jitter, published 4 March 2015: "there are still clusters of calls. Instead of reducing the number of clients competing in every round, we've just introduced times when no client is competing."

Backoff without randomness turns one continuous overload into a series of synchronized thundering herds. The server gets idle gaps it can't use and spikes it can't absorb.

Jitter means adding randomness to the wait so clients stop waking up together. Brooker's post is still the best public comparison of the ways to do that. He simulated clients contending for a remote database over a network with 10ms mean delay and measured both the total work done and the time to finish.

Full Jitter throws the schedule away and picks a random point anywhere inside the window. Equal Jitter holds half the wait fixed and randomizes the rest, which feels safer and performs worse. Decorrelated Jitter ignores the attempt counter entirely and grows from the previous sleep, so it climbs back to normal speed faster once the server recovers.

Brooker's result after all that measurement: Full Jitter did the least work, Decorrelated Jitter finished slightly sooner, and plain exponential backoff lost on both axes. His recommendation was that jittered backoff "should be considered a standard approach for remote clients".

Take Full Jitter if you take nothing else. It's one line, it's the easiest of the three to reason about at 3am, and I've never regretted picking it over the cleverer options.

Backoff controls when a single client retries. It says nothing about how many retries the system as a whole is allowed to make, and that's the number that decides whether you recover.

Google's Site Reliability Engineering book works through this in its chapter on cascading failures. Their example stacks a JavaScript layer, a frontend and a backend, each retrying 3 times on top of its original call. That's four attempts per layer across three layers, so one user action becomes 64 attempts on the database. The fix they suggest is a server wide budget, such as 60 retries per minute per process, after which requests just fail.

A token bucket is the same idea with better ergonomics. gRPC's retry throttling gives each client a bucket, say 10 tokens with a ratio of 0.1.

Every failed call removes a token and every success adds 0.1 back. Once the count drops below half the maximum, retries pause until successes refill it. AWS shipped this same pattern into its SDK in 2016.

Budgets work because they self limit under exactly the condition you care about. When the dependency is healthy, successes keep the bucket full and every retry goes through. When it's down, successes stop, the bucket drains in seconds, and your clients go quiet without anyone paging anyone.

A circuit breaker is the layer above retries. Once the error rate crosses a threshold it stops calls to that dependency entirely, fails fast for a cooldown period, then lets a single trial request through to test the water.

It's the standard answer, and Amazon is openly lukewarm on it. Brooker's position is that circuit breakers "introduce modal behavior into systems that can be difficult to test, and can introduce significant addition time to recovery", which is why Amazon reaches for the token bucket first. A breaker that trips on a false signal takes a healthy dependency offline for every caller at once.

Netflix, who popularized the pattern with Hystrix, put that library into maintenance mode and now point people at resilience4j and at adaptive concurrency limits that react to live latency instead of a threshold somebody guessed at once and never revisited. My rule: a breaker for the one dependency you're willing to shed, a retry budget for everything else.

Sources:

→ Find me on : Social Links

That’s it for today, keep learning!
Scortier, Signing Off!