1. Hook
Retries don’t fix failures. Bad retries create new failures.
A downstream service starts timing out — not dead, just slow, running at 80% capacity. Every client has a retry policy. So the moment requests fail, every client fires again, at roughly the same moment. Then again. Within seconds, a service struggling at 80% load is facing 300% of its normal traffic — all retries. It collapses.
The outage didn’t kill the service. The retries did.
I watched this happen in a production Rust system I maintain, one that fans out to several downstream APIs. Fixing it taught me more about resilience than any textbook chapter. Here’s why naive retries fail, why exponential backoff with jitter fixes them, and how I built it.
2. The Retry Trap
Three strategies dominate the wild. All three are traps.
Immediate retry
async fn call_with_retry(client: &Client) -> Result<Response, Error> {
loop {
match client.get("https://api.example.com").send().await {
Ok(resp) => return Ok(resp),
Err(_) => continue, // retry instantly
}
}
}The worst offender. If the service is failing from overload, retrying instantly adds more load with zero delay — exactly when it needs less. A 500ms blip becomes a multi-minute outage.
Fixed delay
async fn call_with_fixed_delay(client: &Client) -> Result<Response, Error> {
for _ in 0..5 {
if let Ok(resp) = client.get("https://api.example.com").send().await {
return Ok(resp);
}
tokio::time::sleep(Duration::from_secs(1)).await;
}
Err(Error::MaxRetriesExceeded)
}Better, but subtly broken: every client retries in lockstep, one second later, then one second after that. The pile-up just moves down the timeline. The herd stays a herd.
Infinite retry
The most dangerous. It hides failure instead of surfacing it — no fast failure, no signal to the caller, no signal to an operator. It quietly burns connections, threads, memory, until the retrying process is the next thing to fall over.
The common thread: none of these account for the fact that you are not the only client retrying.
3. The Thundering Herd Problem
Picture 1,000 clients calling one API. At T+0, every in-flight request fails at once — a deploy, a GC pause, doesn't matter.
Fixed 1-second retry delay:
T+0.0s 1,000 requests fail simultaneously
T+1.0s 1,000 retries fire simultaneously → still overloaded, fail again
T+2.0s 1,000 retries fire simultaneously → still overloaded, fail again
T+3.0s 1,000 retries fire simultaneously → still overloaded, fail again
...No clean recovery window — every wave is a synchronized spike, not a trickle. This is the thundering herd problem: many clients, released by one event, converging on the same resource at the same instant. Same phenomenon as cache stampedes and connection pool exhaustion — triggered here by retries.
The fix isn’t “retry less.” It’s “retry at different times.”
4. Retry Smarter
Two techniques, combined: exponential backoff and random jitter.
Exponential backoff
Each retry waits longer than the last, typically doubling:
attempt 1: wait 100ms
attempt 2: wait 200ms
attempt 3: wait 400ms
attempt 4: wait 800ms
attempt 5: wait 1600msMore breathing room for the struggling system. But alone, it still keeps clients synchronized — everyone doubles at the same rate. The herd survives, just slower.
Random jitter
Jitter randomizes the delay so retries from different clients land at different times instead of colliding. Production implementation, in Rust:
use rand::Rng;
use std::time::Duration;
pub struct BackoffConfig {
pub base: Duration,
pub max: Duration,
pub max_attempts: u32,
}
impl BackoffConfig {
/// Full jitter: pick a random delay between 0 and the
/// exponential ceiling. This is what actually breaks
/// synchronization between clients.
pub fn delay_for_attempt(&self, attempt: u32) -> Duration {
let exp = self.base.as_millis() as u64 * 2u64.saturating_pow(attempt);
let capped = exp.min(self.max.as_millis() as u64);
let jittered = rand::thread_rng().gen_range(0..=capped);
Duration::from_millis(jittered)
}
}
pub async fn call_with_backoff<F, Fut, T, E>(
config: &BackoffConfig,
mut operation: F,
) -> Result<T, E>
where
F: FnMut() -> Fut,
Fut: std::future::Future<Output = Result<T, E>>,
{
let mut attempt = 0;
loop {
match operation().await {
Ok(value) => return Ok(value),
Err(err) if attempt >= config.max_attempts => return Err(err),
Err(_) => {
let delay = config.delay_for_attempt(attempt);
tokio::time::sleep(delay).await;
attempt += 1;
}
}
}
}This is “full jitter”, popularized by AWS: instead of a small random offset on a fixed value, pick a uniform random delay between zero and the exponential ceiling. It spreads retries far better than “jitter as a tweak.”
Wired into every outbound call to a flaky dependency:
let config = BackoffConfig {
base: Duration::from_millis(100),
max: Duration::from_secs(10),
max_attempts: 5,
};
let result = call_with_backoff(&config, || async {
downstream_client.fetch_data().await
}).await;Same failure, same 1,000 clients — but no synchronized herd. Retries land at 40ms, 380ms, 1.2s, scattered. The service sees a declining trickle, not a repeated spike, and gets room to recover.
5. Why It Works
Reduced load spikes. Retries spread across a window instead of clustering — peak concurrent load drops, often by an order of magnitude at high fanout.
Faster recovery. Recovery needs a window of reduced load. Jitter creates it with zero coordination between clients.
Better success rates. Fewer retries hit the system while it’s still overloaded, so more succeed on the first attempt after backoff instead of triggering another round.
Improved stability. The underrated one: it decouples client behavior from the server’s failure mode. A well-jittered client behaves reasonably whether the server is healthy, degraded, or down — no separate logic branch needed.
One incident I traced: a dependency degraded during a deploy. With fixed-delay retries, a ~15-second blip stretched into ~4 minutes, as three internal services kept re-synchronizing against each other. With full jitter, the same blip resolved in the original 15–20 second window — the retry traffic never re-formed into a spike big enough to sustain the degradation.
6. Trade-offs
Not free. The costs, plainly:
Increased latency. A request that would’ve failed fast might now wait several seconds across retries. For latency-sensitive paths, keep max tight and max_attempts low — you're trading tail latency for reliability, within a budget.
Choosing retry limits. Retrying forever isn’t resilience, it’s a resource leak with manners. Interactive calls get 2–3 attempts and a short ceiling; background jobs can afford 8–10 with a longer one, since nothing’s waiting synchronously.
Responsiveness vs. stability. Direct tension. Fast retries feel responsive to one client but cause thundering herds. Well-jittered retries protect the system but cost individual callers time. No universal answer — only a deliberate choice based on what’s calling what.
7. Lessons Learned
- Never retry immediately. Zero-delay retry bets that the cause already resolved. Under real load, that bet loses.
- Every retry should have a purpose. Don’t copy-paste a policy. Know the failure mode you’re protecting against.
- Randomness improves reliability. Counterintuitive, but at scale, injected randomness is what prevents accidental synchronization. Determinism per client produces chaos system-wide.
- Retries should respect downstream systems. A policy tuned only for your convenience is a liability wearing a feature’s clothes. Backoff and jitter are good citizenship toward every other client sharing that dependency.
Retries aren’t a resilience feature by default. They’re one only when designed around the failure mode, the herd, and the recovery curve. Exponential backoff with jitter isn’t clever — it’s the floor.