1. External Services Eventually Fail

I run a blockchain indexer that publishes millions of events to Redis daily — block confirmations, transaction updates, contract events. For months, it just worked. Then one afternoon, Redis had a ninety-second network blip.

My indexer didn’t survive ninety seconds.

Every publish call blocked. Threads piled up behind threads. Event processing — which had nothing to do with Redis — ground to a halt because it was stuck waiting on a publisher waiting on a socket that was never coming back in time.

That incident wasn’t about blockchain. It wasn’t even about Redis. It was the oldest lesson in distributed systems: dependencies fail. Not if — when. You don’t control that. You control what happens next.

This article isn’t about indexing blocks. It’s about the pattern I reached for after: the Circuit Breaker. Here’s how I built one in Rust to make my Redis publisher self-healing instead of self-destructing.

2. Why Retries Aren’t Enough

Retry the failed call — that’s the first instinct, and usually the right one. Networks are flaky. Most failures are transient. But retries alone aren’t a strategy. They’re a tactic. And tactics without strategy break under real failure.

Infinite retries. No cap means a genuine outage becomes an infinite loop. The task never gives up, never fails loudly, never frees itself for anything else.

Retry storms. Now multiply that by every concurrent request. Redis doesn’t go down for one caller — it goes down for all of them, at once. Synchronized retries turn a partial outage into a self-inflicted DDoS on your own dependency.

Cascading failures. This is what actually took my system down. Blocked threads don’t fail quietly — they hold resources. Connection slots, OS threads, memory. As those run out, unrelated parts of the system start failing too, because nothing can get a thread. A Redis blip becomes a full outage of a service Redis was only a small part of.

Retries answer “what do I do about one failed call?” They don’t answer “what do I do when a dependency is systematically down?” For that, you need something tracking failure over time — a circuit breaker.

3. What Is a Circuit Breaker?

Same idea as the one in your house. It doesn’t fix a short circuit — it stops current from flowing, protecting everything downstream. A software circuit breaker does the same for a failing dependency: stop sending it requests, protect the rest of the system.

Three states.

Closed — Normal Operation

Requests flow through. The breaker counts failures and successes in the background. Stay under threshold, nothing changes.

Open — Fail Fast

Cross the threshold, the circuit trips. No more calls to the dependency — every request gets rejected instantly. No network call, no timeout, no blocked thread. This is the shift that matters: slow failure (waiting out a timeout) becomes fast failure (an instant no your caller can act on).

It feels wrong the first time — giving up without trying. But you already know it’s failing. Trying again gathers no new information. It just wastes resources and adds load to something trying to recover.

Half-Open — Test the Waters

After a cooldown, the breaker doesn’t just flip closed and hope. It lets a few test requests through. Succeed, and it assumes recovery, closes, resumes normal traffic. Fail, and it trips back open for another cooldown.

This cycle is what makes it self-healing. No one has to notice the outage and flip a switch — the system probes itself and recovers on its own.

4. My Implementation: Circuit-Protected Redis Publishing

A simplified version of the breaker guarding my Redis publisher. I kept the state machine explicit rather than pulling in a crate — the core logic is small enough that owning it beats a dependency.

use std::sync::atomic::{AtomicU32, AtomicU64, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};

#[derive(Debug, Clone, Copy, PartialEq)]
enum CircuitState {
    Closed,
    Open,
    HalfOpen,
}

struct CircuitBreaker {
    state: Arc<std::sync::RwLock<CircuitState>>,
    failure_count: AtomicU32,
    success_count: AtomicU32,
    failure_threshold: u32,
    success_threshold: u32,
    open_until: Arc<std::sync::RwLock<Option<Instant>>>,
    cooldown: Duration,
}

impl CircuitBreaker {
    fn new(failure_threshold: u32, success_threshold: u32, cooldown: Duration) -> Self {
        Self {
            state: Arc::new(std::sync::RwLock::new(CircuitState::Closed)),
            failure_count: AtomicU32::new(0),
            success_count: AtomicU32::new(0),
            failure_threshold,
            success_threshold,
            open_until: Arc::new(std::sync::RwLock::new(None)),
            cooldown,
        }
    }
    /// Called before every Redis publish attempt.
    fn allow_request(&self) -> bool {
        let current_state = *self.state.read().unwrap();
        match current_state {
            CircuitState::Closed => true,
            CircuitState::Open => {
                let open_until = *self.open_until.read().unwrap();
                if let Some(deadline) = open_until {
                    if Instant::now() >= deadline {
                        self.transition_to(CircuitState::HalfOpen);
                        true // allow a single probe request through
                    } else {
                        false // still cooling down - fail fast
                    }
                } else {
                    false
                }
            }
            CircuitState::HalfOpen => true, // allow limited probes
        }
    }
    fn record_success(&self) {
        let current_state = *self.state.read().unwrap();
        match current_state {
            CircuitState::HalfOpen => {
                let successes = self.success_count.fetch_add(1, Ordering::SeqCst) + 1;
                if successes >= self.success_threshold {
                    self.reset();
                }
            }
            CircuitState::Closed => {
                self.failure_count.store(0, Ordering::SeqCst);
            }
            CircuitState::Open => {}
        }
    }
    fn record_failure(&self) {
        let current_state = *self.state.read().unwrap();
        match current_state {
            CircuitState::Closed => {
                let failures = self.failure_count.fetch_add(1, Ordering::SeqCst) + 1;
                if failures >= self.failure_threshold {
                    self.trip();
                }
            }
            CircuitState::HalfOpen => {
                // One failure during probing sends us right back to open.
                self.trip();
            }
            CircuitState::Open => {}
        }
    }
    fn trip(&self) {
        self.transition_to(CircuitState::Open);
        *self.open_until.write().unwrap() = Some(Instant::now() + self.cooldown);
        self.failure_count.store(0, Ordering::SeqCst);
    }
    fn reset(&self) {
        self.transition_to(CircuitState::Closed);
        self.failure_count.store(0, Ordering::SeqCst);
        self.success_count.store(0, Ordering::SeqCst);
        *self.open_until.write().unwrap() = None;
    }
    fn transition_to(&self, new_state: CircuitState) {
        *self.state.write().unwrap() = new_state;
    }
}

The publisher checks the breaker before touching the network. A rejected request costs a branch, not a socket.

struct RedisPublisher {
    client: redis::Client,
    breaker: CircuitBreaker,
}

impl RedisPublisher {
    async fn publish(&self, channel: &str, payload: &[u8]) -> Result<(), PublishError> {
        if !self.breaker.allow_request() {
            return Err(PublishError::CircuitOpen);
        }
        let result = self.try_publish(channel, payload).await;
        match &result {
            Ok(_) => self.breaker.record_success(),
            Err(_) => self.breaker.record_failure(),
        }
        result
    }
    async fn try_publish(&self, channel: &str, payload: &[u8]) -> Result<(), PublishError> {
        let mut conn = self.client.get_async_connection().await
            .map_err(PublishError::Connection)?;
        redis::cmd("PUBLISH")
            .arg(channel)
            .arg(payload)
            .query_async(&mut conn)
            .await
            .map_err(PublishError::Publish)
    }
}

The real decision is what happens on Err(PublishError::CircuitOpen). My pipeline treats it differently from a real Redis error: instead of retrying, it routes the event to a bounded, durable, on-disk buffer and moves to the next event immediately. Indexing never stops because publishing is degraded. That decoupling was the whole point.

5. Recovery Strategy: Letting the System Heal Itself

The cooldown-and-probe logic in allow_request does the real work. A tripped breaker isn't blocked forever — it sets a deadline, then lets exactly one request through as a health check.

I tuned the cooldown against Redis’s real recovery patterns: too short, and you’re back to retry-storm behavior; too long, and you’re leaving performance on the table after recovery. I landed on 5 seconds, backing off exponentially to 60 seconds across repeated trips — matching how long our actual failovers took to stabilize.

Half-open matters as much as the timer. One success isn’t proof — it might catch Redis mid-recovery, in a lull before it dips again. Requiring several consecutive successes before fully closing avoids flapping. One failure during half-open, though, sends it straight back to open — no benefit of the doubt. A struggling dependency needs less traffic, not a second chance right away.

This is what turned the incident from “someone gets paged to restart the publisher” into “the system detects the outage, backs off, and resumes on its own.” That’s the entire value of the pattern.

6. Benefits

  • The dependency gets protected, not punished. A tripped circuit sends Redis zero extra load instead of a retry wave — room to actually recover.
  • Resource waste drops. No threads parked on doomed connections. A rejected request during an open circuit costs microseconds, not a timeout.
  • Recovery is automatic and fast. Half-open probing means traffic resumes the moment Redis is healthy — no one has to notice.
  • Failure is honest. Callers get a fast, clear signal instead of an ambiguous hang, and can act on it — buffer, drop, alert.
  • Cascading failure is contained. The publisher fails fast instead of holding resources, so indexing, parsing, and storage keep running while publishing is degraded.

7. Trade-offs

Delayed recovery. The circuit won’t try real traffic again until the cooldown expires, even if Redis recovers a second after tripping. You’re trading recovery latency for protection against storms. Too aggressive a cooldown, and recovery feels sluggish even after the issue is gone.

Thresholds are hard. Failure threshold, success threshold, cooldown length — none of these come from a textbook. They depend on your traffic, your dependency’s failure modes, and how expensive a false trip is for you. I tuned mine against historical outage data. Don’t guess.

Monitoring isn’t optional. A circuit breaker silently changes your system’s behavior. Skip metrics and logs on state transitions, and you won’t know it tripped until someone asks why events stopped reaching Redis twenty minutes ago. I log and emit a metric on every trip() and reset(). Not optional — part of the pattern.

8. Lessons Learned

Bound your retries. Then put a breaker above them. A few retries for genuine blips is fine. Unbounded retries never are. Even bounded retries need a breaker to stop the pattern once the failure rate makes retrying pointless.

Fast failure is a feature. It feels wrong to write code that gives up without trying. But a caller with an immediate, clear error can act — fall back, buffer, degrade. A caller stuck in a 30-second timeout can’t do anything. Slow failure is almost always worse than fast failure.

Isolate every external dependency. Doesn’t have to be Redis — payment gateway, internal service, third-party API. Same pattern. Anything you call over a network can fail in a way that takes down parts of your system that have nothing to do with it. Containing the blast radius isn’t a nice-to-have. It’s baseline.

The outage lasted ninety seconds. The fix took longer. But my system today doesn’t notice when Redis has a bad afternoon — it buffers, waits, probes, resumes. That’s resilience engineering: not preventing failure, making it boring.