Idempotency Over Exactly-Once

1. Every Distributed System Will Process the Same Event More Than Once

Not might. Will.

This isn’t a bug you eventually fix. It’s a property of the environment. Networks drop acknowledgments. Consumers crash mid-processing and restart from the last committed offset. Producers retry on timeout without knowing whether the first attempt actually landed. Load balancers replay requests. Every one of these is normal, expected behavior in a distributed system — and every one of them produces a duplicate.

I run a production Rust event processing system that ingests a high-volume stream and writes results to PostgreSQL. Early on, I treated duplicates as an edge case — something to catch with a clever check here or there. That was the wrong frame. Once the system saw real traffic, real restarts, and real network hiccups, duplicates weren’t rare. They were routine.

The fix wasn’t a smarter way to prevent duplicates. It was accepting that they’ll happen and designing so it doesn’t matter when they do.

2. The Myth of Exactly-Once

“Exactly-once” gets thrown around like it’s an achievable delivery guarantee. It isn’t — not at the network layer, not without enormous cost, and usually not even then. What most systems actually offer is at-least-once delivery, dressed up with enough retry logic that duplicates become rare enough to ignore. Rare isn’t zero.

Here’s why exactly-once collapses in practice:

At-least-once delivery is the honest baseline. A message queue or event stream that promises no data loss has to re-deliver anything it can’t confirm was processed. Confirmation itself travels over a network — which means the confirmation can be lost even when the processing succeeded. The producer has no way to distinguish “the consumer never got it” from “the consumer got it but the ack didn’t.” So it resends. That’s not a flaw, it’s the only safe default.

Retries are the mechanism, and the source, of duplication. Every retry policy assumes the previous attempt might still succeed. If it did, you now have two successful attempts. There’s no version of “retry on failure” that doesn’t also mean “retry on success I didn’t hear about.”

Consumer restarts create the same gap. A consumer crashes after writing to the database but before committing its offset. On restart, it resumes from the last committed point — which is before the write it already made. The event gets reprocessed. This isn’t a rare crash scenario; it’s a normal deploy, a normal OOM kill, a normal pod reschedule.

Network failures blur success and failure into the same signal. A timeout doesn’t tell you what happened on the other end. It tells you the response didn’t arrive in time. The safe assumption under uncertainty is always “it might not have happened” — which pushes every retry-capable system toward at-least-once by default, because failing safe means retrying, and retrying means duplicating.

Preventing duplicates outright would require distributed consensus on every single event, globally ordered, with the network behaving perfectly. That’s not an engineering trade-off away — it’s not on the table. Exactly-once delivery, as a guarantee made by the transport layer, is a myth. What’s real and achievable is exactly-once effect — and that’s a property of your processing logic, not your message broker.

3. Shift the Mindset

The wrong question is “how do I avoid duplicate events?” You can reduce them. You cannot eliminate them, and chasing zero is where teams burn weeks building brittle deduplication machinery that still leaks under real failure conditions.

The right question is “how do I make duplicate events harmless?”

That’s idempotency: designing processing so that handling the same event once or five times produces the identical end state. Not “detect and skip” — though that’s often part of it — but a system where duplication simply doesn’t matter, because the operation converges to the same result regardless of how many times it runs.

This is a mindset shift with real architectural consequences. You stop trying to guarantee something the network can’t guarantee. You start engineering your write path so the guarantee becomes unnecessary.

4. A Practical Strategy: Two-Layer Deduplication

Here’s the approach running in production. It has two layers, and the split matters — each one solves a different class of duplicate, and combining them is what keeps this fast at scale.

Layer 1 — In-Batch Deduplication

Events arrive in batches. Before anything touches the database, duplicates within the same batch get collapsed using a HashSet keyed on the event's unique identifier.

use std::collections::HashSet;

fn dedupe_batch(events: Vec<Event>) -> Vec<Event> {
    let mut seen = HashSet::with_capacity(events.len());
    events
        .into_iter()
        .filter(|event| seen.insert(event.id.clone()))
        .collect()
}

This catches the cheapest, most common case: a retry that lands in the same batch window as the original, or an upstream producer that emits the same event twice in quick succession. It costs a hash insert per event — negligible next to a database round trip — and it means the database never even sees these duplicates.

Layer 2 — Cross-Batch Deduplication Against the Database

The HashSet only knows about the current batch. It has no memory of what was processed five minutes ago, or in the batch before a restart. For that, the database has to be the source of truth — but checking it one row at a time is where naive implementations fall apart.

The wrong way is an existence check per event:

// Don't do this — one round trip per event.
for event in &batch {
    let exists = sqlx::query_scalar!(
        "SELECT EXISTS(SELECT 1 FROM events WHERE id = $1)",
        event.id
    )
    .fetch_one(&pool)
    .await?;
    // ...
}

That’s an N+1 query pattern wearing a disguise. A batch of a thousand events becomes a thousand round trips before a single insert happens. It works in a demo and falls over in production.

The right way is a single query using ANY():

async fn filter_new_events(
    pool: &PgPool,
    batch: &[Event],
) -> Result<Vec<Event>, sqlx::Error> {
    let ids: Vec<String> = batch.iter().map(|e| e.id.clone()).collect();

let existing_ids: Vec<String> = sqlx::query_scalar!(
        "SELECT id FROM events WHERE id = ANY($1)",
        &ids
    )
    .fetch_all(pool)
    .await?;
    let existing: HashSet<String> = existing_ids.into_iter().collect();
    Ok(batch
        .iter()
        .filter(|e| !existing.contains(&e.id))
        .cloned()
        .collect())
}

One query. One round trip, regardless of whether the batch has ten events or ten thousand. Postgres does what it’s good at — a set-based lookup against an indexed column — instead of the application doing it one row at a time.

Combined, the two layers mean: cheap in-memory work eliminates same-batch duplicates before they cost anything, and a single indexed query eliminates everything the database has already seen. Nothing about this scales worse as batch size grows — it scales better, because larger batches amortize the fixed cost of that one query across more events.

5. Why This Works

High throughput. The database is touched once per batch for deduplication, not once per event. Batch size becomes a throughput lever instead of a query-count multiplier.

Minimal database load. ANY() against an indexed primary or unique key is a fast, well-optimized query path. It's the kind of query Postgres was built to make cheap.

Better scalability. The pattern doesn’t degrade as volume grows — it improves, because bigger batches mean the fixed cost of the existence check gets spread thinner.

True idempotent processing. Duplicates don’t cause errors, don’t cause double-writes, and don’t need special-case handling downstream. They just get filtered out before they can do anything.

Simpler architecture. No distributed lock, no consensus protocol, no exactly-once messaging guarantee to configure and hope holds. Just a HashSet and one query. The simplicity is the point — it's less to get wrong.

6. Trade-offs

Memory usage. The HashSet holds every ID in the batch in memory. For very large batches, or for IDs that are large strings rather than compact identifiers, this adds up. Keep IDs small and bound batch size to something proportional to available memory.

Database indexing is not optional. The ANY() query is only fast because the id column is indexed. Skip the index and this pattern silently becomes a sequential scan under load — the exact N+1-style disaster this was built to avoid, just hidden one layer down.

Large batches shift the bottleneck. Bigger batches amortize the fixed query cost, but they also mean more latency before the first event in a batch gets processed, and a bigger blast radius if a batch fails partway through. There’s a real ceiling past which “bigger batch” stops being free.

Event ordering isn’t solved by this. Deduplication tells you an event has already been processed. It says nothing about whether events arrive or get processed in order. If your domain logic depends on ordering — state transitions, sequential balances, anything where order changes the outcome — you need a separate mechanism (sequence numbers, versioning, or ordering keys) on top of this. Idempotency and ordering are different problems that get conflated more often than they should.

7. Lessons Learned

Idempotency beats complexity. Every attempt I’ve seen at guaranteeing exactly-once delivery adds more moving parts than it removes. A HashSet and one indexed query outperformed every more elaborate scheme I tried, both in throughput and in how easy it was to reason about at 2 a.m.

Duplicate events are inevitable — design like it. The moment you stop treating duplicates as an exception path, a lot of defensive code disappears. There’s no special case for “what if this arrives twice,” because arriving twice was already the design assumption.

The database is the source of truth, not the message queue. Kafka, SQS, whatever you’re using — none of them can tell you with certainty what your system has already done. Only your database can answer that, because it’s the thing that recorded the effect. Lean on it.

Design for retries from day one. Retrofitting idempotency into a system that assumed clean, single delivery is far more expensive than building it in from the start. If you’re standing up a new event pipeline right now, assume every event arrives more than once before you write the first line of processing logic. It will save you the incident that taught me this.

Exactly-once is a good story for a slide deck. Idempotency is what actually ships.