1. The Hook
At some point in every growing backend system, a pattern emerges that should raise an eyebrow: every time you add a new type of event to track, you also add a new file, a new struct, and a new pipeline that looks almost exactly like the last one.
That’s where I found myself while building a blockchain indexer. Every new on-chain event — a trade, a market update, a liquidity change, a settlement — needed its own collector. Each one fetched data from an RPC endpoint, parsed the response, validated the event, stored it, and published a message downstream.
The first collector took a day to write. The second took an hour — mostly copy-paste. By the fourth, I wasn’t writing code anymore. I was performing a ritual: duplicate the file, rename the struct, swap the parsing logic, ship it.
That’s the moment you should stop and ask: what part of this is actually different, and what part am I just retyping?
This article isn’t about blockchain indexing. It’s about a design problem that shows up anywhere you process a stream of typed events — webhooks, Kafka consumers, log processors, ETL jobs. The indexer is just the example that made the duplication impossible to ignore.
2. The Problem
Here’s roughly what the four collectors looked like, structurally:
struct TradeCollector { /* rpc client, db pool, publisher */ }
impl TradeCollector {
async fn run(&self) -> Result<()> {
let raw = self.fetch_page().await?; // fetch
let trades = self.parse(raw)?; // parse
let valid = self.validate(trades)?; // validate
self.store(&valid).await?; // store
self.publish(&valid).await?; // publish
Ok(())
}
}Now put MarketCollector, LiquidityCollector, and SettlementCollector next to it. The method names are identical. The control flow is identical. Retry logic, pagination cursors, logging statements — all copy-pasted, all slightly out of sync because someone fixed a bug in TradeCollector's retry loop and forgot to backport it to the other three.
The only things that actually differed were:
- The event type being fetched and stored (Trade vs. Market vs. Liquidity vs. Settlement)
- The parsing logic for turning a raw RPC response into that type
- The storage call — which table, which columns, which conflict-resolution strategy
Everything else — HTTP fetching, pagination, retries, error classification, logging, metrics — was identical scaffolding wrapped around a different payload type.
That’s a strong signal for generics: when the control flow is fixed but the payload type varies, you’re not looking at four problems — you’re looking at one problem with four instances.
3. Finding the Abstraction
Before writing any generic code, it’s worth explicitly separating what’s invariant from what’s variant. Skipping this step is how you end up with a generic abstraction that still leaks event-specific logic through the seams.
Stays the same across every collector:
- Request/response flow (call the RPC endpoint, get a page of results)
- Retry logic (exponential backoff on transient failures)
- Pagination (cursor tracking, “no more pages” detection)
- Logging and metrics (start/end of batch, error counts, latency)
- Top-level error handling and propagation
Changes per event type:
- The event model itself (Trade, Market, Liquidity, Settlement)
- How raw bytes/JSON become that model (parsing)
- Where and how that model gets persisted (storage)
- What downstream topic/queue it gets published to, and in what shape
Once you write these two lists side by side, the design falls out naturally. The invariant part becomes a generic struct. The variant part becomes a trait that each event type implements. The generic struct owns the process; the trait owns the payload behavior.
This is the same shape as the Template Method pattern, but in Rust you get it via composition and trait bounds rather than inheritance — which also means it’s statically checked and has zero runtime dispatch cost if you don’t need dynamic dispatch.
4. Designing GenericEventCollector
The core idea: define a trait that captures everything event-specific, then write one collector struct generic over any type implementing that trait.
#[async_trait]
pub trait CollectableEvent: Sized + Send + Sync {
type Raw: Send; // raw response shape from the source
type Error: std::error::Error + Send + Sync + 'static;
/// Turn a raw fetched page into typed events.
fn parse(raw: Self::Raw) -> Result<Vec<Self>, Self::Error>;
/// Reject malformed or out-of-range events before persistence.
fn validate(&self) -> Result<(), Self::Error>;
/// Persist a validated batch.
async fn store(pool: &PgPool, events: &[Self]) -> Result<(), Self::Error>;
/// Publish a validated batch downstream.
async fn publish(bus: &MessageBus, events: &[Self]) -> Result<(), Self::Error>;
}The generic collector then owns fetching, pagination, retries, and logging — all the invariant machinery — and delegates everything else to the trait:
pub struct GenericEventCollector<E: CollectableEvent> {
source: Arc<dyn EventSource<Raw = E::Raw>>,
pool: PgPool,
bus: MessageBus,
_marker: PhantomData<E>,
}
impl<E: CollectableEvent> GenericEventCollector<E> {
pub async fn run(&self) -> anyhow::Result<()> {
let mut cursor = None;
loop {
let raw = self.fetch_with_retry(cursor.clone()).await?;
let Some(page) = raw else { break }; // no more pages
let events = E::parse(page.data)?;
let valid: Vec<E> = events
.into_iter()
.filter(|e| e.validate().is_ok())
.collect();
E::store(&self.pool, &valid).await?;
E::publish(&self.bus, &valid).await?;
tracing::info!(count = valid.len(), "batch processed");
cursor = page.next_cursor;
}
Ok(())
}
async fn fetch_with_retry(&self, cursor: Option<Cursor>) -> anyhow::Result<Option<Page<E::Raw>>> {
// shared backoff/retry logic lives here, once
retry_with_backoff(|| self.source.fetch(cursor.clone())).await
}
}Each concrete collector is now a one-line type alias plus a trait implementation:
pub struct Trade { /* fields */ }
#[async_trait]
impl CollectableEvent for Trade {
type Raw = TradeResponse;
type Error = TradeError;
fn parse(raw: Self::Raw) -> Result<Vec<Self>, Self::Error> { /* trade-specific */ }
fn validate(&self) -> Result<(), Self::Error> { /* trade-specific */ }
async fn store(pool: &PgPool, events: &[Self]) -> Result<(), Self::Error> { /* ... */ }
async fn publish(bus: &MessageBus, events: &[Self]) -> Result<(), Self::Error> { /* ... */ }
}
type TradeCollector = GenericEventCollector<Trade>;MarketCollector, LiquidityCollector, and SettlementCollector become the same one-liner, each backed by their own CollectableEvent implementation. The retry loop, pagination, and logging exist in exactly one place in the codebase.
5. Why Generics Instead of Copy-Paste
It’s worth being explicit about why this is better than four hand-maintained copies, beyond “less code”:
Compile-time safety. GenericEventCollector<Trade> can never accidentally call Market's storage logic or publish a Trade to the wrong topic. The type parameter ties every operation to the correct event type — the compiler enforces the coupling that used to live only in the file name.
Zero runtime cost. Rust monomorphizes GenericEventCollector<E> at compile time, generating a specialized version per concrete E. There's no vtable, no dynamic dispatch, no boxing unless you explicitly ask for it (e.g., to store heterogeneous collectors in a Vec<Box<dyn Any>>). You get the ergonomics of a generic abstraction with the performance profile of the hand-written version.
Less maintenance surface. A bug fix in retry logic or pagination is written once and applies to every event type automatically. Previously, fixing a bug meant remembering (or grepping for) every collector that needed the same patch.
Easier testing. You can write one test suite against a MockEvent: CollectableEvent and validate the entire pipeline — fetch, retry, parse, validate, store, publish — without spinning up trade-specific or settlement-specific infrastructure. Each concrete implementation only needs tests for its own parse/validate/store logic, not the orchestration around it.
Consistent behavior by construction. Logging format, error classification, and retry timing are guaranteed identical across event types, because they’re literally the same code path — not four implementations that are supposed to behave the same way.
6. Trade-offs
This isn’t free, and it’s worth being honest about the costs:
More complex type signatures. GenericEventCollector<E: CollectableEvent> with associated types (E::Raw, E::Error) is a heavier signature than a concrete struct. Anyone reading the code for the first time needs to understand the trait before they understand the collector.
Harder compiler errors. When a trait bound isn’t satisfied, or an associated type doesn’t line up, Rust’s error messages can get long — especially once async_trait and Send + Sync bounds are involved. Expect to spend more time reading trait-resolution errors than you would debugging a concrete struct.
Learning curve for the team. Generic, trait-based designs ask more of contributors up front. Someone adding a fifth event type now needs to understand CollectableEvent's contract, not just copy a file and rename things. That's a net win for long-term maintainability, but it's a real short-term cost if your team is new to this style of Rust.
In my case, the trade-off was worth it because the number of event types was growing, not fixed. If you genuinely only ever need two or three near-identical collectors and that number isn’t going to change, the abstraction may not pay for itself. Generics are a bet on future duplication — make that bet deliberately, not reflexively.
7. Key Takeaways
- Duplication across types, not duplication of code, is the real signal. If your structs differ only in the type they operate on, copy-paste isn’t a shortcut — it’s technical debt with a delay timer.
- Separate the invariant from the variant before you design anything. Two lists — “always the same” and “always different” — will usually tell you exactly where the trait boundary belongs.
- Traits define the contract; the generic struct owns the process. This split is what keeps the abstraction honest — the generic collector shouldn’t know anything about Trade or Settlement specifically, only about the trait.
- Generics in Rust aren’t just about avoiding repetition — they’re about encoding a guarantee. The compiler now enforces that a Trade collector can never touch Market storage logic. That's a correctness property you don't get from disciplined copy-pasting, no matter how careful you are.
- Reusable components are designed, not extracted. The goal isn’t to mechanically merge four files — it’s to identify the actual abstraction your domain has been asking for, and let the language’s type system hold it in place.
The blockchain indexer eventually shrank from four semi-duplicated collectors to one generic implementation and four small trait impls, each under fifty lines. But the more durable result wasn’t the line count — it was that adding the fifth event type took twenty minutes and didn’t require touching the pipeline at all.