2024 · Backend engineer
Signal
Streaming ingestion that survives its upstreams.
Numbers
2.5K/s
Sustained events99.98%
DeliveryOverview
A pipeline that swallows bursty webhook traffic from partners with no rate limits and no retry etiquette. Backpressure is explicit: the queue is bounded, overflow goes to cold storage, and replay is a first-class operation instead of an incident.
Most of the work was in the failure modes. Every partner is a separate circuit, so one misbehaving integration degrades exactly one tenant.
Stack
GoRedisClickHouseGrafana