QorvexDesigning backpressure that survives real traffic
Backpressure is easy to draw and hard to operate. In this post we share the invariants we rely on to keep delivery predictable under load.
Bound the queues, not the hope
Every stage has an explicit, bounded buffer. When it fills, we shed load upstream instead of silently growing memory. This keeps latency bounded and failures local.
Exactly-once without heroes
We pair idempotent producers with transactional sinks, so a retry after a restart is safe by construction. Offsets commit only after the sink acknowledges.
What we watch
Consumer lag, checkpoint duration and sink error rate are the three signals that predict incidents. They are on every project's default dashboard.