Event-driven architecture without losing operational discipline
Outboxes versus brittle dual-writes, idempotent consumers, poison queues, and drawing lines between choreography and orchestration.
Publishing an event feels lightweight until exactly-once illusions, skewed clocks, retry-induced duplicates, and schema drift collide at 2 AM. Durable asynchronous systems prioritize truthful delivery semantics, replay safety, evolvable contracts, and horizontal scaling without hero operators.
Most managed brokers provide at-least-once delivery—you must embrace idempotency, deduplication budgets, deterministic side effects partitioned by keyed ordering when necessary.
Outbox over hopeful dual-write
Writing a DB row plus immediately publishing externally without bridging invites classic split-brain ghosts: crash after commit-before-publish wedges silent divergence. The transactional outbox pattern persists intent rows inside the originating database transaction later drained by relays—extending coupling slightly while restoring atomic truth.
Operational tradeoff: relays must survive lag spikes without indefinite backlog starvation—monitor drift like any replication slot.
Schema evolution humility
Assume older consumers coexist during rollout windows. Prefer additive changes, optional fields, versioning headers, gradual contract tests. Breaking renames correlate strongly with outage retrospectives penned under caffeine haze.
Choreography versus orchestration
Pure choreography minimizes central coordinator coupling—yet observability fractures into puzzle pieces. Explicit orchestrators (often state machines) localize failure reasoning at the price of bottleneck risk—mitigate horizontally with partitioning.
Neither moral absolute; classify workflows by branching complexity and compensating transaction clarity.
Observability bridging async gaps
Instrument producer timestamps, canonical trace correlation IDs, consumer lag gauges, deliberate replay tooling guarded by safeguards. Incident narratives improve when timelines stitch HTTP edges to broker internals without interpretive archaeology.
Event-driven success is tempered patience: celebrate decoupling only after disciplined compensations exist when messages lie, duplicate themselves, arrive tardy fashionably—or never.
Related writing
- Distributed systems basics adults revisit under pressurePartial failures, clock lies, consensus tradeoffs, idempotency at money boundaries, and load shedding vocabulary operators actually recognize.
- Docker networking for engineers who resent memorizing buzzwordsBridge overlays, NAT and published ports, embedded DNS versus external discovery, and debugging paths beyond restart theater.
- Edge runtime explained — locality, isolation, and the limits of v8 snippetsWhat shifts when handlers run closer to users: cold starts, memory ceilings, cryptography constraints, and why edge does not magically delete physics.