Publishing an event feels lightweight until exactly-once illusions, skewed clocks, retry-induced duplicates, and schema drift collide at 2 AM. Durable asynchronous systems prioritize truthful delivery semantics, replay safety, evolvable contracts, and horizontal scaling without hero operators.
Most managed brokers provide at-least-once delivery—you must embrace idempotency, deduplication budgets, deterministic side effects partitioned by keyed ordering when necessary.
Outbox over hopeful dual-write
Writing a DB row plus immediately publishing externally without bridging invites classic split-brain ghosts: crash after commit-before-publish wedges silent divergence. The transactional outbox pattern persists intent rows inside the originating database transaction later drained by relays—extending coupling slightly while restoring atomic truth.
Operational tradeoff: relays must survive lag spikes without indefinite backlog starvation—monitor drift like any replication slot.
Schema evolution humility
Assume older consumers coexist during rollout windows. Prefer additive changes, optional fields, versioning headers, gradual contract tests. Breaking renames correlate strongly with outage retrospectives penned under caffeine haze.
Choreography versus orchestration
Pure choreography minimizes central coordinator coupling—yet observability fractures into puzzle pieces. Explicit orchestrators (often state machines) localize failure reasoning at the price of bottleneck risk—mitigate horizontally with partitioning.
Neither moral absolute; classify workflows by branching complexity and compensating transaction clarity.
Observability bridging async gaps
Instrument producer timestamps, canonical trace correlation IDs, consumer lag gauges, deliberate replay tooling guarded by safeguards. Incident narratives improve when timelines stitch HTTP edges to broker internals without interpretive archaeology.
Event-driven success is tempered patience: celebrate decoupling only after disciplined compensations exist when messages lie, duplicate themselves, arrive tardy fashionably—or never.