jaykim88/async-messaging

Async Messaging (Transactional Outbox)

Purpose

Make "change state AND notify the world" reliable. The naive approach (write DB, then publish to a queue) loses events on crash between the two steps — the dual-write problem. The Outbox pattern solves it.

Universal — the Outbox pattern, the dual-write problem, and "exactly-once = at-least-once + consumer idempotency" are distributed-systems fundamentals independent of broker/language.

Procedure

1. Recognize the dual-write problem

   • "Write DB, then publish to Kafka/Redis" has no atomicity — crash after DB write but before publish = lost event; crash after publish but before commit = phantom event
   • You cannot atomically write to two systems without a distributed transaction (which you should avoid)

2. Apply the Transactional Outbox

   • Write the business state change AND an outbox row in ONE local DB transaction (see transaction-management)
   • Both commit together or neither does — atomicity guaranteed by the single DB

3. Relay the outbox asynchronously

   • Polling publisher: a background-jobs task reads unsent outbox rows, publishes to the broker, marks sent
   • CDC (Change Data Capture, e.g., Debezium): tail the DB log, publish outbox inserts — lower latency, more infra
   • Start with polling; move to CDC only if latency demands it

4. Design consumers for at-least-once (idempotency)

   • The relay guarantees at-least-once → consumers WILL see duplicates
   • "Exactly-once" delivery is a myth; achieve effective exactly-once via at-least-once + idempotent consumers (dedupe on event id / idempotency key — see resilience-patterns)

4b. Version events from day one (additive-only)

• Include eventType + eventVersion (or schemaVersion) on every payload — consumers will outlive any one publisher
• Additive-only changes on a live event type: new fields nullable, never remove or repurpose; bump the version and run old+new in parallel during cutover. Treat events like a public API contract (see api-contract)
• Never publish two semantically-different shapes under the same eventType + version

4c. Dead-letter queue (DLQ) + poison-message handling

• A retried-N-times failure is a poison message: route to a DLQ after the retry budget rather than blocking the queue forever
• DLQ needs alerting (a growing DLQ = silent failure) + a manual replay path with the original payload + reason
• Cap retry attempts and use backoff + jitter (see resilience-patterns / background-jobs)

5. Choose the broker by needs (criteria, not products)
   • Simple in-app queue — low setup, good for in-process jobs
   • Routing broker — work queues + routing topologies, moderate scale
   • High-throughput log — event replay, multiple consumer groups, high volume
   • Ordering needs a partition key: same-entity events go to the same partition (order_id, user_id). Within a partition: ordered; across partitions: no guarantees. Choose the key by the invariant you actually need
   • Don't reach for a heavyweight log broker by default — it's heavy infra (specific products in Implementation)

5b. Retain only what you need

• The outbox table grows on every write — schedule a cleanup job to delete (or archive) sent rows past N days, or it eats the DB
• For broker retention: replay needs ≠ storage forever; tune by recovery window (e.g., 7 days for backfill, not 1 year "just in case")

6. Validate (validation loop)
   • Crash the service between state write and relay → verify the event is still delivered (outbox row persisted)
   • Deliver an event twice → verify the consumer processes it once (idempotent)
   • If an event is lost → the write wasn't in the same transaction as the outbox row; fix and re-test

Anti-patterns

| ❌ Anti-pattern | ✅ Correct | |---|---| | Write DB then publish to broker (dual write) | Write state + outbox row in one transaction, relay async | | Assuming the broker gives exactly-once | At-least-once + idempotent consumers | | Publishing inside the business transaction (network I/O in tx) | Outbox row in tx; relay outside | | Heavyweight log broker for a simple in-app job | Simple queue first; log broker only at real scale | | Consumer with no dedup | Dedupe on event id | | Two payload shapes under the same eventType (silent break) | eventVersion + additive-only changes; run old+new in parallel on cutover | | Poison messages blocking the queue indefinitely | DLQ after retry budget; alert on DLQ growth; manual replay | | Outbox table growing unbounded | Scheduled cleanup of sent rows past retention window | | Ordering assumed without a partition key | Choose a partition key matching the invariant (same-entity events to the same partition) |

Severity tiers

| Tier | Examples | Action SLA | |---|---|---| | Critical | Dual-write losing payment/order events on crash; consumer with no idempotency double-charging; breaking schema change shipped on a live eventType (consumers crash on next message) | Fix immediately | | Major | Publishing inside a transaction (lock contention); no DLQ for poison messages; outbox table grows without cleanup (DB-eating); no partition key on a flow that requires ordering | Fix this sprint | | Minor | Polling interval untuned; broker over-provisioned (Kafka for tiny load); broker retention not aligned with actual replay needs | Schedule within 2 sprints |

Completion Criteria

• [ ] State change + event written in one transaction (Outbox)
• [ ] Relay (polling or CDC) publishes + marks sent
• [ ] Consumers idempotent (dedup on event id)
• [ ] Every event payload carries eventVersion; schema changes are additive-only
• [ ] DLQ wired + alerted on growth; manual replay path exists
• [ ] Outbox cleanup scheduled; broker retention matches the actual replay window
• [ ] Partition key chosen wherever ordering is required
• [ ] Broker choice justified by actual throughput/replay needs
• [ ] Crash + duplicate scenarios verified

Output

• Outbox table + relay: schema + polling publisher (or CDC config)
• Consumer handlers: idempotent, with dedup
• Architecture doc: event catalog + delivery guarantees
• Commit format: feat(events): add outbox for <event> / feat(events): idempotent consumer for <event>

Implementation

TypeScript + Prisma + BullMQ/Redis (default)

• Outbox: outbox(id, type, payload, created_at, sent_at) written inside prisma.$transaction with the state change
• Relay: a BullMQ repeatable job polls WHERE sent_at IS NULL, publishes, sets sent_at
• Consumers: BullMQ workers with dedup table on event id
• Scale path: Redis Streams → Kafka (KafkaJS) if replay/throughput demands

Other stacks

• Python: SQLAlchemy outbox + Celery relay; or Debezium CDC → Kafka
• Go: outbox + worker; Watermill library for messaging abstractions
• Universal: Outbox is DB-agnostic; Debezium provides CDC for Postgres/MySQL/Mongo; the at-least-once + idempotency principle is broker-independent

Related skills

• transaction-management — the outbox row is written in the same transaction as the state change
• background-jobs — the outbox relay / consumers run as jobs
• webhook-design — outbound webhooks can be driven by the outbox

Reference

• Key insight encoded: Solve the dual-write problem by writing business state and the event to an outbox table in one local transaction, then relay asynchronously (polling or CDC) — this is what makes at-least-once delivery achievable; "exactly-once" is really at-least-once + consumer idempotency. Treat events like a public API: version from day one, additive-only changes, partition key on flows that need ordering. Operational hygiene matters: a DLQ for poison messages + an outbox-cleanup job (else the outbox eats the DB).