jaykim88/webhook-design

Webhook Design

Purpose

Build webhook delivery and consumption that survives the real world: signed, retried, idempotent on the sending side; signature-verified, fast-acknowledged, and deduped on the receiving side.

Universal — HMAC signing, signature verification on raw body, at-least-once + retry, and event.id dedup are protocol-level patterns independent of language.

Procedure

Receiving webhooks

1. Verify the signature against the RAW (unparsed) body

   • Compute HMAC over the exact bytes received; compare to the provider's signature header
   • Body-parsing middleware that mutates the payload breaks signature verification — capture raw body first
   • Reject (401) on mismatch before any processing
   • Enforce a timestamp window (e.g., ±5 min skew tolerance) on the signed payload — a valid signature on an old captured request is a replay vector; reject if now - signedAt exceeds the window

2. Acknowledge fast: enqueue, then return 2xx before any complex logic

   • Do NOT process inline — verify → enqueue to background-jobs → return 2xx
   • Return well under the provider's timeout; slow processing → provider times out → unnecessary retries → duplicate work
   • (Providers don't all publish a fixed timeout; the rule is "ack fast, process async")

3. Dedupe on event.id (idempotency)

   • Providers deliver at-least-once → you WILL receive duplicates
   • Store processed event.id; on replay, skip — ideally dedupe in the SAME transaction as the business write

Sending webhooks

4. Sign outbound payloads (HMAC) + include a timestamp

   • Let receivers verify authenticity; timestamp prevents replay (pair with the receiver's clock-skew window in step 1)
   • Version the payload (eventType + eventVersion): downstream consumers outlive your publisher; additive-only changes once live (see async-messaging for the same discipline)
   • Document your signature scheme

5. Retry with exponential backoff over a long window

   • At-least-once delivery: retry failed deliveries (e.g., escalating over hours/days like Stripe's 3-day schedule)
   • Use resilience-patterns backoff + jitter; consider Outbox (async-messaging) so an event is never lost on crash

6. Validate (validation loop)

   • Send a duplicate event → verify the receiver processes it once (dedup works)
   • Send a tampered payload → verify signature rejection
   • Kill the receiver mid-delivery → verify retry + eventual delivery, no lost event

Anti-patterns

| ❌ Anti-pattern | ✅ Correct | |---|---| | Verify signature after body parsing | Verify on raw body bytes | | Process webhook inline before returning 200 | Verify → enqueue → 200 fast | | No event.id dedup | Dedupe on event id (at-least-once = duplicates happen) | | Fire-and-forget outbound (no retry) | Retry with backoff; Outbox for crash-safety | | Unsigned outbound payloads | HMAC sign + timestamp | | Signature valid but timestamp ancient (replay) | Enforce a window (e.g., ±5 min) on the signed timestamp | | Outbound payload shape changed under the same eventType | eventVersion + additive-only changes |

Severity tiers

| Tier | Examples | Action SLA | |---|---|---| | Critical | No signature verification (forged webhooks accepted); processed inline → provider timeout → unintended duplicates on retry; no event.id dedup on payment / order webhooks (double-effects) | Block release; fix immediately | | Major | Signature verified after body-parse (mutated bytes → false rejects); no timestamp window (replay of old captured payloads); outbound retries without backoff (storming a recovering receiver) | Fix this sprint | | Minor | Outbound payload not versioned; missing event.id on a low-risk flow; receiver timeout not tuned to the provider's window | Schedule within 2 sprints |

Completion Criteria

• [ ] Inbound signature verified on raw body
• [ ] Inbound handler returns 2xx fast — enqueue, don't process inline (well under provider timeout)
• [ ] Inbound dedup on event id
• [ ] Outbound payloads signed + retried with backoff
• [ ] Duplicate + tampered + crash scenarios verified

Output

• Receiver: raw-body signature middleware + enqueue + dedup table
• Sender: signing + retry policy (+ Outbox if crash-safety required)
• Commit format: feat(webhook): verify + enqueue <provider> webhooks / feat(webhook): sign + retry outbound <event>

Implementation

TypeScript + NestJS (default)

• Raw body: NestJS rawBody: true + @Req() for the buffer; verify with crypto.createHmac
• Enqueue: BullMQ job from the handler, return 200 immediately
• Dedup: webhook_events(event_id PK) checked in the processing transaction
• Stripe SDK: stripe.webhooks.constructEvent(rawBody, sig, secret) does verification

Other stacks

• Python / FastAPI: await request.body() for raw bytes; hmac.compare_digest; enqueue to Celery
• Go: read io.ReadAll(r.Body) before parsing; hmac package; enqueue to Asynq
• Universal: HMAC + raw-body verification + at-least-once dedup are HTTP-level; identical across stacks

Related skills

• resilience-patterns — webhook retries + idempotency reuse those primitives
• background-jobs — received webhooks are processed off the request thread
• async-messaging — outbound webhooks are events; consider Outbox for reliability

Reference

• Key insight encoded: Verify the signature against the raw (unparsed) body, then enqueue to a background queue and return 2xx fast (before complex logic) — never process inline; dedupe on event.id in the same transaction as the business write.