jaykim88/resilience-patterns

Resilience Patterns

Purpose

Make every outbound call and every mutating endpoint survive partial failure without causing duplicate effects or cascading outages. Retries, timeouts, breakers, and idempotency keys are the primitives — apply them deliberately, not reflexively.

Universal — capped exponential backoff + jitter, circuit breakers, timeout-from-p99, and idempotency keys are distributed-systems primitives independent of language; only the library differs.

Procedure

1. Set timeouts from p99.9 latency — never infinite

   • Measure the dependency's p99.9; set timeout slightly above it
   • No timeout = one slow dependency exhausts your connection pool → cascading failure

2. Retry ONLY idempotent operations

   • GET / PUT / DELETE are naturally idempotent → safe to retry
   • POST is NOT → retry only with an idempotency key (step 4)
   • Never blindly retry a non-idempotent mutation

3. Use capped exponential backoff + full jitter

   • delay = random(0, min(cap, base * 2^attempt))
   • Full jitter prevents thundering herd (synchronized retries hammering a recovering service)
   • Cap total attempts (e.g., 3-5); after cap, fail to a fallback or DLQ

4. Idempotency keys for mutations

   • Client sends Idempotency-Key: <uuid> header on POST
   • Server stores key → result mapping; on replay, return the stored result (don't re-execute)
   • Store the key in the SAME transaction as the business effect

5. Circuit breaker for repeatedly-failing dependencies

   • Open the breaker after N consecutive failures → fail fast instead of waiting on timeouts
   • Half-open after a cooldown to test recovery
   • Pair with a fallback (cached value, degraded response, queue-for-later)

5b. Bulkhead: isolate critical paths from one bad dependency

• Partition resources (connection pools, worker concurrency) per dependency so one slow integration can't drain the pool that serves the rest
• Without bulkheading, your healthy endpoints fail because the unhealthy dependency holds every connection

5c. Define a fallback chain for high-availability paths

• Primary → secondary → cached/degraded → user-friendly error
• Each rung must be decided in advance (not improvised mid-incident); annotate which rung the response came from so observability sees the degradation

5d. Propagate deadlines across hops

• When request A times out at 800ms, its outbound call to B shouldn't have a 2s timeout — by the time B returns, A is already done. Pass the remaining budget (header or context) so each hop's timeout = min(local-budget, deadline-remaining)
• Otherwise downstream services do work nobody waits for

6. Validate (validation loop)
   • Simulate dependency failure (kill the service / inject latency)
   • Verify: no duplicate effects on retry, no cascading timeout, breaker opens and recovers
   • If duplicates occur → idempotency key not applied correctly; fix and re-test

Anti-patterns

| ❌ Anti-pattern | ✅ Correct | |---|---| | Retry a POST with no idempotency key | Idempotency-Key header + stored result | | Exponential backoff without jitter | Capped exponential + full jitter | | No timeout on outbound call | Timeout from dependency p99.9 | | Infinite retries | Cap attempts → fallback / DLQ | | Catch-all retry on every error | Retry only transient errors (timeout/503), not 4xx | | One bad dependency drains the shared connection pool | Bulkhead per dependency (separate pool / concurrency limit) | | Improvising the fallback during the incident | Predeclared fallback chain (primary → secondary → degraded) | | 2s downstream timeout under a 800ms caller deadline | Propagate the deadline; downstream timeout = min(local, remaining) |

Severity tiers

| Tier | Examples | Action SLA | |---|---|---| | Critical | Payment/order POST retried without idempotency key (duplicate charges); no timeout on a synchronous external call | Block release; fix immediately | | Major | Backoff without jitter; missing circuit breaker on a known-flaky dependency | Fix this sprint | | Minor | Retry cap slightly high; breaker cooldown untuned | Schedule within 2 sprints |

Completion Criteria

• [ ] Every outbound call has a timeout derived from p99.9
• [ ] Every POST that can be retried uses an idempotency key
• [ ] Backoff is capped + jittered
• [ ] Flaky dependencies wrapped in a circuit breaker + fallback
• [ ] Failure simulation confirms no duplicate effects

Output

• Resilience config: timeout/retry/breaker policy per dependency (documented)
• Idempotency table: schema + the transaction wiring
• Commit format: feat(resilience): add idempotency key to <endpoint> / fix(resilience): cap backoff + jitter on <client>

Implementation

TypeScript + NestJS (default)

• Library: Cockatiel (composable retry + circuit breaker + timeout + bulkhead)
• Idempotency: store idempotency_key → response in Postgres, checked in the same Prisma $transaction as the write
• HTTP client: wrap fetch/axios calls in a Cockatiel policy

Other stacks

• Python / FastAPI: tenacity (retry/backoff) + pybreaker (circuit breaker)
• Go: cenkalti/backoff + sony/gobreaker
• Universal: idempotency key pattern is HTTP-level (Stripe's Idempotency-Key is the reference); jitter formula is math, not library-specific

Related skills

• webhook-design — webhook delivery uses the same retry + idempotency primitives
• background-jobs — job retries must be idempotent
• transaction-management — idempotency keys often stored in the same transaction as the effect

Reference

• Key insight encoded: Cap exponential backoff and add full jitter to prevent thundering herd; retry only idempotent operations (use a Stripe-style Idempotency-Key for POST). Set timeouts from p99.9, not guesses.