jaykim88/background-jobs

Background Jobs

Purpose

Move slow, external, or deferrable work off the request path into a durable queue with proper retries, dead-lettering, and idempotent handlers — so failures are visible and recoverable, not silent.

Universal — queue design, retry/backoff, DLQ, concurrency control, and idempotent handlers are job-processing principles; BullMQ is the default implementation.

Procedure

1. Decide what belongs in a job

   • Slow (> ~1s), external I/O, deferrable, or retryable work → job
   • Keep the request fast; return early, do the work async
   • Don't queue work that must be synchronous for the user's next action

2. Make every handler idempotent

   • At-least-once execution means a job CAN run more than once (retry, redelivery)
   • Dedupe on a job/business key; design the side effect to be safe to repeat (see resilience-patterns idempotency)

3. Configure retries: exponential backoff + jitter, capped

   • Set attempts + backoff: { type: 'exponential' } (+ jitter)
   • Distinguish retryable (timeout, 503) from non-retryable (validation, 4xx) failures — don't retry the un-retryable

4. Dead-letter queue for poison messages

   • After max attempts, move to a DLQ instead of dropping or infinite-retrying
   • Alert on DLQ growth; provide a replay path after the bug is fixed

5. Control concurrency — and apply backpressure

   • Set worker concurrency to protect downstream (DB pool, rate-limited API)
   • Rate-limit jobs hitting a quota'd external service
   • Producers outpacing consumers → queue depth climbs. Decide the policy: producer slowdown / circuit-break the producing endpoint / shed (return 429 / drop low-priority). Unbounded queue depth is a delayed outage, not a working system

5b. Set a visibility timeout / heartbeat for long jobs

• A worker that dies mid-job leaves the job in "in progress" — without a visibility timeout it's stuck forever; with one (and an active heartbeat from healthy workers) the job becomes eligible for re-delivery
• Tune the timeout to slightly above the job's p99 duration; emit heartbeats for genuinely-long jobs

5c. Retain only what you need (job table cleanup)

• Completed and failed jobs accumulate — set removeOnComplete (or equivalent) with a retention count/age; archive critical job history elsewhere if needed
• An unbounded job store eats Redis / DB memory silently

6. Make jobs observable

   • Log job start/end/failure with correlation id (see observability-setup)
   • Track queue depth, processing latency, failure rate

7. Validate (validation loop)

   • Force a job to fail → verify retry with backoff, then DLQ after cap (not infinite, not silent)
   • Run a job twice → verify idempotent (no duplicate side effect)
   • If a duplicate side effect occurs → handler not idempotent; fix and re-test

Anti-patterns

| ❌ Anti-pattern | ✅ Correct | |---|---| | Non-idempotent handler (retry double-charges) | Idempotent handler (dedupe key) | | Retry forever on any failure | Capped attempts → DLQ; retry only transient errors | | Failed jobs silently dropped | DLQ + alert + replay path | | Unbounded concurrency exhausting DB pool | Tuned concurrency + rate limits | | Blocking the request on slow work | Enqueue, return fast | | No visibility timeout / heartbeat → job stuck "in progress" forever on worker crash | Visibility timeout slightly above p99; heartbeat for long jobs | | Completed jobs accumulating in Redis until OOM | removeOnComplete retention + archive critical history elsewhere | | Producer faster than consumers; queue grows unbounded | Backpressure policy (slow producer / 429 / shed low-priority) |

Severity tiers

| Tier | Examples | Action SLA | |---|---|---| | Critical | Non-idempotent job double-charging on retry; failed payment jobs silently dropped | Fix immediately | | Major | No DLQ (infinite retry or data loss); no backoff (retry storm) | Fix this sprint | | Minor | Concurrency untuned; missing queue-depth metric | Schedule within 2 sprints |

Completion Criteria

• [ ] Handlers idempotent (verified by double-run)
• [ ] Retries capped with exponential backoff + jitter
• [ ] DLQ configured + alerted
• [ ] Concurrency tuned to protect downstream
• [ ] Queue depth + failure rate observable

Output

• Job definitions + worker config: retry/backoff/concurrency/DLQ
• Idempotency keys per job type
• Commit format: feat(jobs): add <job> with DLQ + backoff / fix(jobs): make <handler> idempotent

Implementation

TypeScript + BullMQ + Redis (default)

• defaultJobOptions: { attempts: 5, backoff: { type: 'exponential', delay: 1000 } } (+ jitter via custom strategy)
• DLQ: a separate queue; move on failed after attempts exhausted; BullMQ failed events
• Concurrency: new Worker(name, fn, { concurrency: N, limiter: { max, duration } })
• Idempotent: dedupe table on job key, or use jobId for natural dedup
• See BullMQ "Going to Production"

Other stacks

• Python: Celery (acks_late=True, max_retries, autoretry_for) + dead-letter via routing
• Go: Asynq or River — same retry/DLQ/concurrency model
• Universal: idempotency + capped backoff + DLQ are queue-agnostic; the at-least-once contract is the same everywhere

Related skills

• async-messaging — the outbox relay and event consumers run as jobs
• resilience-patterns — job retries reuse backoff + idempotency
• webhook-design — received webhooks are processed as jobs

Reference

• Key insight encoded: Configure exponential backoff with jitter in defaultJobOptions, cap attempts, and make handlers idempotent — at-least-once execution means a job can run more than once on retry.