jaykim88/observability-setup

Observability Setup (Backend)

Purpose

Turn production silence into signal by emitting three correlated signals — logs, metrics, traces — joined by one propagated context, so any request can be reconstructed end-to-end and any regression is visible.

Universal — the three-signals model, RED/USE methods, and trace-context propagation are vendor-neutral (OpenTelemetry is the CNCF standard); implemented across Node/Python/Go.

Procedure

1. Propagate ONE correlation context

   • Generate/accept a traceId (W3C Trace Context header) at the edge
   • Include it in every log line, span, and downstream call (incl. background jobs + queue messages)
   • This is what lets logs/metrics/traces join — without it you have three disconnected silos

2. Structured logging (JSON)

   • { level, timestamp, traceId, route, userId(anonymized), msg, ...context }
   • No console.log; one logger utility; levels (debug/info/warn/error)
   • Never log secrets/PII

3. Metrics — RED for services, USE for resources

   • RED (services/endpoints): Rate, Errors, Duration (p50/p95/p99)
   • USE (resources: DB pool, CPU, queue): Utilization, Saturation, Errors
   • Apply RED to request handlers, USE to infrastructure — different questions, different methods
   • Cardinality budget: every unique label combination is a stored time series — putting userId, requestId, email, or unbounded ids in labels explodes the TSDB. Keep labels low-cardinality (route, status code, region); push per-request detail into logs/traces, not metrics

4. Distributed tracing (OpenTelemetry) — and sample it

   • Auto-instrument the framework + DB + HTTP client; add manual spans around key business operations
   • Spans carry the traceId; a trace shows the full request waterfall (where time went)
   • Sampling: 100% trace capture is unaffordable at scale. Choose: head-based (decide at request start, simplest), tail-based (decide after the trace completes — keeps all errors + slow traces, drops the boring middle), or adaptive. Default to head-based at a low rate + always-on for errors

5. Capture errors with context — and scrub PII

   • Errors → error tracker (Sentry) WITH traceId + anonymized user context
   • Any catch that returns gracefully must still record the error (silent catch = invisible failure)
   • The SDK captures more than you set: request URLs, headers, breadcrumbs (and span attributes in tracing) can leak tokens, auth headers, and PII. Configure beforeSend scrubbing + platform-side data scrubbing; treat the trace/log/error pipeline as the same trust boundary as your API responses

6. Alert on patterns, not noise

   • Alert on RED/USE threshold breaches (error rate spike, p99 regression, queue saturation), not every error
   • Define SLOs; alert on burn rate

7. Validate (validation loop)

   • Trigger a test error + a slow request → verify you can find them by traceId across logs + trace + error tracker
   • If the three signals don't join on traceId → context propagation is broken; fix and re-test

Anti-patterns

| ❌ Anti-pattern | ✅ Correct | |---|---| | console.log everywhere | Structured JSON logger with traceId | | Logs/metrics/traces with no shared id | One propagated W3C trace context | | Averages only | p95/p99 (tail latency is what users feel) | | RED applied to infra / USE to endpoints | RED→services, USE→resources | | Alert on every error | Alert on SLO burn / threshold breach | | Silent catch (no record) | Record every handled error with context | | userId / requestId / email in metric labels (TSDB explosion) | Low-cardinality labels (route, status, region); per-request detail in logs/traces | | 100 % trace capture in production | Head- or tail-based sampling; always-on for errors + slow traces | | Trusting SDK auto-captured payloads to be PII-free | beforeSend scrubbing + platform data scrubbing; treat the pipeline as a trust boundary |

Severity tiers

| Tier | Examples | Action SLA | |---|---|---| | Critical | No error tracking in production; silent catches hiding failures; no correlation id (can't trace incidents) | Block release; fix immediately | | Major | Metrics on averages only (no p99); traces not propagated to jobs/queues; PII / auth headers leaking into traces or breadcrumbs (no beforeSend scrubbing) | Fix this sprint | | Minor | Alert noise (per-error alerts); missing USE metrics on a resource; metric cardinality unbudgeted (label explosion risk); trace sampling rate unset | Schedule within 2 sprints |

Completion Criteria

• [ ] One correlation context propagated through requests + jobs + queues
• [ ] Structured JSON logging (no console.log; traceId on every line)
• [ ] RED metrics on services, USE on resources, p95/p99 tracked
• [ ] OpenTelemetry tracing wired (auto + key manual spans)
• [ ] Errors captured with traceId; SLO-based alerts
• [ ] Trace sampling rate set (head- or tail-based); always-on for errors / slow traces
• [ ] Metric labels low-cardinality (no userId / unbounded ids); a cardinality budget documented
• [ ] PII auto-capture scrubbed (beforeSend + platform data scrubbing)
• [ ] Three signals verified to join on traceId

Output

• Instrumentation: OTel setup + structured logger + metrics
• Dashboards: RED + USE
• Alert config: docs/observability-alerts.md — SLOs, thresholds, runbooks
• Commit format: feat(obs): wire OpenTelemetry tracing / feat(obs): RED metrics for <service>

Implementation

TypeScript + NestJS (default)

• Tracing: @opentelemetry/sdk-node + auto-instrumentations (HTTP, Prisma, Redis, BullMQ); export to Tempo/Jaeger/Datadog
• Logging: pino (JSON) with a traceId field from the active span context
• Metrics: prom-client exposing /metrics for Prometheus; Grafana dashboards
• Errors: @sentry/node with tracesSampleRate; attach traceId
• Propagate traceId into BullMQ job data

Other stacks

• Python / FastAPI: opentelemetry-instrumentation-fastapi; structlog; prometheus_client
• Go: go.opentelemetry.io/otel; slog (stdlib structured logging); prometheus/client_golang
• Universal: OpenTelemetry + W3C Trace Context are the cross-language standard; RED/USE are methodologies, not tools

Related skills

• performance-profiling — traces/metrics surface the bottlenecks profiling then drills into
• background-jobs — jobs need the same correlation context as requests
• ai-llm-backend — token/cost/latency are AI-specific metrics on this backbone

Reference

• Key insight encoded: Propagate one correlation context (traceId in every log + W3C Trace Context header) so logs/metrics/traces join; apply RED to services, USE to infrastructure. Three cost-and-safety gates often missed: trace sampling (100% is unaffordable at scale), a metric-cardinality budget (one bad label explodes the TSDB), and SDK-level PII scrubbing — the pipeline auto-captures more than you set.