nbbaier/logging-best-practices

This skill is adpated from "Logging sucks. And here's how to make it better. by Boris Tane.

When helping with logging, observability, or debugging strategies, follow these principles:

Core Philosophy

• Logs are optimized for querying, not writing — always design with debugging in mind
• Context is everything — a log without correlation IDs is useless in distributed systems
• Logs are for humans during incidents, not just for compliance or "just in case"
• If you can't filter and search your logs effectively, they provide zero value

Structured Logging Requirements

• Always use key-value pairs (JSON) instead of string interpolation
• Bad: "Payment failed for user 123"
• Good: {"event": "payment_failed", "user_id": "123", "reason": "insufficient_funds", "amount": 99.99}
• Structured logs are machine-parseable, enabling aggregation, alerting, and dashboards

Required Fields for Every Log Event

• timestamp — ISO 8601 with timezone (e.g., 2025-01-24T20:00:00Z)
• level — debug, info, warn, error (be consistent, don't invent new levels)
• event — machine-readable event name, snake_case (e.g., user_login_success)
• request_id or trace_id — for correlating logs across a single request
• service — which service/application emitted this log
• environment — prod, staging, dev

Examples of High-Cardinality Fields (always include when available):

• user_id, org_id, account_id — who is affected
• request_id, trace_id, span_id — for distributed tracing
• order_id, transaction_id, job_id — domain-specific identifiers

These fields are what make logs actually queryable during incidents. Without them, you're grepping through millions of lines blindly.

Look for opportunities for high-cardinality fields that can help you identify the root cause of an issue quickly.

Context Propagation

• Pass trace/request IDs through all service boundaries (HTTP headers, message queues, etc.)
• Downstream services must inherit correlation IDs from upstream
• Use middleware or interceptors to automatically inject context into every log
• For async jobs, store and restore the original request context

Log Levels — Use Them Correctly:

• debug — Verbose details for local development, usually disabled in production
• info — Normal operations worth recording (user actions, job completions, deploys)
• warn — Something unexpected happened but the system handled it (retries, fallbacks)
• error — Something failed and likely needs human attention (exceptions, failed requests)

Don't log errors for expected conditions (e.g., user enters wrong password)

What to Log:

• Request entry and exit points (with duration)
• State transitions (order created → paid → shipped)
• External service calls (with latency and response codes)
• Authentication and authorization events
• Background job starts, completions, and failures
• Retry attempts and circuit breaker state changes

What NOT to Log:

• Sensitive data (passwords, tokens, PII, credit card numbers)
• Logs inside tight loops (will generate millions of useless entries)
• Success cases that provide no debugging value
• Redundant information already captured by infrastructure (load balancer logs, etc.)

Naming Conventions:

• Be consistent across all services — agree on field names as a team
• Use snake_case for field names: user_id, not userId or user-id
• Use past-tense verbs for events: payment_completed, not complete_payment
• Prefix events by domain when helpful: auth.login_failed, billing.invoice_created

Performance Considerations:

• Use sampling for high-volume debug logs in production
• Avoid logging inside hot paths unless absolutely necessary
• Buffer and batch log writes to reduce I/O overhead
• Consider log levels that can be changed at runtime without redeploying

During Incidents:

• Your logs should answer: Who was affected? What failed? When? Why?
• If you can't answer these within 5 minutes of querying, your logging strategy needs work
• Post-incident: add the logs you wished you had