michaelsvanbeek/logging-metrics

Logging and Metrics Standards

When to Use

• Adding logging to any application or service
• Designing log schemas for analytics pipelines
• Implementing exception logging with safe context
• Adding metrics or tracing to services
• Integrating with dashboards (Grafana, CloudWatch, Datadog)
• Auditing existing logging for missing structure, inconsistent fields, or sensitive data leakage
• Upgrading unstructured print/log statements to structured JSON logging
• Reviewing observability coverage (are errors, latency, and key operations instrumented?)

Core Principle: Logs are Data

Logs are not debug printf statements. They are structured data events that feed analytics pipelines, dashboards, alerts, and incident investigation. Design them like you would design a database schema.

---

Structured Logging

All logs must be JSON-formatted, one JSON object per line. Never use unstructured text, multi-line log messages, or custom delimiters.

Standard Fields

Every log entry must include these base fields:

| Field | Type | Description | |-------|------|-------------| | timestamp | string | ISO 8601 with timezone: 2026-03-19T23:45:00.123Z | | level | string | DEBUG, INFO, WARNING, ERROR, CRITICAL | | service | string | Service/application name | | environment | string | dev, staging, prod | | message | string | Human-readable description of the event |

Contextual Fields (add when applicable)

| Field | Type | When to include | |-------|------|-----------------| | request_id | string (UUID) | Any event within an HTTP request lifecycle | | user_id | string | Any event tied to a user action (never log PII beyond ID) | | trace_id | string | Distributed tracing across services | | span_id | string | Sub-operation within a trace | | route | string | HTTP endpoint path | | method | string | HTTP method | | status | integer | HTTP response status code | | duration_ms | number | Operation timing | | component | string | Module, class, or function name | | action | string | What the code is doing: fetch_users, send_email, sync_cache |

Event-Specific Fields

Add fields relevant to the specific event. Use consistent names across the codebase:

{
  "timestamp": "2026-03-19T23:45:00.123Z",
  "level": "INFO",
  "service": "project-api",
  "environment": "prod",
  "request_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "action": "create_project",
  "user_id": "usr_42",
  "project_id": "prj_789",
  "duration_ms": 142.5,
  "message": "Project created successfully"
}

Field Naming Conventions

• Use snake_case for all field names.
• Use consistent suffixes: _id for identifiers, _ms for milliseconds, _at for timestamps, _count for counts, _bytes for sizes.
• Never change the type of a field. If duration_ms is a number in one log, it must always be a number.
• Avoid nested objects in log fields. Flatten for easier querying: user_id not user.id.

---

Log Levels

Use levels consistently across all services:

| Level | When to use | Example | |-------|-------------|---------| | DEBUG | Detailed diagnostic info for development. Disabled in production. | "Resolved user cache key: usr_42" | | INFO | Normal operations, milestones, state transitions. | "Project created", "Sync completed: 42 items" | | WARNING | Recoverable issue that may need attention. | "Rate limit approaching: 85% of quota" | | ERROR | Operation failed but service continues. | "Failed to send notification email" | | CRITICAL | Service-level failure, requires immediate action. | "Database connection pool exhausted" |

Rules:

• INFO is the default production level. Every significant operation should produce at least one INFO log.
• A request should produce at most 1 INFO log for the happy path (the request summary). Use DEBUG for steps within the request.
• ERROR means something broke. If it's expected behavior (e.g., user not found), use INFO or WARNING.
• Never log at ERROR for client errors (4xx). Log at WARNING for unexpected 4xx, INFO for expected ones.

---

Exception Logging

Include Context

When logging exceptions, include enough context to reproduce and diagnose the issue without accessing the running system:

import logging

logger = logging.getLogger(__name__)

try:
    result = process_order(order_id=order_id, user_id=user_id)
except ExternalServiceError as exc:
    logger.error(
        "Order processing failed",
        extra={
            "action": "process_order",
            "order_id": order_id,
            "user_id": user_id,
            "error_type": type(exc).__name__,
            "error_message": str(exc),
            "retry_count": attempt,
        },
        exc_info=True,
    )
    raise

What to Include in Exception Logs

• action: What operation was being attempted
• error_type: Exception class name
• error_message: Exception message text
• identifiers: IDs of the entities involved (order_id, user_id, etc.)
• state: Relevant state at time of failure (retry_count, batch_index, etc.)
• stack trace: Via exc_info=True (Python) or error.stack (JS). Include in dev/staging; optionally truncate in production.

What to NEVER Include

• Passwords, tokens, API keys, secrets in any form
• Credit card numbers, SSNs, or other PII
• Full request/response bodies that may contain user data
• Database connection strings with credentials
• Environment variables that contain secrets
• Email addresses, phone numbers unless essential and privacy-compliant

Sanitization

Build a sanitization layer that runs before logging:

SENSITIVE_KEYS = {"password", "token", "secret", "api_key", "authorization", "credit_card", "ssn"}

def sanitize_context(context: dict) -> dict:
    """Remove or mask sensitive fields before logging."""
    return {
        k: "***REDACTED***" if any(s in k.lower() for s in SENSITIVE_KEYS) else v
        for k, v in context.items()
    }

For JavaScript/TypeScript:

const SENSITIVE_PATTERNS = /password|token|secret|api_key|authorization|credit_card|ssn/i;

function sanitize(context: Record<string, unknown>): Record<string, unknown> {
  return Object.fromEntries(
    Object.entries(context).map(([k, v]) =>
      SENSITIVE_PATTERNS.test(k) ? [k, "***REDACTED***"] : [k, v],
    ),
  );
}

---

Python Logging Setup

Use the stdlib logging module with a JSON formatter. Never use print().

import json
import logging
import os
import sys
from datetime import datetime, timezone


class JSONFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        entry = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "level": record.levelname,
            "service": os.environ.get("SERVICE_NAME", "unknown"),
            "environment": os.environ.get("ENVIRONMENT", "dev"),
            "component": record.name,
            "message": record.getMessage(),
        }
        # Merge extra fields
        if hasattr(record, "__dict__"):
            for key, value in record.__dict__.items():
                if key not in logging.LogRecord.__dict__ and key not in entry:
                    entry[key] = value
        # Include exception info
        if record.exc_info and record.exc_info[1]:
            entry["error_type"] = type(record.exc_info[1]).__name__
            entry["error_message"] = str(record.exc_info[1])
            entry["traceback"] = self.formatException(record.exc_info)
        return json.dumps(entry, default=str)


def setup_logging(level: str = "INFO") -> None:
    handler = logging.StreamHandler(sys.stdout)
    handler.setFormatter(JSONFormatter())
    logging.root.handlers = [handler]
    logging.root.setLevel(getattr(logging, level.upper()))

---

JavaScript/TypeScript Logging

Use a structured logger like pino for Node.js services:

import pino from "pino";

const logger = pino({
  level: process.env.LOG_LEVEL ?? "info",
  formatters: {
    level: (label) => ({ level: label.toUpperCase() }),
  },
  base: {
    service: process.env.SERVICE_NAME ?? "unknown",
    environment: process.env.ENVIRONMENT ?? "dev",
  },
  timestamp: pino.stdTimeFunctions.isoTime,
});

// Usage
logger.info({ action: "create_project", project_id: "prj_789", duration_ms: 142 }, "Project created");

---

Metrics

What to Measure

Instrument the Four Golden Signals for every service:

| Signal | Metric | Example | |--------|--------|---------| | Latency | Request duration | p50, p95, p99 of response time in ms | | Traffic | Request throughput | Requests per second by endpoint | | Errors | Failure rate | Errors per second, error rate percentage | | Saturation | Resource utilization | CPU, memory, connection pool usage, queue depth |

Metric Naming

Use a consistent naming scheme:

{service}_{component}_{metric}_{unit}

Examples:

• api_http_request_duration_ms
• api_http_request_total
• api_http_error_total
• worker_queue_depth_count
• cache_hit_total, cache_miss_total

Metric Types

| Type | Use when | Example | |------|----------|---------| | Counter | Value only increases | Total requests, total errors | | Gauge | Value goes up and down | Queue depth, active connections, memory usage | | Histogram | Distribution of values | Request latency, response size |

Dashboard Design

• Every service needs a dashboard with at minimum: request rate, error rate, latency (p50/p95/p99), and saturation metrics.
• Align log fields with dashboard dimensions so you can drill from dashboard → filtered log view.
• Use the same service, environment, route, and status field names in both logs and metrics for join-ability.

---

Log Pipeline Integration

Design logs to flow through analytics pipelines:

Application → stdout (JSON) → Log collector → Storage → Query/Dashboard

• Write logs to stdout, never to files. Let the runtime (Docker, Lambda, systemd) handle routing.
• One JSON object per line. No multi-line log entries.
• Keep log entries under 4KB for compatibility with most log aggregation services.
• Include all dimensions needed for filtering in the log entry itself. Don't rely on external metadata enrichment for core fields.
• Use consistent field names across all services so dashboards and alerts work across the fleet.

---

Correlation

• Generate a request_id (UUID) at the entry point of every request. Pass it through all downstream calls and include it in every log entry for that request.
• For distributed systems, propagate trace_id and span_id in headers (W3C Trace Context or similar).
• Log the request_id in API error responses so users can report it for debugging: {"error": "internal_error", "request_id": "abc-123"}.

---

Alerting

What to Alert On

Alert on symptoms, not causes. Page when user-facing behavior is degraded, not on every noisy internal signal.

| Signal | Condition | Severity | |--------|-----------|----------| | Error rate | >1% of requests return 5xx over a 5-minute window | PAGE | | Error rate | >0.1% sustained over 30 minutes | WARN | | p95 latency | >2× baseline over a 10-minute window | PAGE | | p95 latency | >1.5× baseline over 30 minutes | WARN | | Queue depth | >80% of max capacity | WARN | | Queue depth | >95% of max capacity (unbounded growth) | PAGE | | Availability | Health check fails for 2 consecutive minutes | PAGE | | Error budget | SLO error budget <20% remaining in billing period | WARN |

Alert Fatigue Prevention

An alert that fires constantly becomes noise — it trains engineers to ignore it.

• PAGE only on actionable, user-impacting conditions. If the on-call engineer can't do anything about it at 2am, it must not be a page.
• Every alert must link to a runbook in the alert description. No runbook = no alert.
• Use warning notifications (Slack, email) for elevated-but-not-critical conditions. Reserve PagerDuty/phone for true incidents.
• Set thresholds above the noise floor — measure 2 weeks of baseline before configuring thresholds. An alert firing 10 times a day from normal traffic is useless.
• Use multi-condition alerts — require traffic above a minimum AND elevated error rate. One error in 1 request/hour is not an incident.
• Use time windows — 5-minute windows for page-severity, 30-minute windows for warn-severity. Short windows are noisy.
• Review and prune quarterly — remove any alert that fired without resulting in a real incident.

AWS CloudWatch Alarm Example

resources:
  Resources:
    LambdaErrorRateAlarm:
      Type: AWS::CloudWatch::Alarm
      Properties:
        AlarmName: ${self:service}-${self:custom.stage}-error-rate
        AlarmDescription: "Lambda errors >5 in 5 min. Runbook: docs/runbooks/lambda-errors.md"
        MetricName: Errors
        Namespace: AWS/Lambda
        Dimensions:
          - Name: FunctionName
            Value: !Ref ApiLambdaFunction
        Statistic: Sum
        Period: 300
        EvaluationPeriods: 1
        Threshold: 5
        ComparisonOperator: GreaterThanOrEqualToThreshold
        TreatMissingData: notBreaching
        AlarmActions: [!Ref OpsAlertTopic]
        OKActions: [!Ref OpsAlertTopic]

    LambdaLatencyAlarm:
      Type: AWS::CloudWatch::Alarm
      Properties:
        AlarmName: ${self:service}-${self:custom.stage}-latency-p95
        AlarmDescription: "Lambda p95 latency >2000ms. Runbook: docs/runbooks/latency.md"
        MetricName: Duration
        Namespace: AWS/Lambda
        Dimensions:
          - Name: FunctionName
            Value: !Ref ApiLambdaFunction
        ExtendedStatistic: p95
        Period: 600
        EvaluationPeriods: 1
        Threshold: 2000
        ComparisonOperator: GreaterThanOrEqualToThreshold
        TreatMissingData: notBreaching
        AlarmActions: [!Ref OpsAlertTopic]

    OpsAlertTopic:
      Type: AWS::SNS::Topic
      Properties:
        TopicName: ${self:service}-${self:custom.stage}-ops-alerts
        # Subscribe Slack webhook via Lambda or PagerDuty endpoint via HTTPS subscription

Grafana Alerting

For services queried via Grafana (Athena, InfluxDB, Prometheus):

• Create alert rules directly on dashboard panels — keep alerts and visualization co-located.
• Use contact points to route by severity: severity=page → PagerDuty, severity=warn → Slack.
• Use notification policies with label matchers to enforce routing without duplicating contact config.
• Set a pending duration (5 minutes minimum) before firing — eliminates single-spike false positives.
• Use silence rules during planned maintenance windows rather than disabling alert rules.

---

Further Reading

• Compare observability stacks using explicit criteria: ingestion cost, query latency, retention, alerting quality, and operational overhead.
• Record tool choices in ADRs so future maintainers understand trade-offs and revisit assumptions when scale or budget changes.