thesoftwarehouse/tsh-implementing-observability

Observability Patterns

When to Use

• Setting up monitoring and alerting for applications
• Implementing centralized logging
• Adding distributed tracing to microservices
• Designing SLOs/SLIs and error budgets
• Creating dashboards and runbooks

Three Pillars of Observability

| Pillar | Purpose | Tools | |--------|---------|-------| | Metrics | Quantitative measurements over time | Prometheus, CloudWatch, Datadog, Grafana | | Logs | Discrete events with context | ELK, Loki, CloudWatch Logs, Splunk | | Traces | Request flow across services | Jaeger, Zipkin, X-Ray, Tempo |

Stack Detection

Check which observability stack the project uses:

• prometheus.yml or ServiceMonitor → Prometheus
• fluent-bit.conf or fluentd.conf → Fluent Bit/Fluentd
• otel-collector-config.yaml → OpenTelemetry
• AWS with aws_cloudwatch_* resources → CloudWatch
• datadog-agent or DD_* env vars → Datadog

Use context7 to look up stack-specific configuration syntax.

Solution Decision Matrix

Metrics Stack

| Scenario | Recommended Solution | |----------|---------------------| | Kubernetes-native, cost-sensitive | Prometheus + Grafana | | AWS-native, simple setup | CloudWatch Metrics | | Multi-cloud, enterprise | Datadog or New Relic | | OpenTelemetry-first | Prometheus with OTLP receiver |

Logging Stack

| Scenario | Recommended Solution | |----------|---------------------| | Kubernetes, cost-sensitive | Loki + Grafana | | AWS-native | CloudWatch Logs | | High volume, complex queries | Elasticsearch (ELK) | | Multi-cloud, managed | Datadog Logs or Splunk |

Tracing Stack

| Scenario | Recommended Solution | |----------|---------------------| | Kubernetes, open-source | Jaeger or Tempo | | AWS-native | X-Ray | | Multi-cloud, correlated | Datadog APM | | Vendor-agnostic | OpenTelemetry → any backend |

Kubernetes Observability Pattern

┌─────────────────────────────────────────────────────┐
│                   Applications                      │
│  (instrumented with OpenTelemetry SDK or auto-inst) │
└──────────────────────┬──────────────────────────────┘
                       │ OTLP
                       ▼
┌─────────────────────────────────────────────────────┐
│            OpenTelemetry Collector                  │
│  (receives, processes, exports telemetry)           │
└───────┬─────────────────┬─────────────────┬─────────┘
        │                 │                 │
        ▼                 ▼                 ▼
   Prometheus          Loki             Tempo/Jaeger
   (metrics)          (logs)            (traces)
        │                 │                 │
        └────────────────┬┴─────────────────┘
                         ▼
                      Grafana
                   (visualization)

SLO/SLI Framework

Key Metrics (RED Method for Services)

| Metric | Description | Example SLI | |--------|-------------|-------------| | Rate | Requests per second | rate(http_requests_total[5m]) | | Errors | Failed requests | rate(http_requests_total{status=~"5.."}[5m]) | | Duration | Latency distribution | histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) |

Key Metrics (USE Method for Resources)

| Metric | Description | Example | |--------|-------------|---------| | Utilization | % time resource is busy | CPU usage, memory usage | | Saturation | Queue depth, waiting | Pod pending, connection pool | | Errors | Error count | OOM kills, disk errors |

SLO Definition Template

# Example: API availability SLO
slo:
  name: api-availability
  description: "API returns successful responses"
  sli:
    metric: |
      sum(rate(http_requests_total{status!~"5.."}[5m]))
      /
      sum(rate(http_requests_total[5m]))
  target: 99.9%
  window: 30d
  error_budget: 0.1%  # ~43 minutes/month downtime allowed

Alerting Strategy

Alert Severity Levels

| Severity | Response | Example | |----------|----------|---------| | Critical | Page on-call immediately | Service down, data loss risk | | Warning | Investigate within hours | Error rate elevated, disk 80% | | Info | Review during business hours | Deployment completed, scaling event |

Alert Quality Rules

• Actionable: Every alert must have a clear response action
• Relevant: Alert on symptoms (user impact), not causes
• Unique: Avoid duplicate alerts for same incident
• Timely: Alert early enough to prevent impact

Alert Template (Prometheus)

groups:
  - name: api-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m])) > 0.01
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value | humanizePercentage }} (threshold: 1%)"
          runbook_url: "https://runbooks.example.com/high-error-rate"

Structured Logging

Log Format (JSON)

{
  "timestamp": "2024-01-15T10:30:00Z",
  "level": "error",
  "message": "Payment processing failed",
  "service": "payment-api",
  "trace_id": "abc123",
  "span_id": "def456",
  "user_id": "user-789",
  "error": {
    "type": "PaymentGatewayError",
    "message": "Connection timeout"
  },
  "context": {
    "payment_id": "pay-123",
    "amount": 99.99
  }
}

Required Log Fields

| Field | Purpose | Correlation | |-------|---------|-------------| | timestamp | When event occurred | Time-based queries | | level | Severity (debug/info/warn/error) | Filtering | | service | Source service name | Service filtering | | trace_id | Distributed trace identifier | Cross-service correlation | | message | Human-readable description | Search |

Process

1. Discover context → Check existing observability setup (Prometheus, CloudWatch, etc.)
2. Choose stack → Use decision matrix based on environment and requirements
3. Instrument apps → Add OpenTelemetry SDK or auto-instrumentation
4. Configure collection → Set up collectors, exporters, and storage
5. Define SLOs → Establish SLIs, targets, and error budgets
6. Create alerts → Implement actionable alerts with runbooks
7. Build dashboards → Create service and infrastructure dashboards
8. Document runbooks → Write response procedures for each alert

Checklist

• [ ] All services emit metrics, logs, and traces
• [ ] Trace IDs propagated across service boundaries
• [ ] Structured logging with consistent format (JSON)
• [ ] SLOs defined with error budgets
• [ ] Alerts are actionable with runbook links
• [ ] Dashboards show service health at a glance
• [ ] Log retention policy configured
• [ ] PII/sensitive data excluded from logs
• [ ] On-call rotation defined for critical alerts

Anti-Patterns

| Don't | Do | |-------|-----| | Alert on every metric threshold | Alert on user-impacting symptoms | | Log everything at DEBUG in production | Use appropriate log levels | | Unstructured log messages | Structured JSON logging | | Missing trace context | Propagate trace IDs across services | | Dashboards with 50+ panels | Focused dashboards per service/domain | | Alerts without runbooks | Every alert links to response procedure | | Store logs indefinitely | Define retention based on compliance needs |

Related Skills

• tsh-implementing-kubernetes - For K8s-native observability setup
• tsh-implementing-ci-cd - For pipeline observability integration
• tsh-managing-secrets - For secure credential storage for observability tools