dsivov/production-monitoring

Production Monitoring Expertise

When This Applies

Apply this guidance when:

• Assessing whether a release is production-ready
• Setting up or reviewing health checks
• Evaluating system stability after a deployment
• Defining monitoring and alerting requirements
• Responding to production incidents

Health Check Design

Application Health Endpoint

Every service should expose a health endpoint:

GET /health
Response:
{
  "status": "healthy" | "degraded" | "unhealthy",
  "version": "1.2.3",
  "timestamp": "2026-02-28T10:00:00Z",
  "checks": {
    "database": "healthy",
    "cache": "healthy",
    "external_api": "degraded"
  }
}

Health Check Levels

| Level | What to Check | Frequency | |-------|--------------|-----------| | Liveness | Process is running | Every 10s | | Readiness | Can handle requests | Every 30s | | Deep health | All dependencies ok | Every 60s |

Key Metrics to Monitor

The Four Golden Signals

1. Latency — Response time distribution (p50, p95, p99)
2. Traffic — Request rate (requests/second)
3. Errors — Error rate (5xx responses / total)
4. Saturation — Resource utilization (CPU, memory, disk, connections)

Service-Level Indicators (SLIs)

| SLI | Target | Measurement | |-----|--------|-------------| | Availability | 99.9% | Successful responses / total | | Latency | p95 < 200ms | Response time percentiles | | Error rate | < 0.1% | Error responses / total | | Throughput | > N req/s | Requests per second |

Post-Deployment Verification

After merging to main and deploying:

Immediate (0-5 minutes)

• [ ] Health endpoint returns healthy
• [ ] No error spike in logs
• [ ] Response times are within normal range
• [ ] Key user flows work (manual smoke test)

Short-term (5-30 minutes)

• [ ] Error rate is stable or improving
• [ ] No memory leaks (memory usage stable)
• [ ] No connection pool exhaustion
• [ ] No increase in support tickets or alerts

Medium-term (30 min - 2 hours)

• [ ] Performance metrics are within historical norms
• [ ] Background jobs are processing normally
• [ ] No data integrity issues reported
• [ ] All scheduled tasks ran successfully

Alerting Guidelines

Alert Severity

| Severity | Condition | Response | |----------|-----------|----------| | Critical | Service down, data loss risk | Page on-call, immediate action | | Warning | Degraded performance, elevated errors | Investigate within 30 min | | Info | Anomaly detected, approaching threshold | Review during business hours |

Alert Design Rules

1. Actionable — Every alert should have a clear response action
2. No noise — If an alert fires > 5 times/day without action needed, fix or remove it
3. Contextual — Include what's wrong, what threshold was breached, and where to look
4. Escalation — If not acknowledged within N minutes, escalate to next level

Incident Documentation

After any production incident, document in reports/INCIDENT_<YYYYMMDD>_<NNN>.md:

# Incident Report — <date>

## Summary
<One-line description of what happened>

## Timeline
- HH:MM — Issue detected
- HH:MM — Investigation started
- HH:MM — Root cause identified
- HH:MM — Fix applied
- HH:MM — Verified resolved

## Impact
- Duration: X minutes
- Users affected: N
- Data impact: None / Describe

## Root Cause
<What caused the issue>

## Resolution
<How it was fixed>

## Prevention
<What changes will prevent recurrence>