sawrus/areas/software/platform/skills/incident-response

Skill: Incident Response Runbooks

When to load

When responding to a production alert, diagnosing an outage, or writing a postmortem.

Severity Classification

| Severity | Definition | Response Time | |:---|:---|:---| | P0 | Complete outage, data loss | Immediate | | P1 | Significant degradation, key feature broken | 15 min | | P2 | Minor degradation, workaround exists | 1 hour | | P3 | Non-user-facing | Next business day |

P0 Response Playbook

T+0:  ACKNOWLEDGE — "I'm on this"
T+5:  SCOPE — What's broken? Who's affected? Since when?
T+10: COMMUNICATE — Post status page update; notify stakeholders
T+15: MITIGATE — Rollback > fix. Prefer reversible actions.
       Order: rollback deploy → feature flag off → scale up → redirect traffic
T+30: STABILIZE — Confirm metrics returning to normal
T+60: DOCUMENT — Write preliminary postmortem
T+24h: POSTMORTEM — Full 5-whys analysis, action items

Common Runbooks

# High error rate: check recent deploys
kubectl rollout history deployment/api
kubectl logs -l app=api --since=10m | grep ERROR | tail -50
kubectl rollout undo deployment/api  # If recent deploy

# DB connection exhaustion
psql -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"
psql -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity
         WHERE state = 'idle' AND age(clock_timestamp(), state_change) > interval '10 minutes';"