qa-aman/runbook

Overview

Based on the Google SRE Book. A runbook is the operational contract for a service: how to start it, stop it, diagnose it, and recover it. Google SRE's standard: a runbook should be executable by an on-call engineer who has never seen the service before. If it requires tribal knowledge, it's not a runbook - it's an assumption.

The test: can an on-call engineer follow this runbook at 3am with a degraded service, no time to research, and adrenaline affecting their cognition? If not, it's not operational-grade.

Workflow

Step 1: Write the service overview

Service: [your service]
Owner team: [your team]
On-call rotation: [who gets paged]
SLO: [availability target, e.g. 99.9% uptime]
Dependencies: [upstream services this depends on]
Dependents: [downstream services that depend on this]
Dashboards: [links]
Logs: [how to access]
Alerts: [link to alerting config]

Step 2: Document steady-state operations

What does normal look like?

Health check:

# Command to verify the service is healthy
curl https://[your service]/health
# Expected response: {"status": "ok", "version": "x.y.z"}

Key metrics (normal ranges):

| Metric | Normal | Warning | Critical | |--------|--------|---------|----------| | Error rate | < 0.1% | 0.1-1% | > 1% | | p99 latency | < 200ms | 200-500ms | > 500ms | | CPU usage | < 60% | 60-80% | > 80% |

Step 3: Document alert runbooks

For each alert, write a response procedure:

Alert: [Alert name]
Severity: [SEV1/SEV2/SEV3]
Fires when: [condition]
Impact: [what users experience]

Diagnosis:
  1. Check [dashboard link] for [specific signal]
  2. Run: [command] - Expected: [output]
  3. Check: [logs location] for [error pattern]

Common causes:
  A. [Cause] - Fix: [command or action]
  B. [Cause] - Fix: [command or action]

If none of the above: escalate to [team/person]

Step 4: Document common operations

List every routine operational task with exact commands:

Restart the service:

kubectl rollout restart deployment/[service-name] -n [namespace]
kubectl rollout status deployment/[service-name] -n [namespace]

Scale up:

kubectl scale deployment/[service-name] --replicas=[n] -n [namespace]

Rollback a deploy:

kubectl rollout undo deployment/[service-name] -n [namespace]
# Verify rollback succeeded:
kubectl rollout status deployment/[service-name] -n [namespace]

Step 5: Document recovery procedures

For each failure mode, step-by-step recovery:

Failure: [service is unresponsive / returning 5xx / high latency]
Verify: [how to confirm this is the problem]
Recover:
  1. [first action]
  2. [second action]
  3. [verify recovery: command - expected output]
Escalate if: [condition that means this runbook is insufficient]

Step 6: Document escalation paths

Who to call when the runbook doesn't work:

| Situation | Contact | How | |-----------|---------|-----| | Database issue | [DB team] | [PagerDuty / Slack] | | Network issue | [Infra team] | [contact] | | Vendor outage | [Vendor support] | [ticket URL] |

Step 7: Add a testing section

How does an on-call engineer practice with this runbook in a non-emergency?

• Chaos engineering scenarios
• Runbook review frequency (recommend: quarterly)
• Last tested: [date]

Anti-Patterns

1. Runbook that requires tribal knowledge Bad: "Check if the DB is having issues." (how? where? what to look for?) Good: "Run psql -h [host] -U [user] -c 'SELECT count(*) FROM pg_stat_activity' - if > 100 connections, connection pool exhaustion is likely."

2. Commands without expected output Bad: "Run the health check command." Good: "Run: curl https://[your service]/health - Expected: {"status":"ok"}. If you see {"status":"degraded"}, proceed to Step 3."

3. No escalation path Bad: Runbook ends with "investigate further." Good: Every runbook has an explicit "escalate if" condition and names who to escalate to.

4. Runbook never tested Bad: Runbook written once, never validated in a real incident. Good: Run quarterly fire drills. Note the last tested date in the runbook.

Quality Checklist

• [ ] Service overview complete (owner, SLO, dependencies, links)
• [ ] Normal operating metrics defined with ranges
• [ ] Every alert has a response procedure with exact commands
• [ ] Commands include expected outputs (not just what to run)
• [ ] Common operations documented step-by-step
• [ ] Recovery procedures cover the top 3 failure modes
• [ ] Escalation path named and reachable
• [ ] Last tested date recorded
• [ ] Executable at 3am by someone unfamiliar with the service