sawrus/slo-sli-design

Skill: SLO/SLI Design

Expertise: SLI selection, SLO target setting, error budget calculation, burn rate alerting, Sloth/pyrra integration.

When to load

When defining SLOs for a new service, setting up error budget tracking, or reviewing existing SLOs after an incident.

SLI Selection Framework

Step 1: What does the user care about?
  → "The checkout completes successfully and quickly"

Step 2: What CAN we measure?
  → HTTP 2xx responses, p99 latency

Step 3: Define the SLI formula
  → Availability SLI: good_requests / total_requests
     where good = status < 500 AND latency < 500ms

Step 4: Pick SLO target (start conservative, tighten later)
  → 99.5% (don't chase 99.99% without data — high budget wasted on caution)

Step 5: Calculate error budget
  → 100% - 99.5% = 0.5% over 28 days = 0.5% × 28 × 24 × 60 = 201.6 minutes

Prometheus SLO Implementation (manual)

# Recording rules for SLO tracking
groups:
  - name: slo.checkout-service
    interval: 30s
    rules:
      # Good requests (2xx, latency < 500ms)
      - record: slo:http_requests_good:rate5m
        expr: |
          sum(rate(http_requests_total{
            service="checkout-service",
            status=~"2..",
            duration_bucket="0.5"
          }[5m]))

      # Total requests
      - record: slo:http_requests_total:rate5m
        expr: |
          sum(rate(http_requests_total{service="checkout-service"}[5m]))

      # SLI = good / total
      - record: slo:http_availability:ratio_rate5m
        expr: slo:http_requests_good:rate5m / slo:http_requests_total:rate5m

      # 28-day rolling availability
      - record: slo:http_availability:ratio_rate28d
        expr: |
          sum_over_time(slo:http_availability:ratio_rate5m[28d]) / (28 * 24 * 12)

Burn Rate Alerts (multiwindow)

# Multi-window, multi-burn-rate alerting (Google SRE Workbook pattern)
groups:
  - name: slo.checkout-service.burn-rate
    rules:
      # Fast burn: 14.4× rate (burns 1h of budget in 5 min)
      - alert: SLOFastBurn
        expr: |
          (
            slo:http_availability:ratio_rate1h{service="checkout-service"} < (1 - 14.4 * 0.005)
          ) and (
            slo:http_availability:ratio_rate5m{service="checkout-service"} < (1 - 14.4 * 0.005)
          )
        labels:
          severity: critical
          slo: checkout-service-availability
        annotations:
          summary: "Fast error budget burn — checkout-service (> 14.4× rate)"
          runbook_url: "https://runbooks.internal/slo-fast-burn"

      # Slow burn: 3× rate (burns 10% of budget in 6h)
      - alert: SLOSlowBurn
        expr: |
          (
            slo:http_availability:ratio_rate6h{service="checkout-service"} < (1 - 3 * 0.005)
          ) and (
            slo:http_availability:ratio_rate30m{service="checkout-service"} < (1 - 3 * 0.005)
          )
        labels:
          severity: warning
          slo: checkout-service-availability
        annotations:
          summary: "Slow error budget burn — checkout-service (> 3× rate)"

Sloth (SLO as Code)

# slo/checkout-service.yaml — Sloth generates all recording rules + alerts
version: "prometheus/v1"
service: checkout-service
labels:
  team: backend
slos:
  - name: requests-availability
    objective: 99.5
    description: "99.5% of checkout requests succeed with latency < 500ms"
    sli:
      events:
        error_query: |
          sum(rate(http_requests_total{service="checkout-service", status=~"5.."}[{{.window}}]))
        total_query: |
          sum(rate(http_requests_total{service="checkout-service"}[{{.window}}]))
    alerting:
      name: CheckoutServiceSLO
      page_alert:
        labels: { severity: critical }
      ticket_alert:
        labels: { severity: warning }

# Generate Prometheus rules from Sloth definition
sloth generate -i slo/checkout-service.yaml -o prometheus-rules/slo-checkout.yaml

Error Budget Dashboard (Grafana)

Key panels:

1. SLI over 28 days — current ratio vs SLO target line
2. Error budget remaining — percentage + time remaining (burn at current rate)
3. Burn rate — 1h, 6h, 1d, 7d windows
4. Events causing budget consumption — top error causes by count