asiaostrich/observability

Observability Assistant | 可觀測性助手

Language: English | 繁體中文

Version: 1.0.0 Last Updated: 2026-06-19 Applicability: Claude Code Skills

Core Standard: This skill implements Observability Standards. For the authoritative methodology (full Metrics/Traces detail, sampling, OTel integration), refer to the core standard.

Guide observability implementation across the three pillars: Logs, Metrics, and Traces.

引導三支柱可觀測性實作：Logs、Metrics、Traces。

Capabilities | 功能

| Capability | Description | 說明 | |------------|-------------|------| | Instrumentation Check | Pre-launch observability checklist | 上線前可觀測性檢查表 | | Maturity Assessment | L0-L4 maturity self-evaluation | L0-L4 成熟度自評 | | Metric Design | Help design metrics (type, naming, labels) | 協助設計 Metrics | | Alert Design | Design SLO-based alerts with noise reduction | 設計 SLO-based 告警 | | Golden Signals | Verify 4 golden signals coverage | 驗證四大黃金信號覆蓋 |

Usage | 使用方式

/observability                        # Show observability guide
/observability --checklist            # Run instrumentation checklist
/observability --maturity             # Maturity assessment (L0-L4)
/observability --alerting             # Alert design guide
/observability "payment-service"      # Guide for specific service

Three Pillars Framework | 三支柱框架

Each pillar gives a different lens; their power is in correlation.

| Pillar | What It Captures | When to Use | Granularity | |--------|-----------------|-------------|-------------| | Logs | Discrete events with context | Debugging, audit trails, error details | High (per-event) | | Metrics | Numerical measurements over time | Dashboards, alerting, capacity planning | Low (aggregated) | | Traces | Request flow across services | Latency analysis, dependency mapping | Medium (per-request) |

Correlation fields: trace_id links Logs ↔ Traces ↔ Metrics (via Exemplars); service.name filters all three pillars. Workflow: metric anomaly → exemplar → trace → trace_id in logs.

Golden Signals | 四大黃金信號

Based on Google SRE. Every service SHOULD monitor all four before production.

| Signal | Measure | Example Metric | Alert (SLO-based) | |--------|---------|----------------|-------------------| | Latency | P50/P95/P99 via Histogram, split success/error | http.server.request.duration.seconds | P99 > X ms for 5 min | | Traffic | Requests/sec, by route/method | http.server.request.total (rate) | drop > 50% or spike > 200% | | Errors | errors / total requests | ...request.total{status=~"5.."} ÷ total | error rate > X% for 5 min | | Saturation | CPU/mem/pool/disk utilization | system.cpu.utilization (Gauge) | resource > 80% for 10 min |

Metric Types | Metric 型別

| Type | Behavior | Use When | |------|----------|----------| | Counter | Only goes up (resets on restart) | request count, error count, bytes sent | | Gauge | Point-in-time, up/down | queue depth, active connections, memory | | Histogram | Distribution across buckets | request duration, response size | | Summary | Client-computed percentiles | legacy, no server-side aggregation |

Naming: <domain>.<entity>.<action>.<unit> in snake_case (e.g. db.client.query.duration.seconds). Label cardinality: keep labels under ~1000 unique values — never use user_id / request_id / raw url / ip as labels; record those in Logs or Traces instead.

Maturity Model (L0–L4) | 成熟度模型

| Level | Name | Characteristics | Upgrade Action | |-------|------|-----------------|----------------| | L0 | No Observability | only stdout/stderr; debug via SSH + tail -f | structured logging; centralize collection | | L1 | Basic Logging | structured JSON logs, centralized, searchable | add business metrics; first dashboard | | L2 | Metrics-Driven | Logs + Metrics, dashboards, threshold alerts | enable tracing; SLO-based alerting | | L3 | Full Observability | three pillars + correlation + SLO alerts + Golden Signals | anomaly detection; auto-remediation | | L4 | Intelligent | AIOps anomaly detection, predictive alerts, auto-remediation | maintain, optimize, share learnings |

Self-check: find logs for one request across services (L1+) → dashboards of error rate & latency (L2+) → trace request ingress→DB→back (L3+) → auto-detect anomalies before users report (L4).

Instrumentation Checklist | 上線前檢查表

Before deploying a service to production:

• [ ] Structured logging — JSON with trace_id correlation
• [ ] HTTP/gRPC entry metrics — request count, duration histogram, error rate per endpoint
• [ ] Business operation metrics — key events tracked (orders, payments)
• [ ] Distributed tracing — span propagation, sampling rate, W3C Trace Context headers
• [ ] Health checks — liveness (running?) and readiness (can serve?) probes
• [ ] Golden Signals dashboard — Latency, Traffic, Errors, Saturation visualized
• [ ] Alert rules — at minimum SLO burn-rate alerts
• [ ] Log retention — policies set per log level
• [ ] Sensitive data excluded — no passwords, tokens, or PII in logs/traces

Next Steps Guidance | 下一步引導

可觀測性引導完成。建議下一步：

• 執行 /slo 定義 SLI/SLO/Error Budget ⭐ 推薦
• 執行 /incident 設定事故回應流程
• 執行 /checkin 提交變更

Reference | 參考

• Core standard: observability-standards.md
• Core standard: alerting-standards.md
• Core standard: slo-standards.md