kienbui1995/skills/cloud/gcp/cloud-monitoring

Cloud Monitoring

When to Use

• Setting up observability for GCP services
• Designing SLO/SLI framework and error budgets
• Troubleshooting production issues with logs and traces
• Preparing for GCP Professional Cloud DevOps Engineer exam

Core Jobs

1. Observability Stack

| Tool | Purpose | |------|---------| | Cloud Monitoring | Metrics, dashboards, alerting, uptime checks | | Cloud Logging | Log ingestion, storage, search, routing | | Error Reporting | Auto-groups errors from logs; shows frequency/trend | | Cloud Trace | Distributed tracing; latency analysis across services | | Cloud Profiler | CPU and memory profiling for production code |

2. SLO/SLI Design

• SLI — what you measure (e.g., % of requests < 200ms)
• SLO — target for the SLI (e.g., 99.9% of requests < 200ms)
• Error budget — allowed failures = 1 - SLO (e.g., 0.1% = 43.8 min/month)
• Create SLOs in Cloud Monitoring with time-series data
• Alert on error budget burn rate (not raw error rate)

3. Alerting Policies

• Condition — metric threshold, absence, or rate of change
• Notification channels — email, PagerDuty, Slack, Pub/Sub, Cloud Functions
• Alert duration — condition must be true for N minutes before firing
• Avoid alert fatigue: alert on symptoms (slow SLO burn) not causes (high CPU)

4. Log-Based Metrics

• Create custom metrics from log entries matching a filter
• Counter metric: count occurrences of matching log entries
• Distribution metric: extract numeric value from log field (e.g., latency)
• Use for: custom application metrics without SDK instrumentation

5. Uptime Checks

• HTTP/HTTPS/TCP checks from multiple global locations
• Alert if check fails from N locations
• Use for: external SLA monitoring, health check dashboards

Key Concepts

• Workspace — Cloud Monitoring scope; can monitor multiple projects
• Log sink — route logs to BigQuery/GCS/Pub/Sub for long-term storage/analysis
• Log exclusions — reduce logging costs by excluding verbose/low-value logs
• Structured logging — JSON logs; query specific fields in Log Explorer

Checklist

• [ ] SLO defined for each user-facing service?
• [ ] Alerts on error budget burn rate (not raw metrics)?
• [ ] Log-based metrics for custom application events?
• [ ] Uptime checks configured for external endpoints?
• [ ] Log sinks to BigQuery for audit/compliance logs?
• [ ] Traces enabled for all services in a request path?

Output Format

• 🔴 Critical — no alerting on user-facing SLOs, no structured logging (hard to query)
• 🟡 Warning — alerts on CPU/memory (causes) instead of latency/errors (symptoms)
• 🟢 Suggestion — error budget burn rate alerts, Cloud Profiler for production optimization

Exam Tips

• SLI = metric; SLO = target; Error budget = 1 - SLO (allowed downtime)
• Alert on burn rate (how fast error budget is consumed) not raw error rate
• Log-based metrics = create Cloud Monitoring metrics from log entries (no SDK needed)
• Cloud Trace = distributed tracing; find slow span in a microservice call chain
• Error Reporting auto-detects exceptions from logs (no explicit integration usually needed)
• LOG_ID filter = filter specific audit log types in Cloud Logging