rio/analyzing-k8s-events

Analyzing Kubernetes Events

Investigates cluster and namespace events to build timeline and identify issues.

Quick Event Commands

# Recent events in namespace (sorted by time)
kubectl get events -n <ns> --sort-by=.lastTimestamp

# Recent events cluster-wide
kubectl get events -A --sort-by=.lastTimestamp | head -50

# Warning events only
kubectl get events -n <ns> --field-selector type=Warning

# Events in last hour
kubectl get events -n <ns> --sort-by=.lastTimestamp | head -30

Filtering Events

By Resource

# Events for specific pod
kubectl get events -n <ns> --field-selector involvedObject.name=<pod>

# Events for specific deployment
kubectl get events -n <ns> --field-selector involvedObject.name=<deployment>,involvedObject.kind=Deployment

# Events for specific PVC
kubectl get events -n <ns> --field-selector involvedObject.kind=PersistentVolumeClaim

By Type

# Warning events (problems)
kubectl get events -n <ns> --field-selector type=Warning

# Normal events
kubectl get events -n <ns> --field-selector type=Normal

By Reason

# Common failure reasons
kubectl get events -A --field-selector reason=Failed
kubectl get events -A --field-selector reason=FailedScheduling
kubectl get events -A --field-selector reason=FailedMount
kubectl get events -A --field-selector reason=BackOff
kubectl get events -A --field-selector reason=Unhealthy
kubectl get events -A --field-selector reason=OOMKilling

Common Event Reasons

| Reason | Meaning | Action | |--------|---------|--------| | FailedScheduling | No node can run pod | Load debugging-k8s-scheduling | | FailedMount | Volume mount failed | Load debugging-k8s-storage | | BackOff | Container crashing | Load debugging-k8s-pods | | Unhealthy | Probe failing | Check liveness/readiness probe | | Killing | Pod being terminated | Check termination reason | | Pulled | Image pulled successfully | Informational | | Created | Container created | Informational | | Started | Container started | Informational |

Event Timeline Analysis

Build Timeline for Incident

# Wide output with timestamps
kubectl get events -n <ns> --sort-by=.lastTimestamp -o wide

# Custom columns for analysis
kubectl get events -n <ns> --sort-by=.lastTimestamp \
  -o custom-columns=TIME:.lastTimestamp,TYPE:.type,REASON:.reason,OBJECT:.involvedObject.name,MESSAGE:.message

Correlate Events Across Resources

# Events mentioning a specific string
kubectl get events -n <ns> -o json | jq '.items[] | select(.message | contains("<search-term>"))'

# Or using grep
kubectl get events -n <ns> --sort-by=.lastTimestamp -o wide | grep <term>

Event Details

# Full event details (JSON)
kubectl get events -n <ns> --field-selector involvedObject.name=<resource> -o json

# Key fields to examine:
# - reason: short reason code
# - message: detailed message
# - type: Normal or Warning
# - count: how many times this event occurred
# - firstTimestamp/lastTimestamp: time range

Event Retention

Events are stored in etcd with limited retention (default ~1 hour). For older events:

• Check logging system (if deployed)
• Check monitoring/alerting history
• Events are ephemeral, don't rely on them for long-term history

Quick Investigation Pattern

# 1. Get recent warnings
kubectl get events -n <ns> --field-selector type=Warning --sort-by=.lastTimestamp

# 2. Focus on specific resource
kubectl get events -n <ns> --field-selector involvedObject.name=<resource> --sort-by=.lastTimestamp

# 3. Get full details if needed
kubectl describe <resource-type> <resource-name> -n <ns>

Notes

• Events are ephemeral (typically 1 hour retention)
• Warning type events indicate problems
• Normal type events are informational
• After identifying issue type, load specialized skill for deeper investigation