curiositech/kubernetes-debugging-runbook

Kubernetes Debugging Runbook

kubectl describe is the single most useful command in Kubernetes. The events at the bottom answer 80% of "why isn't this working" questions if you read them. This runbook is the playbook for the other 20%.

When to use

• Pod stuck in Pending, CrashLoopBackOff, ImagePullBackOff, Init:Error.
• Pod restarts climb but logs show clean shutdown.
• Service routes to no endpoints despite the deployment being ready.
• HPA shows targets but never scales.
• Inter-pod traffic blocked despite no NetworkPolicy you wrote.
• Disk pressure or eviction storms on a node.

Core capabilities

The triage sequence

kubectl get pods -n NAMESPACE                              # status overview
kubectl describe pod POD -n NAMESPACE                      # events at the bottom
kubectl logs POD -n NAMESPACE --previous                   # logs from the crashed instance
kubectl logs POD -n NAMESPACE -c CONTAINER                 # specific container in a multi-container pod
kubectl get events -n NAMESPACE --sort-by='.lastTimestamp' # cluster-wide context

--previous is critical for CrashLoopBackOff — the current logs are from the not-yet-crashed instance.

Decoding pod status

| Status | Likely cause | |--------|-------------| | Pending + no events | Scheduler hasn't tried yet; usually transient. | | Pending + "no nodes available" | Resource requests too high, taint mismatch, PV affinity. | | ContainerCreating (long) | ImagePull, volume attach, or sidecar init delay. | | ImagePullBackOff | Image name typo, registry auth, image deleted. | | CrashLoopBackOff | App exits non-zero on startup. Check --previous logs. | | Init:Error | Init container failed. kubectl logs POD -c INIT_NAME. | | OOMKilled (in describe) | Memory limit exceeded. | | Error exit code 137 | SIGKILL — usually OOM or liveness probe failure. | | Completed | Job ran. For Deployment, this means the container exited 0 immediately — usually a misconfigured command. |

CrashLoopBackOff playbook

# 1. Last logs before the crash
kubectl logs POD --previous

# 2. Exit code
kubectl describe pod POD | grep -A3 "Last State"

# 3. Resource limits — was it OOMKilled?
kubectl describe pod POD | grep -A2 "Limits"

# 4. Liveness probe killed it?
kubectl describe pod POD | grep -A5 "Liveness"

# 5. Configmap/secret references resolved?
kubectl describe pod POD | grep -A1 "Mounts"

Common roots:

• App reads an env var the container doesn't have → exit 1 in 0.1s.
• Liveness probe hits an endpoint that takes longer than timeoutSeconds to respond.
• Memory request set; limit not — OOMKilled when traffic arrives.
• Init container ran a migration that's already applied; exits non-zero.

OOMKilled

kubectl describe pod POD | grep -E "OOMKilled|Last State|Limits"

Set both requests and limits:

resources:
  requests: { memory: 256Mi, cpu: 100m }
  limits:   { memory: 512Mi, cpu: 500m }

Limit too low → OOMKilled under load. No limit → noisy neighbor evicts other pods. Use kubectl top pod to see actual usage and size accordingly.

ImagePullBackOff

kubectl describe pod POD | grep -A3 "Events:"
# "Failed to pull image": typo, deleted tag, or auth.
# "no such host": private registry not reachable from nodes.

Verify the image exists from the cluster:

kubectl run -i --rm test --image=YOUR_IMAGE --restart=Never --command -- sh -c "echo OK"

For private registries, imagePullSecrets must reference a Docker-config secret in the pod's namespace:

kubectl create secret docker-registry regcred \
  --docker-server=ghcr.io \
  --docker-username=... \
  --docker-password=... \
  -n YOUR_NAMESPACE

Pending: no nodes match

kubectl describe pod POD | grep -A5 "Events"
# "0/N nodes are available: N Insufficient cpu, N Insufficient memory."
# "0/N nodes are available: N node(s) had untolerated taint."
# "0/N nodes are available: N node(s) didn't find available persistent volumes to bind."

For taint mismatches, add a toleration:

tolerations:
- key: "dedicated"
  operator: "Equal"
  value: "gpu"
  effect: "NoSchedule"

For PV binding, check the PVC's storage class and the PV's claimRef:

kubectl get pvc -n NS
kubectl describe pvc PVC -n NS
kubectl get pv | grep PVC_NAME

Service has no endpoints

kubectl get endpoints SVC -n NS                            # empty?
kubectl describe svc SVC -n NS | grep "Selector"           # selector
kubectl get pods -n NS --selector=KEY=VALUE -l            # what does the selector match?

Endpoints empty = the selector matches no pods, OR matched pods aren't ready. Pods need both Running AND passing readinessProbe.

Networking inside the cluster

# Run a transient debug pod with curl + dig.
kubectl run -i --rm dbg --image=nicolaka/netshoot --restart=Never -- sh

# From dbg:
nslookup my-svc.my-ns.svc.cluster.local
curl -v http://my-svc.my-ns.svc.cluster.local:8080/health

If DNS fails, check CoreDNS pods (kubectl get pods -n kube-system | grep coredns).

If DNS works but curl times out, suspect a NetworkPolicy:

kubectl get netpol -n NS
kubectl describe netpol POLICY -n NS

NetworkPolicies are deny-by-default once any policy targets a pod. A default-deny ingress policy blocks all traffic until you explicitly allow it.

HPA not scaling

kubectl describe hpa HPA -n NS | grep -A5 "Conditions"
# "FailedGetResourceMetric" → metrics-server not running or pod has no requests.

HPA needs resources.requests on the pod containers to compute utilization. Without requests.cpu, the CPU-percentage HPA is undefined.

kubectl get apiservices | grep metrics
kubectl top pods -n NS                                     # quick "is metrics-server alive" check

Disk pressure / evictions

kubectl describe node NODE | grep -A5 "Conditions"
kubectl get events --field-selector reason=Evicted -A

Common roots: unbounded log files (set up logrotate or use a log driver), emptyDir volumes filling local disk, container image cache full.

Anti-patterns

Reading current logs on a CrashLoopBackOff

Symptom: kubectl logs POD shows nothing useful; pod is restarting. Diagnosis: Reading the in-flight container's logs, which haven't started writing yet. Fix: kubectl logs POD --previous to read the last instance's logs.

Liveness probe set to the same endpoint as readiness

Symptom: Healthy pods get killed mid-flight. Diagnosis: Liveness asserts the app is wedged; readiness asserts it can serve traffic. Conflating them causes a slow GC pause to trigger a kill. Fix: Liveness should check process aliveness only (a TCP socket open, /healthz that returns 200 from a goroutine independent of the request path). Readiness checks dependencies.

Memory limit without requests

Symptom: Pod scheduled on a fully-utilized node and OOMKilled at random. Diagnosis: Without requests, scheduler treats the pod as 0-memory; node accepts it past capacity. Fix: Always set requests ≤ typical usage and limits 1.5-2x requests.

kubectl exec to "fix" a misbehaving pod

Symptom: Engineer manually kubectl execs in and edits config. Diagnosis: Pod is ephemeral; restart wipes the change. Worse, the pod is now an undocumented snowflake. Fix: Reproduce in a debug pod (kubectl run --rm), commit the fix to the manifest.

NetworkPolicy default-deny without per-namespace audit

Symptom: Adding a deny policy in one namespace breaks unrelated services. Diagnosis: Cluster-wide controllers (Prometheus, cert-manager) traffic blocked. Fix: Always pair a default-deny with explicit allows for monitoring, ingress controllers, and any cross-namespace traffic the policy needs to permit.

imagePullSecrets in the wrong namespace

Symptom: ImagePullBackOff after migrating a Deployment to a new namespace. Diagnosis: Secret existed in default only; deployment's namespace doesn't have it. Fix: Recreate the secret in the new namespace, or use serviceAccount.imagePullSecrets referenced everywhere.

Quality gates

• [ ] Every container has both requests and limits for memory and CPU.
• [ ] Liveness probes check process aliveness only; readiness checks dependencies.
• [ ] Production deploys disable latest tag; pin to immutable tags or digests.
• [ ] NetworkPolicies tested in staging with explicit allow rules for monitoring.
• [ ] Critical pod restart events alert (Prometheus rule on kube_pod_container_status_restarts_total).
• [ ] OOMKilled events alert separately from generic restart spikes.
• [ ] HPA targets configured only on metrics with requests defined.
• [ ] Pod logs go to a centralized destination, not just node disk.
• [ ] Runbook links from each alert pointing to the relevant kubectl invocation.

NOT for

• Cluster setup (kubeadm, kops, k3s installation) — different domain.
• Cloud-provider-specific managed K8s admin (EKS node groups, GKE Autopilot quirks, AKS networking).
• Helm chart authoring — separate skill.
• Service mesh internals (Istio, Linkerd) — different debugging surface.
• Operator pattern development — controller-runtime is its own deep area. No dedicated skill yet.
• Image build & cache strategy — once it's ImagePullBackOff from a missing layer, → dockerfile-build-cache-mastery.
• Span-level latency triage / missing trace correlation — → opentelemetry-instrumentation.
• Cluster-wide SLO dashboards — → grafana-dashboard-builder.
• Log volume blowing up from chatty pods — → structured-logging-design.