scitix/deployment-rollout-debug

Deployment Rollout Failure Diagnosis

When a Deployment rollout is stuck, not progressing, or shows replica mismatches, follow this flow to identify the root cause.

Scope: This skill is for diagnosis only. Once you identify the root cause, report it to the user and stop. Do NOT attempt to roll back, scale, or modify the Deployment — that should be left to the user.

Diagnostic Flow

1. Check rollout status

kubectl rollout status deployment/<name> -n <ns> --timeout=5s

This shows whether the rollout is progressing, complete, or stuck. A timeout indicates the rollout is not making progress.

2. Get Deployment details

kubectl get deployment <name> -n <ns> -o wide

Compare the columns:

• DESIRED — target replica count
• CURRENT — total pods (old + new)
• UP-TO-DATE — pods running the new version
• AVAILABLE — pods ready to serve traffic

If UP-TO-DATE < DESIRED or AVAILABLE < DESIRED, the rollout is incomplete.

3. Describe the Deployment

kubectl describe deployment <name> -n <ns>

Focus on:

• Conditions — look for Progressing (status and reason) and Available
• Events — look for scaling events, errors, or warnings
• Strategy — note maxSurge and maxUnavailable settings

4. Check ReplicaSets

kubectl get rs -n <ns> -l app=<name>

If the label app=<name> doesn't match, find ReplicaSets owned by the Deployment:

kubectl get rs -n <ns> --sort-by='.metadata.creationTimestamp' | grep <name>

You should see the old RS (with reduced replicas) and the new RS (scaling up). If the new RS has 0 ready replicas, its pods are failing.

5. Check new ReplicaSet's pods

Find pods from the new RS:

kubectl get pods -n <ns> -l app=<name> --sort-by='.metadata.creationTimestamp'

Check the status of the newest pods. Based on their status:

• Pending → Use the pod-pending-debug skill
• CrashLoopBackOff / Error → Use the pod-crash-debug skill
• ImagePullBackOff / ErrImagePull → Use the image-pull-debug skill
• Running but not Ready → Check readiness probe below

6. Match deployment-level patterns

---

ProgressDeadlineExceeded — Rollout timed out

The Deployment's spec.progressDeadlineSeconds (default 600s) has been exceeded without progress.

This is a symptom, not a cause. The root cause is in the new pods — check step 5 for why new pods are not becoming ready.

---

MinimumReplicasUnavailable — Not enough replicas available

The Deployment cannot maintain the minimum number of available replicas during the rollout.

Check the new pods' status (step 5). The new version's pods are failing to start or pass readiness probes.

---

New pods Running but not Ready — Readiness probe failing

The new pods started successfully but are failing their readiness probe. The rollout will not progress because the new pods are not considered available.

Check the readiness probe configuration and failures:

kubectl describe pod <new-pod> -n <ns>

Look for Readiness probe failed events. Common causes:

• Application not listening on the expected port
• Health endpoint returning errors
• Probe timing too aggressive (initialDelaySeconds too low)

---

Rollout stuck with maxSurge: 0 and maxUnavailable: 0 — Invalid strategy

If both maxSurge and maxUnavailable are 0, the rollout cannot proceed because it cannot create extra pods and cannot remove existing pods. At least one must be greater than 0.

---

New RS created but not scaling up — Admission webhook or quota

The new ReplicaSet exists but has 0 replicas or cannot create pods:

kubectl describe rs <new-rs> -n <ns>

Check events for:

• Admission webhook denied — a webhook is rejecting the new pod spec
• Exceeded quota / LimitRange violation — Use quota-debug for native ResourceQuota/LimitRange diagnosis, or volcano-queue-diagnose for Volcano gang scheduling clusters
• FailedCreate — other creation failures

---

Old RS not scaling down — Waiting for new pods

The old ReplicaSet keeps its replicas because the new RS pods are not ready yet. This is expected behavior — Kubernetes will not remove old pods until new pods are available. Fix the new pods first.

Notes

• kubectl rollout status waits for the rollout to complete by default. Use --timeout to avoid blocking.
• If the user wants to undo a failed rollout: kubectl rollout undo deployment/<name> -n <ns> (but let the user decide, do not execute this).
• For Deployments managed by Helm or ArgoCD, the rollout may be triggered by those tools — check if the issue is in the Helm chart values or ArgoCD sync.
• StatefulSet rollouts follow a different pattern (ordered, one-at-a-time by default). This skill is specific to Deployments.