aldengolab/k8s-rwo-deployment-recreate-strategy

Single-Replica RWO Deployment: Recreate Strategy

Problem

A Deployment with one replica and a ReadWriteOnce PVC deadlocks during rolling updates. The new pod cannot start because the old pod holds the RWO volume, and the old pod will not terminate until the new pod is Ready.

Context / Trigger Conditions

• Pod stuck in Init:0/1 or Pending
• Event: Multi-Attach error for volume "pvc-...": Volume is already used by pod(s) <old-pod>
• Deployment has replicas: 1 and a RWO PVC
• Default RollingUpdate strategy: maxSurge: 25% (rounds to 1), maxUnavailable: 25% (rounds to 0)
• Common for Grafana, single-instance databases, or any stateful single-replica app

Solution

Step 1: Set Recreate strategy in Helm values

For kube-prometheus-stack Grafana:

grafana:
  deploymentStrategy:
    type: Recreate

For a raw Deployment:

spec:
  strategy:
    type: Recreate

Step 2: Handle the rollingUpdate field conflict

Changing an existing Deployment from RollingUpdate to Recreate via kubectl apply or ArgoCD sync fails with:

Deployment.apps "<name>" is invalid: spec.strategy.rollingUpdate:
Forbidden: may not be specified when strategy type is 'Recreate'

This happens because the live Deployment still has rollingUpdate sub-fields set. Fix: delete the Deployment and let the controller recreate it with the new spec.

kubectl delete deployment <name> -n <namespace>
# ArgoCD will immediately recreate it with the correct spec on next sync
argocd app sync <app-name>

Verification

kubectl get deployment <name> -n <namespace> \
  -o jsonpath='{.spec.strategy.type}'
# Should output: Recreate

Notes

• Recreate strategy means brief downtime during upgrades (old pod deleted before new one starts). Accept this trade-off for single-replica stateful apps with RWO PVCs.
• Do NOT try to patch spec.strategy in-place on a live Deployment that has rollingUpdate fields — always delete and recreate.
• Applies to any chart that wraps Deployment creation: kube-prometheus-stack, Bitnami charts, custom charts, etc.
• The intermediate symptom (deleting old pod immediately recreated) is the ReplicaSet controller doing its job — it does not indicate the fix failed, it means the Deployment still has the old RS active.

See Also

• k8s-helm-immutable-field-workaround — general pattern for other immutable Helm resource fields (Services, PVCs); includes PreSync hook template
• k8s-helm-immutable-field-workaround/references/delete-and-recreate.md — all delete-and-recreate variants (PreSync hook, kubectl replace --force for PVs)