aldengolab/kubernetes-csi-pv-spec-update

Updating Immutable CSI PersistentVolume Spec Fields

Problem

A CSI PersistentVolume stores incorrect or outdated values in its spec (commonly spec.csi.volumeAttributes) that were written at provision time and now need to change. Attempting to patch the PV fails with:

The PersistentVolume "<name>" is invalid:
  spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation

Context / Trigger Conditions

• kubectl patch pv <name> fails with "immutable after creation"
• A CSI driver is using stale connection info from a PV (e.g., wrong IP, endpoint, or config key) that differs from the current driver config
• The node plugin logs show it connecting to an address different from what the driver config specifies — because volumeAttributes set at provision time takes precedence over current driver config for existing volumes
• Any field under spec.csi needs to change after the PV was created

Solution

spec.persistentvolumesource is enforced as immutable by the Kubernetes API server itself (not an admission webhook), so there is no way to patch it. The only path is delete and recreate the PV using kubectl replace --force.

Step 1: Remove the pv-protection finalizer

When a PVC is bound to a PV, the kubernetes.io/pv-protection controller adds a finalizer that blocks PV deletion. kubectl replace --force will hang until the PV is fully deleted, which won't happen while the finalizer is present. Remove it first:

kubectl patch pv <pv-name> -p '{"metadata":{"finalizers":null}}'

The PVC will briefly emit a ClaimLost warning event during the window between PV deletion and recreation. This is transient — the PVC rebinds automatically when the new PV appears. If the storage has a Retain reclaim policy (recommended for stateful workloads), no data is at risk.

Step 2: Export, edit, and replace

kubectl get pv <pv-name> -o json | \
  python3 -c "
import json, sys
pv = json.load(sys.stdin)
# Make your changes here, e.g.:
pv['spec']['csi']['volumeAttributes']['transports'] = 'tcp://192.168.0.19:4420'
# Strip fields that cause conflicts on re-create
pv['metadata'].pop('managedFields', None)
pv['metadata'].pop('resourceVersion', None)
print(json.dumps(pv))
" | kubectl replace --force -f -

Step 3: Verify rebind

kubectl get pv <pv-name> -o jsonpath='{.status.phase}'   # should be Bound
kubectl get pvc -n <namespace>                            # should show Bound

Bulk update

For multiple PVs with the same fix:

for pv in pv-name-1 pv-name-2 pv-name-3; do
  kubectl patch pv $pv -p '{"metadata":{"finalizers":null}}'
  kubectl get pv $pv -o json | \
    python3 -c "
import json, sys
pv = json.load(sys.stdin)
pv['spec']['csi']['volumeAttributes']['<key>'] = '<new-value>'
pv['metadata'].pop('managedFields', None)
pv['metadata'].pop('resourceVersion', None)
print(json.dumps(pv))
" | kubectl replace --force -f -
done

Verification

# Confirm the field is updated
kubectl get pv <pv-name> -o jsonpath='{.spec.csi.volumeAttributes}'

# Confirm PVC is rebound
kubectl get pvc -n <namespace>

Notes

• The pv-protection finalizer is added by the PersistentVolumeProtection admission controller and removed only when the PVC is deleted or the PV is released. Removing it manually is safe as long as you immediately recreate the PV.
• ClaimLost warning events on the PVC during the replace cycle are expected and transient. They do not indicate data loss when reclaim policy is Retain.
• CSI volumeAttributes are written by the provisioner at CreateVolume time from information returned by the storage backend. Changing the driver config does not retroactively update existing PVs — only newly provisioned volumes will have the corrected values.
• If pods are currently running with the affected PVC mounted, they must be stopped before the replace or the volume will be in use and unmountable during the cycle. Pods already stuck in Init (volume never mounted) can be left running — they will pick up the new PV automatically on next kubelet retry.

See Also

• democratic-csi-nvmeof-attach-failure — diagnose when the wrong transport IP is being used by the NVMe-oF CSI node plugin