rio/debugging-k8s-storage

Debugging Kubernetes Storage

Investigates PersistentVolumeClaim, PersistentVolume, and mount issues.

Common Storage Issues

| Symptom | Likely Cause | First Check | |---------|-------------|-------------| | PVC Pending | No matching PV, StorageClass issue | PVC events | | Mount failed | PV not available, node issue | Pod events | | Multi-attach error | RWO volume on multiple nodes | Access mode | | Permission denied | fsGroup/runAsUser mismatch | Security context |

Investigation Workflow

Step 1: Check PVC Status

# List PVCs
kubectl get pvc -n <ns>

# Detailed PVC info
kubectl describe pvc <pvc> -n <ns>

PVC Pending? Check Events section for reason.

Step 2: Check PV Binding

# List PVs
kubectl get pv

# Check PV details
kubectl describe pv <pv-name>

PVC binds to PV when:

• StorageClass matches (or empty for static)
• Access modes compatible
• Capacity sufficient

Step 3: Check StorageClass

# List StorageClasses
kubectl get storageclass

# Check default StorageClass
kubectl get storageclass -o jsonpath='{range .items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")]}{.metadata.name}{end}'

# StorageClass details
kubectl describe storageclass <name>

Step 4: Check Pod Mount Events

# Pod events show mount failures
kubectl describe pod <pod> -n <ns> | grep -A10 "Events:"

# Events for PVC
kubectl get events -n <ns> --field-selector involvedObject.name=<pvc>

Specific Issues

PVC Stuck in Pending

Common reasons:

1. No matching PV (static provisioning)
2. StorageClass can't provision (dynamic provisioning)
3. Capacity not available
4. Wrong access mode

# Check what the PVC is requesting
kubectl get pvc <pvc> -n <ns> -o yaml | grep -A5 "spec:"

# Check events for provisioning errors
kubectl describe pvc <pvc> -n <ns> | grep -A10 "Events:"

Multi-Attach Error

# Check access mode (RWO = ReadWriteOnce = single node)
kubectl get pvc <pvc> -n <ns> -o jsonpath='{.spec.accessModes}'

# Check which node has the volume
kubectl get pod -n <ns> -o wide

RWO volumes can only attach to one node. If pods are on different nodes, one will fail.

Options:

• Use ReadWriteMany (RWX) if storage supports it
• Ensure pods schedule to same node

Volume Mount Timeout

# Check node where pod is scheduled
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.nodeName}'

# Check node conditions
kubectl describe node <node> | grep -A5 "Conditions:"

May indicate:

• Cloud provider API issues
• Node can't reach storage backend
• CSI driver problems

Permission Denied on Volume

# Check pod security context
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.securityContext}'

# Check container security context
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.containers[*].securityContext}'

Fix with fsGroup or runAsUser in pod spec.

Quick Debug Commands

# Overview of all PVC/PV
kubectl get pvc,pv -A

# Check CSI drivers (if using CSI)
kubectl get csidrivers

# Storage-related events
kubectl get events -A --field-selector reason=FailedMount
kubectl get events -A --field-selector reason=FailedAttachVolume

Access Modes Reference

| Mode | Short | Description | |------|-------|-------------| | ReadWriteOnce | RWO | Single node read-write | | ReadOnlyMany | ROX | Multiple nodes read-only | | ReadWriteMany | RWX | Multiple nodes read-write | | ReadWriteOncePod | RWOP | Single pod read-write |

Notes

• Load debugging-k8s-pods if pod has other issues besides storage
• Load analyzing-k8s-events for storage event timeline