aldengolab/democratic-csi-nvmeof-attach-failure

democratic-csi NVMe-oF Volume Attach Failure

Problem

Pods that require NVMe-oF PVCs are stuck in Init:0/1. PVCs provision successfully (TrueNAS creates volumes), but attachment at the node fails. The democratic-csi node plugin logs "unable to attach any nvme devices" on every NodeStageVolume call.

Context / Trigger Conditions

• Kubernetes events: MountVolume.MountDevice failed ... unable to attach any nvme devices
• democratic-csi node plugin logs: handler error - method: NodeStageVolume error: {"message":"unable to attach any nvme devices"}
• PVCs show Bound (provisioning OK) but pods never leave Init state
• democratic-csi controller logs show successful CreateVolume responses from TrueNAS

Solution

Work through four failure modes in order:

1. Confirm provisioning vs. attachment split

Check controller logs to confirm volumes were created on TrueNAS:

kubectl logs -n democratic-csi <controller-pod> -c csi-driver --tail=50 | grep -i "CreateVolume\|error"

If CreateVolume succeeded but NodeStageVolume fails, the problem is network/transport, not credentials or config.

2. Check kernel modules

kubectl exec -n democratic-csi <node-pod> -c csi-driver -- cat /proc/modules | grep nvme

Required modules: nvme_tcp, nvme_fabrics, nvme_core. If missing, the host kernel lacks NVMe-oF support.

3. Test nvme binary hostname resolution

kubectl exec -n democratic-csi <node-pod> -c csi-driver -- \
  nvme discover -t tcp -a <shareHost-value> -s <sharePort>

If output contains "No support for hostname IP address resolution; recompile with libnss support":

• The nvme binary in the container is statically compiled and can't resolve hostnames via NSS
• Workaround: resolve the hostname from within the cluster and use the IP in shareHost
• Resolve hostname: kubectl run -it --rm dns-test --image=busybox --restart=Never -- nslookup <hostname>

4. Test transport connectivity with IP

kubectl exec -n democratic-csi <node-pod> -c csi-driver -- \
  nvme discover -t tcp -a <resolved-IP> -s <sharePort>

If output is "Connection refused" or "failed to get transport address":

• The NVMe-oF TCP port (default: 4420) is not reachable from the cluster
• This is a network/firewall/interface binding issue on the storage server

Confirm with a busybox TCP probe:

kubectl run -it --rm tcp-test --image=busybox --restart=Never -- \
  sh -c "echo connected > /dev/tcp/<IP>/<port> && echo OPEN || echo REFUSED"

5. Fix: NVMe-oF port not reachable

Common cause: the storage server (e.g., TrueNAS) binds the NVMe-oF target to a LAN interface but shareHost in the driver config points to a different IP (e.g., Tailscale overlay IP). The storage controller API (HTTP) works on the overlay but NVMe-oF TCP doesn't.

Fix options:

• Change shareHost in the driver config to the LAN/storage-network IP
• Or configure the storage server to also bind NVMe-oF on the overlay interface

If the config is managed via ESO (ExternalSecrets), update the backing secret in the secret store, then force a sync:

kubectl annotate externalsecret <name> -n <ns> force-sync=$(date +%s) --overwrite

After updating, restart the democratic-csi node plugin pods to pick up the new config.

6. Check PV volumeAttributes for stale transport address

If steps 1–5 pass (kernel modules present, transport reachable at the correct IP, driver config looks right) but NodeStageVolume still fails, check whether the PV itself stores the wrong transport IP:

kubectl get pv <pv-name> -o jsonpath='{.spec.csi.volumeAttributes.transports}'

Diagnostic signal: node plugin logs will show connecting to transport: tcp://<wrong-IP>:4420 where <wrong-IP> differs from the transports in the current driver config. This happens because volumeAttributes are written at provision time from the storage backend API response and take precedence over the current driver config for existing volumes.

Fix: spec.persistentvolumesource is immutable — it cannot be patched in place. See kubernetes-csi-pv-spec-update for the replace --force + finalizer removal procedure.

Root fix: Correct the NVMe-oF port binding on TrueNAS so future volumes are provisioned with the correct storage-network IP. Existing PVs must still be replaced manually.

Verification

# Confirm NodeStageVolume no longer errors
kubectl logs -n democratic-csi <node-pod> -c csi-driver --tail=20 | grep -i "error\|NodeStage"

# Check pod status
kubectl get pods -n <app-namespace>

Pods should transition from Init:0/1 to Running.

Notes

• PVC provisioning (CreateVolume via HTTP API) and volume attachment (NodeStageVolume via NVMe-oF TCP) use completely different network paths — one can work while the other fails.
• The nvme binary's hostname resolution failure ("recompile with libnss support") is a separate error from transport connectivity failure ("Connection refused"). Always test with IP after seeing the libnss error.
• hostPID: true on the node DaemonSet is required for nsenter-based nvme operations if you go that route.
• The driver config secret key must be named driver-config-file.yaml (democratic-csi chart requirement).

References

• democratic-csi GitHub — configuration reference for nvmeof driver