aldengolab/truenas-jsonrpc-debug

TrueNAS JSON-RPC 2.0 WebSocket Debugging

The TrueNAS REST API is deprecated as of v25.10. All queries now go through a JSON-RPC 2.0 WebSocket endpoint at ws://<host>/api/current (or wss:// for HTTPS).

Use the bundled script scripts/truenas_query.py for all TrueNAS queries.

Prerequisites

pip install websockets pyyaml

Step 1: Generate Credentials File

The credentials file is required for every session. Generate it at the start of each debugging session — it pulls the TrueNAS host and API key from the cluster secret each time:

python3 <skill-path>/scripts/truenas_query.py --get-creds --creds ~/.truenas-creds.json

This writes ~/.truenas-creds.json (persistent, survives reboots unlike /tmp). The file contains the host IP, port, protocol, and API key extracted from the democratic-csi-driver-config secret.

One-Time Setup: API Key for JSON-RPC

The democratic-csi API key works for REST provisioning but is rejected by the JSON-RPC WebSocket endpoint with AUTH_ERR. You need a separate key created in the TrueNAS UI:

1. Open TrueNAS UI → Credentials → API Keys → Add
2. Name it (e.g., claude-debug), create it under your admin account
3. Edit ~/.truenas-creds.json:
   • Set "username" to the account that owns the key (e.g., "truenas_admin")
   • Set "api_key" to the new key value

After this one-time edit, --get-creds will refresh the host/port and API key from the cluster secret on future runs but will preserve any username already in the file — so you only need to set it once.

Auth method used:

• If username is set in creds: auth.login_ex with API_KEY_PLAIN mechanism
• If username is null: auth.login_with_api_key (legacy, may not work with v25.10+ keys)

Step 2: Run Diagnostics

NVMe-oF target state (subsystems, ports, namespaces)

python3 <skill-path>/scripts/truenas_query.py \
  --creds ~/.truenas-creds.json --diag nvme --out /tmp/tn_nvme.json

Key fields to check in output:

• nvmet_ports[*].addr_trsvcid — port number the target is listening on (default 4420)
• nvmet_ports[*].addr_traddr — IP address the target is bound to (if 0.0.0.0, all interfaces)
• nvmet_subsystems — list of NQN subsystems (should match what democratic-csi provisioned)
• nvme_services — whether the NVMe service is running

ZFS pool and CSI dataset health

python3 <skill-path>/scripts/truenas_query.py \
  --creds ~/.truenas-creds.json --diag storage --out /tmp/tn_storage.json

Key fields:

• pools[*].status — should be ONLINE
• csi_datasets — all datasets matching csi-pvc-* (provisioned by democratic-csi)
• disks[*].devname and disks[*].status — physical disk health

System info

python3 <skill-path>/scripts/truenas_query.py \
  --creds ~/.truenas-creds.json --diag system

Everything at once

python3 <skill-path>/scripts/truenas_query.py \
  --creds ~/.truenas-creds.json --diag all --out /tmp/tn_full.json

Step 3: Call Any Method Directly

# List all services
python3 <skill-path>/scripts/truenas_query.py \
  --creds ~/.truenas-creds.json --method service.query

# Query a specific dataset by name
python3 <skill-path>/scripts/truenas_query.py \
  --creds ~/.truenas-creds.json \
  --method pool.dataset.query \
  --params '[["id", "=", "nvme-of-k8s/democratic-csi-provisioner"]]'

# Check NVMe target host port bindings
python3 <skill-path>/scripts/truenas_query.py \
  --creds ~/.truenas-creds.json --method nvmet.port.query

Protocol Reference

Connection URL: ws://<host>:<port>/api/current

Request format:

{"jsonrpc": "2.0", "id": 1, "method": "pool.query", "params": []}

• Batch requests are not supported — send one at a time
• Server may send unsolicited collection_update notifications (no "id") — ignore these
• Auth must succeed before any other method call

Useful Methods for NVMe/Storage Debugging

| Method | Purpose | |--------|---------| | system.info | Version, uptime | | nvmet.subsystem.query | NVMe subsystem list | | nvmet.port.query | NVMe listener ports and bound IPs | | nvmet.namespace.query | NVMe namespaces (backed by zvols) | | service.query | All service states | | pool.query | ZFS pool health | | pool.dataset.query | All datasets (supports filter params) | | disk.query | Physical disk inventory and health | | interface.query | Network interfaces and IP addresses | | network.configuration.config | Hostname, DNS, gateway |

Notes

• TrueNAS host and port come from httpConnection in the driver config secret — this is the HTTP/WS management interface, not the NVMe-oF data port (4420). They are different.
• The shareHost/sharePort in the driver config is the NVMe-oF data endpoint. The host/port in httpConnection is the management API endpoint.
• If the cluster can't reach TrueNAS at the management IP, check CoreDNS and Tailscale routing. From within the cluster: kubectl run -it --rm dns-test --image=busybox --restart=Never -- nslookup <host>

References

• TrueNAS JSON-RPC 2.0 API docs