apecloud/kubeblocks-troubleshoot

Troubleshoot KubeBlocks Clusters

Overview

This skill helps diagnose and fix common issues with KubeBlocks-managed database clusters. Follow the diagnostic flowchart and sections below to systematically identify root causes. This skill guides the agent through diagnostic steps; it does not perform actions.

Quick Diagnostic Flowchart

User reports cluster problem
│
├─ Is the KubeBlocks operator healthy?
│  └─ kubectl -n kb-system get pods
│     ├─ Pods not Running → Operator Issues (Section 5)
│     └─ Pods Running → continue
│
├─ Is the cluster in an abnormal state?
│  └─ kubectl get cluster <name> -n <ns>
│     ├─ Phase: Creating/Updating/Abnormal/Deleting → Cluster Status Issues (Section 1)
│     └─ Phase: Running → continue
│
├─ Is there a failed OpsRequest?
│  └─ kubectl get opsrequest -n <ns>
│     ├─ Failed/Running (stuck) → OpsRequest Failures (Section 3)
│     └─ Succeeded → continue
│
└─ Are pods crashing?
   └─ kubectl get pods -n <ns> -l app.kubernetes.io/instance=<cluster>
      ├─ CrashLoopBackOff/ImagePullBackOff/Pending → Pod Issues (Section 2)
      └─ All Running → check pod logs and events for application errors

1. Cluster Status Issues

Cluster stuck in Creating

| Cause | How to verify | Action | |-------|---------------|--------| | Addon not installed | kubectl get addon | Install the required addon (e.g. kbcli addon enable mysql) | | Insufficient resources | kubectl describe pod <pending-pod> → Events | Increase node resources or reduce cluster requests | | Image pull errors | kubectl describe pod → ImagePullBackOff | Fix registry access, image name, or imagePullSecrets |

Cluster stuck in Updating

1. Check if all pods are Running: kubectl get pods -n <ns> -l app.kubernetes.io/instance=<cluster>
2. Check pod logs for errors: kubectl logs <pod> -c <container>
3. Verify pod roles: kubectl get po <pod> -L kubeblocks.io/role
4. Check for failed OpsRequest: kubectl get opsrequest -n <ns> and kubectl describe opsrequest <name>
5. Ensure container image in status.containerStatuses matches spec.containers.image; mismatches can block updates

Cluster in Abnormal

Usually caused by pod failures or storage issues. Check pod status and events (see Pod Issues below), then kubectl describe cluster <name> -n <ns> for conditions.

Cluster stuck in Deleting

If KubeBlocks logs show has no pods to running the pre-terminate action, the cluster cannot run the pre-terminate lifecycle. To skip:

kubectl annotate component <COMPONENT_NAME> -n <ns> apps.kubeblocks.io/skip-pre-terminate-action=true

2. Pod Issues

CrashLoopBackOff

1. Check logs: kubectl logs <pod> -c <container> --previous
2. Describe pod: kubectl describe pod <pod> -n <ns> (check Events)
3. Common causes: Wrong config, bad credentials, storage mount failure, OOM, database init failure

ImagePullBackOff

1. Describe pod: kubectl describe pod <pod> → Events show pull error
2. Common causes: Private registry without imagePullSecrets, wrong image name/tag, network/registry unreachable

Pending

1. Describe pod: kubectl describe pod <pod> → Events
2. Common causes: Insufficient CPU/memory, no nodes matching nodeSelector/affinity, PVC pending (StorageClass or capacity)

3. OpsRequest Failures

How to check failed OpsRequest

kubectl get opsrequest -n <namespace>
kubectl describe opsrequest <ops-name> -n <namespace>

Check status.conditions and Events for failure reason.

Common OpsRequest failure reasons

• Resource constraints (scaling beyond available capacity)
• Preconditions not met (e.g. cluster not Running)
• Timeout or step failure during the operation

How to cancel a stuck OpsRequest

Only VerticalScaling and HorizontalScaling OpsRequests in Running state can be cancelled:

kubectl patch opsrequest <OPSREQUEST_NAME> -n <ns> -p '{"spec":{"cancel":true}}' --type=merge

4. Operator Issues

KubeBlocks operator pod not running

kubectl -n kb-system get pods
kubectl -n kb-system describe pod <kubeblocks-pod>

CRD version mismatch

If installing on K8s ≤ 1.23, you may see unknown field "x-kubernetes-validations". Apply CRDs with --validate=false (see official docs).

Check operator logs

kubectl -n kb-system logs -l app.kubernetes.io/name=kubeblocks --tail=100 -f
# or
kubectl -n kb-system logs deployments/kubeblocks -f

ComponentDefinition status Unavailable

If kubectl get componentdefinition shows Unavailable (e.g. "immutable fields can't be updated"):

kubectl annotate componentdefinition <NAME> apps.kubeblocks.io/skip-immutable-check=true

5. Useful Commands Reference

| Purpose | Command | |---------|---------| | Operator health | kubectl -n kb-system get pods | | Cluster status | kubectl get cluster -A | | Cluster details | kubectl describe cluster <name> -n <ns> | | OpsRequest status | kubectl get opsrequest -n <ns> | | Pod status | kubectl get pods -n <ns> -l app.kubernetes.io/instance=<cluster> | | Pod logs | kubectl logs <pod> -c <container> -n <ns> | | Pod events | kubectl describe pod <pod> -n <ns> | | Operator logs | kubectl -n kb-system logs -l app.kubernetes.io/name=kubeblocks --tail=100 | | Cluster resources | kubectl get cmp,its,po -l app.kubernetes.io/instance=<cluster> -n <ns> | | Generate report | kbcli report cluster <name> --with-logs --mask |

6. Official Troubleshooting Docs

| Resource | URL | |----------|-----| | FAQs (cluster exception) | https://kubeblocks.io/docs/preview/user_docs/troubleshooting/handle-a-cluster-exception | | Known Issues | https://kubeblocks.io/docs/preview/user_docs/troubleshooting/known-issues | | Full doc index | https://kubeblocks.io/llms-full.txt |

Known issues to check

• PostgreSQL password with special chars (v0.9.4 and before, v1.0.0): Upgrade to v1.0.1-beta.6+ or set symbolCharacters in passwordConfig to avoid YAML parsing errors.
• Excessive secrets (KubeBlocks v1.0.0 on K8s ≤ 1.24): Upgrade to v1.0.1-beta.3+.