kurtosis-tech/k8s-debug-pods

K8s Debug Pods

Diagnose and fix issues with Kurtosis pods on Kubernetes.

Quick triage

# See all kurtosis-related pods across namespaces
kubectl get pods -A | grep kurtosis

# Check for problem pods (not Running)
kubectl get pods -A | grep kurtosis | grep -v Running

# Get events for a specific pod
kubectl describe pod <POD_NAME> -n <NAMESPACE> | tail -30

Common pod states and fixes

Pending — Unschedulable

The pod can't be scheduled because of node taints, resource pressure, or affinity rules.

# Check node taints
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

# Check node conditions (DiskPressure, MemoryPressure, etc.)
kubectl get nodes -o custom-columns=NAME:.metadata.name,CONDITIONS:.status.conditions[*].type

Fix: Add tolerations to the kurtosis config at ~/Library/Application Support/kurtosis/kurtosis-config.yml or fix the node condition.

ImagePullBackOff

The image tag doesn't exist on the registry.

# Check which image is failing
kubectl describe pod <POD_NAME> -n <NAMESPACE> | grep -A5 "Image:"

# Verify image exists on Docker Hub
docker manifest inspect <IMAGE>:<TAG>

Fix: Push the correct image tag, or fix the image reference in the code.

CrashLoopBackOff

The container starts but crashes immediately.

# Check container logs
kubectl logs <POD_NAME> -n <NAMESPACE>
kubectl logs <POD_NAME> -n <NAMESPACE> --previous

Evicted

The node evicted the pod due to resource pressure.

# Check which nodes have pressure
kubectl get nodes -o custom-columns=NAME:.metadata.name,STATUS:.status.conditions[-1].type

# Clean up evicted pods
kubectl get pods -A | grep Evicted | awk '{print $2 " -n " $1}' | xargs -L1 kubectl delete pod

Kurtosis-specific pod types

| Pod pattern | Component | Image source | |-------------|-----------|-------------| | kurtosis-engine-* | Engine server | engine/server/Dockerfile | | kurtosis-api (in kt-* namespaces) | API Container (APIC) | core/server/Dockerfile | | kurtosis-logs-collector-* | Fluentbit DaemonSet | Pulled from registry | | kurtosis-logs-aggregator-* | Vector deployment | Pulled from registry | | remove-dir-pod-* | Fluentbit cleanup pods | busybox | | files-artifact-expander (init container) | Files artifacts | core/files_artifacts_expander/Dockerfile |

Engine start failures

If kurtosis engine start fails:

1. Check if old kurtosis namespaces exist: kubectl get ns | grep kurtosis
2. Delete them: kubectl get ns | grep kurtosis | awk '{print $1}' | xargs -r kubectl delete ns
3. Retry engine start

Logs collector issues

The logs collector is a DaemonSet that runs on every node. If some nodes are unhealthy:

# Check DaemonSet status
kubectl get ds -A | grep kurtosis

# See which pods are not running
kubectl get pods -A | grep logs-collector | grep -v Running

Nodes with DiskPressure or other taints may not schedule collector pods — this is expected and the engine should start with a warning about partially degraded collection.