scitix/job-debug

Job / CronJob Failure Diagnosis

When a Job has failed, is stuck, or a CronJob is not triggering as expected, follow this flow to identify the root cause.

Scope: This skill is for diagnosis only. Once you identify the root cause, report it to the user and stop. Do NOT attempt to delete, restart, or modify Jobs or CronJobs — that should be left to the user.

Diagnostic Flow

1. Check Job status

kubectl get jobs -n <ns>

Note the COMPLETIONS (succeeded/total) and DURATION columns. For a specific Job:

kubectl describe job <job-name> -n <ns>

Focus on:

• Conditions — look for Complete (success) or Failed (failure) with the reason
• Events — look for BackoffLimitExceeded, DeadlineExceeded, or pod creation errors
• Pods Statuses — counts of Succeeded, Failed, and Active pods

2. Check the Job's pods

kubectl get pods -n <ns> -l job-name=<job-name> --sort-by='.metadata.creationTimestamp'

Note pod statuses. For failed pods, check logs:

kubectl logs <pod-name> -n <ns>

If the pod was terminated, check the exit code:

kubectl get pod <pod-name> -n <ns> -o jsonpath='{.status.containerStatuses[*].state.terminated}'

3. For CronJobs — check schedule and history

kubectl get cronjobs -n <ns>

Note the SCHEDULE, SUSPEND, ACTIVE, and LAST SCHEDULE columns.

For details:

kubectl describe cronjob <cronjob-name> -n <ns>

Focus on:

• Schedule — the cron expression
• Suspend — if True, the CronJob will not create new Jobs
• Last Schedule Time — when the last Job was triggered
• Active Jobs — currently running Jobs
• Events — look for creation events or errors

List Jobs created by the CronJob:

kubectl get jobs -n <ns> -l job-name --sort-by='.metadata.creationTimestamp' | grep <cronjob-name>

4. Match patterns and conclude

---

BackoffLimitExceeded — Too many pod failures

The Job's pods have failed more times than the backoffLimit (default: 6). The Job is marked as Failed.

Check why the pods are failing — look at the logs of the most recent failed pod (step 2). Common causes:

• Application error — the program exits with a non-zero code
• OOMKilled — the container exceeded its memory limit (use pod-crash-debug for deeper analysis)
• Configuration error — missing environment variables, wrong arguments, or missing config files

Advise the user to fix the underlying pod failure, then create a new Job.

---

DeadlineExceeded — Job took too long

The Job did not complete within its activeDeadlineSeconds limit. All running pods are terminated.

kubectl get job <job-name> -n <ns> -o jsonpath='{.spec.activeDeadlineSeconds}'

Check if the Job's pods are slow or stuck:

• The workload may genuinely take longer than the deadline allows
• Pods may be stuck waiting for resources, network, or external dependencies

Advise the user to either increase activeDeadlineSeconds or investigate why the workload is slow.

---

Pods in Pending — Scheduling issues

Job pods are created but cannot be scheduled.

Use the pod-pending-debug skill to diagnose the scheduling failure. Common causes for batch jobs:

• Resource constraints — the Job requests more resources than available
• Node affinity — the Job's pods can only run on specific nodes that are busy

---

Pods in ImagePullBackOff — Image issue

The Job pod cannot pull its container image. Use the image-pull-debug skill.

---

Job succeeded but results are wrong — Application-level issue

The Job completed (status Complete) but produced incorrect results. This is not a Kubernetes issue — check the application logs:

kubectl logs <succeeded-pod> -n <ns>

If the pod has been garbage collected, check if ttlSecondsAfterFinished caused it to be deleted:

kubectl get job <job-name> -n <ns> -o jsonpath='{.spec.ttlSecondsAfterFinished}'

---

CronJob not triggering — Schedule or suspend issue

The CronJob exists but is not creating Jobs on schedule.

• Suspended — spec.suspend is true. No new Jobs will be created until it is unsuspended.
• Invalid schedule — the cron expression may be malformed. Kubernetes uses standard 5-field cron syntax (minute, hour, day-of-month, month, day-of-week).
• Concurrency policy — if concurrencyPolicy: Forbid and a previous Job is still running, the new Job will be skipped.

kubectl get cronjob <cronjob-name> -n <ns> -o jsonpath='suspend={.spec.suspend} concurrencyPolicy={.spec.concurrencyPolicy} schedule={.spec.schedule}'

Check if there are active Jobs blocking the next run:

kubectl get jobs -n <ns> | grep <cronjob-name>

---

CronJob creates Jobs but they keep failing — Recurring failure

If every Job created by the CronJob fails, the root cause is in the Job template. Diagnose the most recent Job using steps 1-2 above.

Check the startingDeadlineSeconds — if set, Jobs that miss their window (e.g., because the previous Job ran too long) are skipped:

kubectl get cronjob <cronjob-name> -n <ns> -o jsonpath='{.spec.startingDeadlineSeconds}'

Also check successfulJobsHistoryLimit and failedJobsHistoryLimit — if these are set to 0, completed/failed Jobs and their pods are deleted immediately, making diagnosis harder.

Notes

• Jobs created by CronJobs are named <cronjob-name>-<timestamp>. Use this pattern to find related Jobs.
• By default, failed Job pods are kept (not deleted) so you can inspect their logs. The backoffLimit controls how many retries happen before the Job is marked as Failed.
• completions and parallelism control how many pods need to succeed and how many run concurrently. A Job with completions: 5, parallelism: 2 runs 2 pods at a time until 5 have succeeded.
• For indexed Jobs (completionMode: Indexed), each pod gets a unique index. A failure in one index does not affect others — check which specific index is failing.