scitix/volcano-scheduler-logs

Volcano Scheduler Logs

Retrieve and analyze Volcano scheduler logs to understand scheduling decisions, failures, and performance issues.

Scope: This skill is for diagnosis only. It retrieves logs for analysis but does not modify any cluster state.

Usage

bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh [options]

Parameters

| Parameter | Required | Description | |-----------|----------|-------------| | --keyword KEYWORD | no | Filter logs by keyword (case-insensitive) | | --pod POD | no | Filter logs related to specific pod name | | --since TIME | no | Show logs newer than relative time (e.g., 10m, 1h) | | --lines N | no | Number of lines to show (default: 100) | | --follow | no | Stream logs in real-time (Ctrl+C to stop) | | --previous | no | Show logs from previous container instance (after restart) |

Examples

Get recent scheduler logs:

bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh

Search for error messages:

bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --keyword error

Get logs for a specific pod:

bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --pod my-job-0

Get last 500 lines from the past hour:

bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --since 1h --lines 500

Stream logs for gang scheduling issues:

bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --keyword gang --follow

Check logs from previous scheduler instance (after crash/restart):

bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --previous --lines 200

Common Keywords for Filtering

| Keyword | Use Case | |---------|----------| | error | Find error messages and failures | | FailedScheduling | Scheduling failures | | allocate | Resource allocation attempts | | gang | Gang scheduling decisions | | minMember | MinMember constraint issues | | preempt | Preemption events | | reclaim | Resource reclamation | | enqueue | Queue admission decisions | | bind | Pod binding attempts | | queue | Queue-related decisions | | proportion | Proportion plugin decisions | | priority | Priority-related decisions |

Understanding Scheduler Logs

Log Format

Volcano scheduler logs typically follow this format:

I0102 15:04:05.123456       1 scheduler.go:123] Starting scheduling session
I0102 15:04:05.234567       1 allocate.go:456] Try to allocate resources for Job <namespace>/<job-name>
E0102 15:04:05.345678       1 gang.go:789] Failed to schedule pod <pod-name>: minMember not satisfied

Log levels:

• I - Info: Normal operation information
• W - Warning: Unusual but non-fatal conditions
• E - Error: Failures and errors
• F - Fatal: Critical errors causing shutdown

Common Log Patterns

Session Start

Starting scheduling session
Starting scheduling loop

• Indicates scheduler is processing a new batch of pending pods

Enqueue Decisions

Try to enqueue pod group
PodGroup <name> is enqueued
PodGroup <name> is pending

• Shows whether pod groups are admitted to the queue

Allocation Attempts

Try to allocate resources for Job
Try to allocate for task

• Shows scheduling attempts for specific jobs/pods

Gang Scheduling

minMember not satisfied
gang member not ready
Waiting for gang members

• Indicates Gang constraint preventing scheduling

Resource Shortage

Insufficient cpu
Insufficient memory
0 nodes are available

• Indicates resource constraint preventing scheduling

Preemption

Preempting pods
Found victim pods

• Shows preemption decisions for high-priority workloads

Reclaim

Try to reclaim resources
Reclaiming resources from queue

• Shows resource reclamation between queues

Diagnostic Use Cases

Case 1: Pod Stuck in Pending

Find relevant scheduler decisions:

bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --pod <pod-name> --since 30m

Look for:

• FailedScheduling events
• minMember not satisfied
• Insufficient resource messages
• enqueue decisions (is the PodGroup being admitted?)

Case 2: Gang Scheduling Issues

Check Gang plugin behavior:

bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --keyword gang --since 1h

Look for:

• minMember related messages
• Gang constraint validation
• Comparison of running vs required members

Case 3: Queue Resource Issues

Check proportion and reclaim decisions:

bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --keyword "reclaim\|proportion" --since 30m

Look for:

• Queue resource calculations
• Reclaim triggers
• Over-commit handling

Case 4: Scheduler Performance

Check for scheduling delays:

bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --lines 500 | grep -E "(Starting|Finished) scheduling"

Look for:

• Long gaps between "Starting" and "Finished"
• High frequency of scheduling loops
• Errors causing retries

Case 5: Preemption Analysis

Check preemption decisions:

bash skills/core/volcano-scheduler-logs/scripts/get-scheduler-logs.sh --keyword preempt --since 1h

Look for:

• Which pods are being preempted
• Priority comparisons
• Preemption success/failure

Environment Variables

| Variable | Default | Description | |----------|---------|-------------| | VOLCANO_SCHEDULER_NS | volcano-system | Scheduler namespace | | VOLCANO_SCHEDULER_LABEL | app=volcano-scheduler | Label selector for scheduler pods |

Limitations

1. Log retention: Logs may be rotated based on cluster configuration
2. Multi-scheduler: If running multiple schedulers, logs will be interleaved
3. Log level: Default log level may not show all debug information
4. Previous logs: --previous only works if the container has restarted

Tips for Effective Log Analysis

1. Use time ranges: Narrow down with --since to focus on recent issues
2. Combine keywords: Search for error\|Failed\|failed to catch all failures
3. Check pod context: Always include --pod when investigating specific pods
4. Look for patterns: Repeating errors may indicate systemic issues
5. Correlate with events: Compare with kubectl get events timestamps

See Also

• volcano-diagnose-pod - Diagnose individual pod issues
• volcano-gang-scheduling - Gang scheduling specific diagnosis
• volcano-queue-diagnose - Queue resource analysis
• volcano-resource-insufficient - Resource shortage diagnosis