awslabs/hyperpod-slurm-debugger

HyperPod Slurm Debugger

Diagnostic-only. Identify and classify Slurm scheduler and node-daemon issues on HyperPod Slurm clusters. Do not run, recommend, or print any state-mutating command. For remediation, link to the official AWS or Slurm documentation.

When to invoke

Invoke when the user reports any of the symptoms in the decision table.

When NOT to invoke

• Cluster has Orchestrator.Eks — invoke hyperpod-node-debugger or hyperpod-nccl.
• Single-node hardware fault with healthy Slurm scheduler — invoke hyperpod-node-debugger.
• NCCL training-hang investigation — invoke hyperpod-nccl.
• Node unreachable via SSM — invoke hyperpod-ssm.

Constraints

• Read-only. Do not run, recommend, or print state-mutating commands.
• For any remediation, link to AWS or Slurm docs. The user authorizes and executes.
• IaC-managed cluster (Terraform / CloudFormation / CDK): warn that direct mutation drifts the live state from the IaC plan.

Canonical recovery URLs: references/slurm-details.md → Authoritative recovery documentation.

Prerequisites

• AWS CLI v2, authenticated for the target account and region with permissions:
  • sagemaker:DescribeCluster, sagemaker:ListClusterNodes
  • ssm:StartSession on the HyperPod-created SSM document
• Session Manager plugin installed locally.
• jq ≥ 1.6.
• unbuffer (from the expect package). Required — without it aws ssm start-session returns empty stdout intermittently with Cannot perform start session: EOF and every check silently misreports. Install: expect package on Amazon Linux / RHEL / Debian / Ubuntu / macOS. Script exits at prerequisite check if missing.

Procedure

Step 1 — Collect inputs

Ask the user for:

1. HyperPod cluster name (not Slurm partition name).
2. AWS region.
3. Optional: a specific Slurm node name.

Step 2 — Confirm orchestrator

aws sagemaker describe-cluster --cluster-name <NAME/ARN> --region <REGION> \
  --query 'Orchestrator' --output json

If Orchestrator.Eks is present, stop. Route per When NOT to invoke.

Step 3 — Run the diagnostic script

bash scripts/slurm-diagnose.sh --cluster <NAME> --region <REGION>
# Scope to a node:
bash scripts/slurm-diagnose.sh --cluster <NAME> --region <REGION> --node <SLURM_NODE>

Relay the script output to the user verbatim.

Step 4 — Map findings → docs

For each finding, look up the section in the decision table and link the user to the corresponding AWS / Slurm doc. Do not type out remediation commands.

Decision table

| Symptom (sinfo -o "%N %T %30E" or script finding) | Section | | ----------------------------------------------------------- | ------------------------------------------------------ | | Node state = down or down*, reason other than below | A: Node Down | | Node state = down*, Reason = Node unexpectedly rebooted | B: Unexpected Reboot | | Jobs PENDING with REASON=Resources while nodes are idle | C: Controller State | | Jobs stuck COMPLETING after node replacement | C: Controller State | | scontrol ping returns DOWN for the controller | C: Controller State | | GRES (GPU) counts incorrect or not released | C: Controller State | | state=fail issued but no recovery occurred | D: Action Reason Mismatch | | Accounting errors or RPC errors mentioning dbd | C: Controller State (slurmdbd) | | slurm.conf edited; new partitions or nodes not visible | C: Controller State (config) | | Job exited on a hardware failure but did not restart | E: Auto-resume |

Defaults

| Behavior | Default | Override | | -------------------- | -------------------------------------------------------------------------------------------------- | -------------------------- | | Mode | read-only — always; no remediation flag exists | n/a | | Region | $AWS_DEFAULT_REGION, falling back to us-east-1 | --region <R> | | Scope | all nodes in down / drain / fail / "unexpectedly rebooted" | --node <SLURM_NODE_NAME> | | Output | colorized terminal | --no-color | | SSM target format | sagemaker-cluster:<clusterId>_<instanceGroupName>-<instanceId> (derived) | n/a | | Controller discovery | --controller-group (if set) → SlurmConfig.NodeType=Controller → provisioning_parameters.json | --controller-group <N> |

Error handling

| Failure | Skill behavior | Required user action | | -------------------------------------------------- | -------------------------------------- | ----------------------------------------------- | | describe-cluster fails | Print AWS error; exit 1 | Fix credentials/region; verify cluster name | | Cluster has Orchestrator.Eks | Exit 1 with pointer to EKS-side skills | Use hyperpod-node-debugger or hyperpod-nccl | | session-manager-plugin missing / SSM unreachable | sinfo returns empty; exit 1 | Install plugin; verify node InService | | Disk ≥ 95 % full on a down node | Report finding disk-full-<node> | Refer to AWS troubleshooting docs | | Missing jq or aws | Exit 1 at prerequisite check | Install per Prerequisites |

---

A: Node Down

Node is down because slurmd stopped responding. Causes: slurmd crash, disk full, OOM, network partition, hardware fault.

Script checks: systemctl is-active slurmd, srun -w <NODE> hostname (RPC layer), disk, memory.

Link: https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md

If node returns to down after a manual resume → escalate to hyperpod-node-debugger.

Context: references/slurm-details.md § A.

---

B: Unexpected Reboot

Node is down* with Reason "Node unexpectedly rebooted" because slurmd re-registered after an out-of-band reboot. Upstream Slurm behavior, not HyperPod. Node is typically healthy.

Links:

• https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md
• https://slurm.schedmd.com/scontrol.html (state=resume semantics)

If node reboots again within minutes → escalate to hyperpod-node-debugger.

Context: references/slurm-details.md § B.

---

C: Controller State

slurmctld in-memory state can desync from the on-disk state. A controller restart reloads from StateSaveLocation and clears bad caches. User decides and executes.

Restart may help:

| Symptom | Why | | -------------------------------------------------- | ------------------------------------------- | | PENDING with REASON=Resources, idle nodes | Re-evaluates the queue | | Jobs stuck COMPLETING after node replacement | Controller held a reference to the old node | | GRES (GPU, EFA) not released after a job ends | Resource accounting de-synced | | Nodes stuck Unknown after reboot, slurmd is up | Re-registration was not processed | | scontrol ping times out | Controller event loop is hung | | Lost connection to slurmdbd / RPC errors | DBD connection wedged |

Do NOT restart when:

• HyperPod replacement (Action:Replace) in progress on any node — concurrent changes fail the replacement.
• Only one compute node is bad — restart slurmd on that node.
• sinfo and squeue are responsive — problem is elsewhere.
• journalctl -u slurmctld not reviewed yet — panic / OOM will reproduce.
• slurm.conf was just edited — try scontrol reconfigure first.

Folded triggers

• slurmdbd disconnected — sacct fails, accounting fields show Unknown, controller log spams Unable to contact slurmdbd. Restore slurmdbd before considering controller restart. https://slurm.schedmd.com/accounting.html · details.
• Stale config — slurm.conf / topology.conf mtime > slurmctld start. scontrol reconfigure first; restart is fallback. https://slurm.schedmd.com/scontrol.html · details.

Restart procedure / what's preserved:

• https://slurm.schedmd.com/slurmctld.html
• https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md

Context: references/slurm-details.md § C.

---

D: Action Reason Mismatch

scontrol update state=fail reason=... was issued with a reason that does not match Action:Reboot or Action:Replace exactly. HyperPod silently ignores anything else. Script detects near-misses on nodes in fail state.

Required strings (case-sensitive, no whitespace, no punctuation):

• Action:Reboot
• Action:Replace

Link: https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency-slurm-replace-faulty-instance.html

Context: references/slurm-details.md § Action reason-string validation.

---

E: Auto-resume

--auto-resume=1 is an srun step option. It re-runs the step after HMA (the Health Monitoring Agent) flags a node and Automatic node recovery replaces it.

Why it didn't restart the job:

• Flag on sbatch not srun — per-step; sbatch directives are silently ignored.
• HMA did not flag the node — failure was application/transient, not hardware. Step exits as a normal Slurm failure.
• Cluster NodeRecovery is None — faulty nodes are labeled but not replaced.
• No checkpointing — step restarts from process zero each iteration.
• AMI predates HMA support (released 2025-09-11) — needs AMI / cluster-software update.

Link: https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency-slurm-auto-resume.html

Context: references/slurm-details.md § HyperPod auto-resume.

---

Escalation

| Condition | Next skill | | --------------------------------------------------------------- | ------------------------------------- | | Node returns to down shortly after a manual resume | hyperpod-node-debugger (hardware) | | slurmd logs contain CUDA / NVIDIA / XID errors | hyperpod-node-debugger § G | | Disk full or /dev/shm exhausted | hyperpod-node-debugger § I | | Node unreachable via SSM | hyperpod-ssm | | Controller restart does not clear COMPLETING after 2 attempts | hyperpod-issue-report + AWS Support |