awslabs/hyperpod-cluster-debugger

HyperPod Cluster Debugger

Operating policy. Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a Suggested command (run this yourself) block and wait for the customer to run it. Destructive order: investigate → reboot → replace (replace destroys root + secondary volumes; not supported on Slurm controller nodes).

Before any state-changing CLI: ask if it's IaC-managed. HyperPod clusters, SGs, EKS access entries, and IAM are usually provisioned via CloudFormation / CDK / Terraform. If yes, the fix belongs in IaC — running the CLI will drift and the next deploy reverts it. Use the CLI only when IaC is unavailable (locked out, predates IaC, mid-review).

scripts/diagnose-cluster.sh is read-only: it collects state via AWS APIs (and SSM for Slurm controller health) and prints each issue as [FAIL] ... → references/<file>.md § <section>.

| Reference | Open when | | ------------------------------------------------------------------------- | ------------------------------------------------------------------- | | cluster-diagnostics-detail.md | Per-finding remediation runbook (§ A–L) | | cluster-operations.md | Operational deep-dives (EFA SG, EKS access, SSM, Slurm, filesystem) | | cloudformation-errors.md | § H needs the full per-resource CFN error catalog | | capacity-planning.md | § B or --validate flags capacity / subnet sizing | | lifecycle-scripts.md | § C points at a specific lifecycle failure | | iam-permissions.md | Full IAM policy for the diagnostic |

---

Workflow

1. Collect HyperPod cluster name (not EKS name), region, exact error string.
2. Run scripts/diagnose-cluster.sh (or --validate for pre-create).
3. For every [FAIL] line, Read the referenced section.
4. Present finding, root cause, and the Suggested-command block verbatim. Wait for customer approval.
5. Re-run the diagnostic to confirm.

---

Step 1: Run diagnostics

# Diagnose an existing cluster:
bash scripts/diagnose-cluster.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>

# Pre-flight (no cluster needed) — validates SGs, subnets, IAM, VPC endpoints,
# optionally S3 lifecycle scripts and per-AZ capacity:
bash scripts/diagnose-cluster.sh --validate --region <REGION> \
  --sg-ids <sg-1,sg-2> --subnet-ids <sub-1,sub-2> [--iam-role <role-arn>] \
  [--s3-uri s3://<BUCKET>/path/] [--instance-type ml.p5.48xlarge]

Pass --instance-type when the target instance type is known — enables the per-AZ capacity check (warns if none of the provided subnets are in an AZ that offers that type, which causes insufficient-capacity failures at creation time).

Tags: [PASS] · [FAIL] (counted, has → references/... pointer) · [WARN] · [INFO]. Priorities: P0 blocks operation · P1 degraded · P2 informational.

---

Step 2: Match signal → section

Error messages / events:

| Signal | Section | | ---------------------------------------------------------------------------- | -------------------------------------------------------------- | | "EFA health checks did not run successfully" (public-doc verbatim signal) | A: EFA Health Checks | | Insufficient-capacity or AZ-mismatch failure at creation | B: Capacity & AZ | | Lifecycle-script failure or timeout during provisioning | C: Lifecycle Scripts | | kubectl auth error (server asks for credentials / no API group list) | D: EKS Access | | InService but not all instances visible | E: Cluster Provisioning | | "Target is not connected" / SSM errors | F: SSM Connectivity | | Node replacement not happening / batch-replace not working | G: Node Replacement | | "Embedded stack failed" / any CloudFormation error | H: CloudFormation Errors | | UpdateClusterSoftware failed or cluster in post-maintenance rollback state | J: AMI & Cluster Updates | | Dangling / orphaned nodes in EKS vs list-cluster-nodes | K: Dangling Nodes & Cleanup | | Cluster Autoscaler breaks after HyperPod attached | L: Autoscaler Compatibility | | Slow I/O, FSx throughput saturated | cluster-operations.md § 9 | | Slurm node name → instance ID lookup | I: Utilities |

---

A: EFA Health Checks

SG missing self-reference. Add inbound + outbound self-ref to every SG on the cluster, plus least-privilege egress for the AWS APIs the node needs (HTTPS 443 to S3 / ECR / SageMaker / SSM / STS / CloudWatch Logs — via VPC-endpoint prefix-lists when possible). Full procedure: cluster-diagnostics-detail.md § A.

B: Capacity & AZ

Instance type unavailable in the requested AZ. Verify with describe-instance-type-offerings, then change AZ, use Flexible Training Plans, or request ODCR. Full: § B · strategy: capacity-planning.md.

C: Lifecycle Scripts

Script failed or timed out during provisioning. Read CloudWatch under /aws/sagemaker/Clusters/<name>/<id> — common causes: missing S3 VPC endpoint, IAM gap, CRLF line endings, instance-group name mismatch. Full: § C · layout: lifecycle-scripts.md.

D: EKS Access / kubectl

IAM identity not in EKS access entries. Verify with sts get-caller-identity, create an access entry with admin policy, update kubeconfig. Full: § D.

E: Cluster Provisioning

InService without all instances is expected under Continuous Provisioning — failures surface as events, not cluster errors. For stuck Creating/Updating/Deleting: check CFN nested stacks (§ H), IAM, capacity, events; if stuck Deleting check VPC ENI dependencies. Full: § E.

F: SSM Connectivity

Target is not connected: use sagemaker-cluster:<CLUSTER_ID>_<GROUP>-<INSTANCE_ID> format (not raw EC2 ID), install session-manager-plugin, confirm node Running. Check IAM + VPC endpoints on timeouts. Full: § F.

G: Node Replacement

Auto-repair: confirm NodeRecovery=Automatic, check Health Monitoring Agent (HMA) logs + node labels / Slurm reason, confirm capacity. Manual: reboot first, replace only if reboot fails. Replace requires the cluster to have been patched via UpdateClusterSoftware at least once and cannot target a Slurm controller node. Full: § G.

H: CloudFormation Errors

Embedded stack failed hides the real error. Drill into nested stacks via Events tab (filter Failed) until you reach a non-stack resource. CLI: describe-stack-events --query 'StackEvents[?ResourceStatus==\CREATE_FAILED`]'`. Also covers SLR creation failures and permission-boundary denials. Full: § H · catalog: cloudformation-errors.md.

I: Utilities

Map Slurm node names (ip-10-x-y-z) to HyperPod instance IDs via list-cluster-nodes or on-node /opt/ml/config/resource_config.json. Full: § I.

J: AMI & Cluster Updates

UpdateClusterSoftware fails and rolls back, or the cluster stays in a post-maintenance rollback state. Common causes: lifecycle script incompatible with new AMI, HMA version too old, insufficient rolling-update capacity. If the cluster has active nodes, collect diagnostics and escalate rather than delete-and-recreate. Full: § J.

K: Dangling Nodes & Cleanup

Nodes in kubectl get nodes but not in list-cluster-nodes (ghost EKS nodes), or the inverse (HyperPod nodes that never registered kubelet). Script flags both. Full: § K.

L: Autoscaler Compatibility

Cluster Autoscaler errors on HyperPod provider IDs and breaks autoscaling for all node groups. No officially endorsed workaround — escalate to AWS Support. Karpenter does not conflict with HyperPod nodes by default. Full: § L.

---

Prerequisites

• aws CLI v2.13+ authenticated to the cluster's account
• jq, python3, bash 4.2+
• kubectl authenticated to the EKS cluster (EKS checks skipped if absent)
• session-manager-plugin (Slurm controller health checks only)

IAM policy: references/iam-permissions.md.

Defaults

• Region — required: pass --region or set $AWS_DEFAULT_REGION.
• Mode — --cluster <NAME> (diagnose) or --validate (pre-create).
• Event window — up to 500 most recent events (5 × 100, paginated).
• Colors — auto-disabled on non-TTY; --no-color to force off.

Error handling

| Failure | Script | Tell the customer | | --------------------------------------------------- | ---------------------------------------------------------- | ----------------------------------------------------- | | aws sts get-caller-identity fails | Exit 1 | "Fix AWS credentials and rerun." | | Cluster not found | Exit 1 after listing region's clusters | "Confirm HyperPod cluster name (not EKS) and region." | | sagemaker:* / ec2:* / eks:* / logs:* denied | Warn, add Missing IAM permission for <API>, continue | "Grant the listed IAM action and rerun." | | kubectl absent or unauthenticated | Skip EKS checks (access entries, add-ons, aws-auth, nodes) | "Install/authenticate kubectl." | | session-manager-plugin absent (Slurm) | Skip Slurm controller probe | "Install session-manager-plugin." | | SSM throttled / times out (180s) | Retry with backoff; warn and continue if still failing | "Rerun later — script is idempotent." | | CloudWatch log group not found | Skip CloudWatch check | "CloudWatch not configured on this cluster." |

Exit codes: 0 no critical failures · 1 one or more critical failures (cluster not found, fatal prerequisite missing, or any [FAIL] in diagnose or --validate mode). [WARN] lines do not affect the exit code.

Skill delegation

| Need | Use | | ------------------------------- | -------------------------- | | Shell on nodes | hyperpod-ssm | | Version comparison across nodes | hyperpod-version-checker |

Escalate to AWS Support

Escalate when:

1. EFA health checks fail despite correct SG rules.
2. Capacity errors persist despite a valid Flexible Training Plan / ODCR.
3. Node replacement fails repeatedly without clear events / log signal.
4. Cluster stuck in a non-terminal state (Creating, Updating, or a post-maintenance rollback state) for an extended period.
5. CloudFormation root-cause is an internal service error.

Before opening the case

Run these commands and attach the output. Goal: AWS Support has everything at case open.

# 1. Cluster identity + status (confirms region, ARN, orchestrator, instance groups)
aws sagemaker describe-cluster --cluster-name <CLUSTER> --region <REGION>

# 2. Full cluster-level diagnostic bundle
bash scripts/diagnose-cluster.sh --cluster <CLUSTER> --region <REGION> > diag.txt

# 3. Per-node log/config bundle to S3 (delegates to hyperpod-issue-report skill)
#    See skills/hyperpod-issue-report/SKILL.md for the exact invocation.

Include in the case

• Cluster name + ARN (or ClusterId suffix) and AWS region
• ClusterStatus + FailureMessage from describe-cluster
• Timestamp window (UTC start / end) of the failure
• Exact error strings observed (copy verbatim from events / logs / console)
• Affected instance IDs / NodeLogicalIds / instance group names
• diag.txt from step 2 above
• S3 URI of the hyperpod-issue-report bundle from step 3