tianyudu/marlowe-slurm-operator

Marlowe Slurm operator

This skill adapts the generic Slurm workflow to Stanford's Marlowe Computing Platform. Use Marlowe-specific documented defaults as hints, but verify current cluster state with live commands before acting.

Additional resources

• Use references/marlowe-reference.md for Stanford-specific details, policies, and examples.
• Use templates/preempt-interactive.sh for an interactive preempt shell.
• Use templates/batch-gpu.sbatch for a medium-project GPU batch job.
• Use templates/hero-gpu.sbatch for a large-project GPU batch job.

Core rules

1. Always verify cluster state first with read-only discovery commands.
2. Do not guess Marlowe account names, suffixes, partition limits, GPU counts, memory defaults, or time limits.
3. Follow Marlowe account conventions:
   • Basic account: marlowe-<project-id>
   • Medium-project suffix: marlowe-<project-id>-pmXX
   • Large-project suffix: marlowe-<project-id>-plXX
4. If a job runs on preempt with a medium or large suffix account, assume it consumes GPU hours unless current site docs say otherwise.
5. Respect documented partition caps, but re-check them with sinfo and scontrol show partition before submission.
6. Load required modules in Marlowe job scripts: slurm, nvhpc, and cudnn/cuda12/9.3.0.75.
7. If MPI jobs report network fabric or component errors, try loading gcc/64 after nvhpc.
8. Do not propose workflows that violate Stanford usage policy, such as login-node inference, resource benchmarks on shared nodes, or bypassing login requirements with web tunnels or userspace VPNs.
9. Treat Open OnDemand code-server sessions as CPU-only and limited; use srun, salloc, or sbatch for GPU work.
10. Prefer sbatch --test-only or srun --test-only when the user wants validation rather than immediate execution.

Marlowe defaults to verify

These are documented defaults, not permanent guarantees:

• preempt: basic access, up to 8 nodes, up to 12 hours, preemptible with about 15 minutes notice
• batch: medium projects, up to 16 nodes, up to 2 days, requires a medium suffix account
• hero: large projects, up to 25 nodes, up to 24 hours, requires a large suffix account
• Hardware: DGX H100 nodes with 8 H100 80 GB GPUs and 2 TB memory per node

Verify these with:

scontrol version
scontrol show config
sinfo --summarize
sinfo --Node --long
sinfo -o "%#P %.5a %.10l %.10L %.6D %G"
scontrol show partition

Use scontrol show node <node_name> before making assumptions about per-node GPUs, memory, features, or constraints.

Account and suffix validation

Before submitting, confirm which account form applies:

sacctmgr show assoc where user=$USER format=cluster,account,user,partition,defaultqos,qos,maxjobs,maxsubmitjobs,maxtresperjob,maxtrespernode,maxwall

Use the result plus current site docs to decide:

• preempt without GPU-hour charging intent: prefer the basic marlowe-<project-id> account
• batch: require the correct -pmXX suffix
• hero: require the correct -plXX suffix
• preempt with a suffix account: warn that GPU hours may be charged

If the user forgets a valid account, expect an error similar to ACCOUNT ERROR: Did you remember to set your account?.

If sacctmgr is restricted, say so clearly and fall back to verified site documentation plus known working account examples.

Submission workflow

Interactive shell on preempt

Use a command like:

srun -N <nodes> -G <gpus> -A marlowe-<project-id>[-<suffix>] -p <partition> --pty bash

Before running it:

1. verify the partition exists and is available
2. verify the account form and whether a suffix is required
3. warn if a suffix-coded preempt job will consume GPU hours
4. keep nodes, GPUs, and time inside current partition limits

Interactive allocation

salloc -N <nodes> -A marlowe-<project-id>[-<suffix>] -p <partition> -t <time>

Batch jobs

Use sbatch with explicit directives for account, partition, nodes, GPUs, time, and output paths.

Minimum checklist:

1. validate account and suffix choice
2. validate partition and time cap
3. validate node and GPU counts against visible inventory
4. load required modules in the script
5. dry-run when appropriate:

sbatch --test-only job.sh

Monitoring and GPU-hour accounting

Use standard Slurm commands for job state:

squeue -j <job_id> --start
scontrol show jobid=<job_id>
sstat -j <job_id>
sacct -j <job_id>

For medium and large projects, track GPU-hour usage with sreport:

sreport cluster UserUtilizationByAccount -T gres/gpu Start=<YYYY-MM-DD> End=<YYYY-MM-DD> account=marlowe-<project-id>-<suffix> -t hours

Always verify the suffix in the usage query. If the task involves batch, hero, or suffix-coded preempt jobs, mention whether GPU hours are expected to be consumed.

Permissions and restricted visibility

If sacctmgr, sshare, or sreport output is restricted:

• say which command is unavailable
• say what certainty was lost
• continue with sinfo, scontrol, squeue, and documented Marlowe conventions
• do not invent hidden limits, balances, or suffix mappings

Response checklist

When helping on Marlowe:

1. summarize verified cluster facts first
2. list the exact commands used to verify them
3. show the smallest valid Marlowe command or script
4. separate verified facts from documented defaults and assumptions
5. explicitly call out account suffix requirements and possible GPU-hour charging
6. prefer unknown over a guess