a-green-hand-jack/compute-budget-planner

Compute Budget Planner

Estimate and control compute spend before jobs are submitted. This skill prevents GPU-hour waste from under-scoped smoke tests, over-specified ablations, and jobs that could have been half the cost with a smarter design.

Use this skill when:

• a planned training run's cost is unknown before submission
• a smoke test needs to be sized to give signal without wasting a full GPU-day
• an ablation matrix needs to be costed before scheduling all variants
• a cheaper model variant, shorter sequence, or smaller dataset could answer the same question
• available compute is limited and job priority needs to be set
• a paper's experimental budget needs to be disclosed or defended to reviewers

Do not use this skill to submit jobs — use run-experiment after sizing is done. Do not use this skill for experiment design — use experiment-design-planner first.

Pair this skill with:

• experiment-design-planner to align experiment scope with compute budget before design is locked
• run-experiment to submit the sized job with the correct resource spec
• baseline-selection-audit when baseline replication costs need to be included in the total budget
• research-project-memory to record compute decisions as evidence or risk items

Skill Directory Layout

<installed-skill-dir>/
├── SKILL.md
└── references/
    └── estimation-guide.md

Progressive Loading

• Always read references/estimation-guide.md.
• Read existing job configs in jobs/, configs/, or experiments/ when a previous run exists.
• Read memory/evidence-board.md when the compute estimate is tied to a paper claim.

Core Principles

• Estimate before submitting, not while waiting for the job to time out.
• A smoke test that uses less than 5% of the full job cost should be the default first step.
• Compute cost compounds: one model × one dataset × one seed is the atom; ablations multiply.
• Cheaper alternatives should always be explored before committing to expensive runs.
• Reviewer trust is higher when compute is disclosed and reproducibility is feasible on standard hardware.
• Underestimating compute is a submission risk: a result that cannot be reproduced because the hardware is unavailable is a reviewer concern.

Step 1 — Gather Training Context

Collect:

• model name and parameter count (or approximate FLOPs if known)
• dataset size: number of training examples, sequence length or image size
• planned training duration: epochs or steps
• batch size and gradient accumulation steps
• hardware: GPU type (A100, H100, V100, RTX 3090), VRAM, number of GPUs
• framework and mixed precision: FP16, BF16, or FP32
• existing throughput: samples/second or tokens/second from a prior run (if available)

If no prior run exists, use references/estimation-guide.md to estimate throughput.

Step 2 — Estimate Full Run Cost

Compute:

total_steps = ceil(dataset_size / batch_size) × epochs
             or planned_steps (if step-based)

gpu_hours = total_steps × seconds_per_step / 3600 × num_gpus

Report:

• estimated GPU-hours (single GPU equivalent)
• wall-clock time with the planned number of GPUs
• whether the run fits in the available time limit (e.g., SLURM 24h, RunAI policy)

If throughput is unknown, provide a range using the estimation guide's reference values for the model class and hardware.

Step 3 — Size the Smoke Test

A smoke test should:

• use ≤ 5% of the full run's compute (ideally 1–2%)
• complete within 10–30 minutes on one GPU
• produce a loss curve with a clear downward trend (proves the code runs correctly)
• exercise the full data pipeline, model forward/backward, checkpointing, and logging

Recommend:

Smoke test steps: max(100, 0.01 × total_steps)
Smoke dataset: subsample 1% of training data, or use 1 batch per shard
Expected smoke wall-clock: < 30 min on 1× GPU

Flag if the smallest feasible smoke run exceeds 30 minutes. This is a sign the setup overhead is too high.

Step 4 — Cost the Ablation Matrix

For each ablation dimension, record:

| Variant | Changed variable | Cost multiplier | GPU-hours | Priority | |---|---|---|---|---| | Baseline | — | 1× | | required | | Ablation A | [removed component] | 1× | | required | | Ablation B | [changed config] | 0.5× | | should-have | | Sweep C | [hyperparameter] | 3× | | optional |

Classify each variant as:

• required: paper claim depends on this comparison
• should-have: reviewer will likely ask; defensible to omit with a reason
• optional: nice-to-have; drop first if compute is limited

Report:

• total required GPU-hours
• total should-have GPU-hours
• total optional GPU-hours
• recommended minimum run set

Step 5 — Identify Cheaper Alternatives

Check whether any of the following can answer the same question at lower cost:

• Smaller model / fewer parameters: Can a 125M model prove the concept before scaling to 1B?
• Shorter sequences / smaller images: Does the method work at half resolution?
• Fewer training steps: Can early stopping or learning rate warmup give a valid comparison at 50% of steps?
• Smaller dataset: Can a 10% subsample reproduce the trend with ±5% relative error?
• Lower-rank approximation: Can LoRA or adapter training substitute for full fine-tuning for this ablation?
• Faster hardware: Can an A100 replace 4× V100s and finish in one-quarter the wall-clock time?

For each alternative, record the expected signal quality and the risk of a misleading result.

Step 6 — Write the Budget Plan

Save to:

code/.agent/compute-budget.md

The plan should include:

• full run estimate (GPU-hours and wall-clock)
• smoke test spec (steps, data fraction, expected wall-clock)
• ablation matrix with priority and cost
• cheaper alternatives considered and rejected or accepted
• available resource envelope and whether budget fits
• recommended submission order: smoke → required ablations → should-have → optional

Memory Writeback

• If compute cost affects a paper claim's feasibility, add to memory/risk-board.md
• If a cheaper alternative is accepted and changes the experimental design, update memory/decision-log.md
• Update memory/evidence-board.md when compute details need to be cited in the paper (hardware, training duration, GPU-hours)

Final Sanity Check

Before finalizing:

• smoke test spec is under 30 minutes on 1 GPU
• full run estimate fits available time limit
• ablation matrix has priorities assigned
• cheaper alternatives were considered
• required GPU-hours are within the available resource envelope or a risk entry is filed