athola/cpu-gpu-performance

Table of Contents

• When to Use
• Required TodoWrite Items
• Step 1: Establish Current Baseline
• Step 2: Narrow the Scope
• Step 3: Instrument Before You Optimize
• Step 4: Throttle and Sequence Work
• Step 5: Log Decisions and Next Steps
• Output Expectations

CPU/GPU Performance Discipline

When To Use

• At the beginning of every session (auto-load alongside token-conservation).
• Whenever you plan to build, train, or test anything that could pin CPU cores or GPUs for more than a minute.
• Before retrying a failing command that previously consumed significant resources.

When NOT To Use

• Simple operations with no resource impact
• Quick single-file operations

Required TodoWrite Items

1. cpu-gpu-performance:baseline
2. cpu-gpu-performance:scope
3. cpu-gpu-performance:instrument
4. cpu-gpu-performance:throttle
5. cpu-gpu-performance:log

Step 1: Establish Current Baseline

• Capture current utilization:

  • uptime
  • ps -eo pcpu,cmd | head
  • nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv

  Note which hosts/GPUs are already busy.

• Record any CI/cluster budgets (time quotas, GPU hours) before launching work.

• Set a per-task CPU minute / GPU minute budget that respects those limits.

Step 2: Narrow the Scope

• Avoid running "whole world" jobs after a small fix. Prefer diff-based or tag-based selective testing:
  • pytest -k
  • Bazel target patterns
  • cargo test <module>
• Batch low-level fixes so you can validate multiple changes with a single targeted command.
• For GPU jobs, favor unit-scale smoke inputs or lower epoch counts before scheduling the full training/eval sweep.

Step 3: Instrument Before You Optimize

• Pick the right profiler/monitor:
  • CPU work:
    • perf
    • intel vtune
    • cargo flamegraph
    • language-specific profilers
  • GPU work:
    • nvidia-smi dmon
    • nsys
    • nvprof
    • DLProf
    • framework timeline tracers
• Capture kernel/ops timelines, memory footprints, and data pipeline latency so you have evidence when throttling or parallelizing.
• Record hot paths and I/O bottlenecks in notes so future reruns can jump straight to the culprit.

Step 4: Throttle and Sequence Work

• Use nice, ionice, or Kubernetes/Slurm quotas to prevent starvation of shared nodes.
• Chain heavy tasks with guardrails:
  • Rerun only the failed test/module
  • Then (optionally) escalate to the next-wider shard
  • Reserve the full suite for the final gate
• Stagger GPU kernels (smaller batch sizes or gradient accumulation) when memory pressure risks eviction; prefer checkpoint/restore over restarts.

Step 5: Log Decisions and Next Steps

Conclude by documenting the commands that were run and their resource cost (duration, CPU%, GPU%), confirming whether they remained within the per-task budget. If a full suite or long training run was necessary, justify why selective or staged approaches were not feasible. Capture any follow-up tasks, such as adding a new test marker or profiling documentation, to simplify future sessions.

Output Expectations

• Brief summary covering:
  • baseline metrics
  • scope chosen
  • instrumentation captured
  • throttling tactics
  • follow-up items
• Concrete example(s) of what ran (e.g.):
  • "reran pytest tests/test_orders.py -k test_refund instead of pytest -m slow"
  • "profiled nvidia-smi dmon output to prove GPU idle time before scaling"

Exit Criteria

• [ ] uptime and ps baseline captured and recorded before any build, training run, or test suite starts
• [ ] Scope narrowed to diff-based or tag-based targets (e.g., pytest -k, cargo test <module>); full-suite justification documented if selective approach was not feasible
• [ ] Output summary includes: duration, CPU% or GPU% consumed, and whether the run stayed within the per-task budget
• [ ] Any follow-up tasks (new test markers, profiling docs) written to a todo or issue so they survive the session