amdpilot-org/auto-benchmark-rocm

Auto Benchmark — AMD ROCm / MI355X

This skill is for repeatable, AI-driven performance tuning of vLLM and SGLang on AMD Instinct GPUs.

Adapted from the upstream SGLang sglang-auto-benchmark skill. All CUDA-specific references have been replaced with ROCm equivalents; attention backends, environment variables, resource classes, and Docker image selection reflect the AMD MI355X (gfx950) and MI300X (gfx942) ecosystem.

Preconditions

• GPU architecture confirmed: run rocminfo | grep gfx to verify gfx950 (MI355X) or gfx942 (MI300X).
• ROCm version checked: cat /opt/rocm/.info/version — this skill targets ROCm 7.x (7.0 or 7.2).
• Docker image identified: know which base image you are in (rocm/sgl-dev or rocm/vllm-dev). Use the env-probe skill first if unsure.
• Server can launch: vLLM or SGLang can already start and serve the target model in this container.
• Model path exists: local path under /data/ preferred over NFS /mnt/dcgpuval/ for I/O speed.
• Goal is clear:
  • benchmark a fixed QPS list, or
  • search the maximum QPS that satisfies max_ttft_ms / max_tpot_ms.

If any precondition is not met, fix it before running a large search.

Most Important Rule

If the user wants the best command for a real production or real workload scenario, the benchmark must use their real request distribution — real prompt lengths, output lengths, multi-turn patterns, and sampling settings. sharegpt, random, and generated-shared-prefix are useful for sanity checks, but they are not a substitute for real traffic.

AMD-Specific Attention Backends

On ROCm, the available attention backends differ from NVIDIA:

| Backend | Description | Notes | |---------|-------------|-------| | aiter | ROCm-native unified attention kernel library | Primary AMD backend, best for MI355X/MI300X | | triton | Cross-platform Triton kernels | Works on ROCm, good fallback | | torch_native | PyTorch native SDPA | Baseline, no AMD-specific optimization |

Do NOT use NVIDIA-specific backends: fa3, fa4, flashinfer, flashmla, trtllm_*, cutlass_*. These will fail or silently produce wrong results on ROCm.

aiter-Specific Constraints

• MLA (Multi-Head Latent Attention): aiter ASM kernels require heads_per_gpu % 16 == 0. For models with 64 heads, TP must be ≤ 4 (giving 16 heads/GPU).
• FP8 prefill attention on gfx950: enable with SGLANG_AITER_FP8_PREFILL_ATTN=1.
• MLA persist design for FP8 KV cache: enable with SGLANG_AITER_MLA_PERSIST=1.

Environment Variables

Replace CUDA environment variables with ROCm equivalents:

| CUDA (do not use) | ROCm equivalent | Purpose | |--------------------|-----------------|---------| | CUDA_VISIBLE_DEVICES | HIP_VISIBLE_DEVICES | GPU device selection | | CUDA_LAUNCH_BLOCKING | HIP_LAUNCH_BLOCKING | Synchronous kernel launch for debugging | | — | SGLANG_USE_AITER=1 | Explicitly enable aiter backend | | — | SGLANG_AITER_MLA_PERSIST=1 | Enable MLA persist design | | — | SGLANG_AITER_FP8_PREFILL_ATTN=1 | FP8 prefill on gfx950 | | — | HSA_FORCE_FINE_GRAIN_PCIE=1 | Fine-grain PCIe for host-device transfers |

GPU Topology and Resource Classes

Our node has 8× AMD Instinct MI355X (gfx950), ROCm 7.2.0.

| Resource class | GPU count | Typical use case | |----------------|-----------|------------------| | single-gpu | 1 | Small models (≤8B), quick sanity checks | | multi-gpu | 4 | Medium models (32B–70B), TP=4 | | full-node | 8 | Large models (70B+), TP=8 or TP=4×DP=2 |

When server.parallel is used and dp_size is not set explicitly:

dp_size = visible_gpus / (tp_size * pp_size)

Visible GPU count is inferred from HIP_VISIBLE_DEVICES, or from server.parallel.gpu_count.

Supported Dataset Kinds

Same as upstream:

• sharegpt — auto-download supported, converted to canonical JSONL.
• custom — old bench_serving format or canonical autobench JSONL.
• random — synthetic/random benchmark path.
• generated-shared-prefix — shared-prefix synthetic generator.

Canonical Dataset Format

Identical to upstream. JSONL, one request per line:

{"prompt": "Write a summary.", "output_len": 256}
{"prompt": [{"role": "user", "content": "Summarize."}], "output_len": 256}

Search Tiers

• Tier 1: Fast sanity sweep. Baseline + small one-at-a-time scan.
• Tier 2: Good default. Small cartesian on high-priority keys + expansion for rest.
• Tier 3: Full cartesian product. Slowest, but thorough when space is bounded.

YAML key order matters. Put the most important search keys first.

What Is Tunable (AMD/ROCm)

server.base_flags and server.search_space are passed to the server launcher. Any valid vLLM/SGLang server CLI flag can be set or searched.

Kernel / Backend (AMD-specific)

• attention_backend — search [aiter, triton]
• prefill_attention_backend — if split prefill/decode is supported
• decode_attention_backend — if split prefill/decode is supported
• sampling_backend

Batching / Scheduling

• max_running_requests
• chunked_prefill_size — common values: [4096, 8192, 16384, 131072]
• prefill_max_requests
• max_prefill_tokens
• schedule_policy — [lpm, fcfs]
• schedule_conservativeness
• num_continuous_decode_steps

Memory / Cache

• mem_fraction_static — critical for ROCm; MI355X HBM3e is larger than H100, so ranges differ. Typical search: [0.80, 0.85, 0.88, 0.90]
• max_total_tokens
• page_size
• disable_radix_cache
• kv_cache_dtype — [auto, fp8_e4m3] for FP8 KV cache on MI355X

Parallel / Distributed

• tp_size — must respect aiter head constraints (heads_per_gpu % 16 == 0)
• pp_size
• dp_size
• load_balance_method
• enable_dp_attention
• enable_aiter_allreduce_fusion — AMD-specific distributed optimization

Runtime / HIP Graph

• Keep HIP graph enabled by default for performance benchmarking (same concept as CUDA graph on ROCm).
• cuda_graph_max_bs — flag name is unchanged in vLLM/SGLang even on ROCm
• disable_cuda_graph_padding
• Do not put disable_cuda_graph into the default search space.

Optional Speculative / EAGLE Stage

Speculative decoding support on ROCm may be limited. Verify availability before enabling:

• speculative_num_steps
• speculative_eagle_topk
• speculative_num_draft_tokens

Order: always tune the non-speculative base server first, then optionally add speculative search.

Base Tuning Before EAGLE

Never start by tuning EAGLE first. Use this order:

1. Tune the non-speculative base server first.
2. Find the best normal config for the target dataset and SLA.
3. Only if the user explicitly asks and draft model assets exist, run speculative search.

Running The Workflow

Prepare dataset

python3 -m sglang.auto_benchmark convert \
  --kind sharegpt \
  --tokenizer /data/meta-llama/Meta-Llama-3.1-70B-Instruct \
  --num-prompts 1200 \
  --output /tmp/sharegpt.autobench.jsonl

Run from config

python3 -m sglang.auto_benchmark run --config /path/to/config.yaml

Outputs

• Prepared canonical dataset JSONL
• Per-run results.jsonl
• Summary results.csv
• Per-candidate server logs

Integration with amdpilot

When running benchmarks through the amdpilot executor, the benchmark task spec maps to task_spec_json in the queue DB:

{
  "gpu_count": 8,
  "resource_class": "full-node",
  "base_image": "rocm/sgl-dev:v0.5.9-rocm720-mi35x-20260317",
  "gpu_free_ids": [0, 1, 2, 3, 4, 5, 6, 7],
  "disk_free_gb": 2100,
  "timeout_minutes": 120
}

The dashboard GET /api/{job_name}/system_info endpoint reads from this column to display experiment runtime info (GPU arch, ROCm version, base image, resource class).

Benchmark results should be written as structured artifacts so the dashboard can render them and feed them into the data flywheel for downstream LoRA/SFT training signal.

Config Template

Use the reference configs in references/:

| Config | Model | GPUs | Notes | |--------|-------|------|-------| | config-example-rocm.yaml | Generic | 4 | Starting template | | llama3.1-70b-mi355x.yaml | Llama 3.1 70B Instruct | 8 | Full-node TP=8 | | qwen3-32b-mi355x.yaml | Qwen3 32B | 4 | TP=4, aiter search | | deepseek-r1-mi355x.yaml | DeepSeek R1 671B (FP8/MXFP4) | 8 | FP8 KV cache, AllReduce fusion |

What To Report Back

After a run, summarize:

• Hardware: GPU arch (gfx950/gfx942), ROCm version, Docker image tag
• Search config: which tier, dataset kind (synthetic vs real)
• Best config found: attention backend, TP/DP, key flags
• Best QPS that satisfied SLA (or fixed QPS results)
• Whether speculative tuning was skipped or run
• Paths to artifacts: dataset JSONL, results.jsonl, results.csv, server logs
• Anomalies: OOM events, HIP graph capture failures, aiter constraint violations

Differences From Upstream (CUDA) Skill

| Aspect | CUDA / NVIDIA | ROCm / AMD | |--------|---------------|------------| | GPU visibility | CUDA_VISIBLE_DEVICES | HIP_VISIBLE_DEVICES | | Attention backends | fa3, flashinfer | aiter, triton | | Graph runtime | CUDA graph | HIP graph (flag names unchanged — cuda_graph_max_bs, disable_cuda_graph etc. still use "cuda" prefix in vLLM/SGLang CLI even on ROCm) | | GPU query | nvidia-smi | rocm-smi --showproductname | | Arch detection | N/A | rocminfo \| grep gfx | | AllReduce fusion | N/A | --enable-aiter-allreduce-fusion | | FP8 prefill | Built-in | SGLANG_AITER_FP8_PREFILL_ATTN=1 | | MLA persist | Built-in | SGLANG_AITER_MLA_PERSIST=1 | | Memory range | 0.85–0.92 typical | 0.80–0.90 typical (HBM3e larger) | | Docker images | nvcr.io/nvidia/* | rocm/sgl-dev, rocm/vllm-dev |