BBuf/sglang-torch-profiler-analysis

SGLang Torch Profiler Analysis

Overview

Use this skill for SGLang torch.profiler analysis.

There is only one public workflow:

• triage

Use the unified entrypoint:

• scripts/analyze_sglang_torch_profile.py

triage always prints the same three tables:

• kernel table
• overlap-opportunity table
• fuse-pattern table

By default, all three tables only render rows at or above 1.0% cumulative GPU-time share. Treat anything below that as noise unless the user explicitly asks for a lower cutoff.

The script-level fuse-pattern table should stay source-backed and deterministic. Do not build a fuzzy string-matching engine into the script for typo-tolerance.

If exact/source-backed matching is weak but the agent judges that a cluster of kernels still looks semantically close to a known pattern, add a short AI note after the table with one of these labels:

• high: very likely the same pattern family; naming drift or minor implementation reshaping is the main uncertainty
• medium: several signals line up, but one important piece is still ambiguous
• low: weak resemblance only; mention it only if it is still worth a human follow-up

When To Use It

• inspect an SGLang torch profiler trace or profile directory
• profile a live SGLang server and immediately analyze the output
• summarize which kernel families dominate prefill or decode
• map kernels back to Python code paths
• judge whether a code path still has overlap headroom
• check whether an already-known fusion or overlap path should have applied

Diffusion Backend Gate

For diffusion benchmark or profiling work, only analyze traces produced by the native SGLang diffusion backend.

If the run that generated the trace logs any of:

• Falling back to diffusers backend
• Using diffusers backend
• Loaded diffusers pipeline

stop the workflow instead of analyzing the trace. Treat it as a backend-selection issue, not as valid SGLang diffusion profiler evidence.

Main Flows

1. Single-trace triage from an existing profile dir or trace

python3 scripts/analyze_sglang_torch_profile.py \
  --input /path/to/profile_dir_or_trace.json.gz

Use this when you want the fastest read on kernel share and likely fused-kernel pattern matches. The overlap table stays conservative in single-trace mode and will tell you when a mapping/formal pair is needed.

2. Single-trace triage from a running server

python3 scripts/analyze_sglang_torch_profile.py \
  --url http://127.0.0.1:30000 \
  --num-steps 5 \
  --profile-by-stage

3. Two-trace triage from existing profile dirs or traces

python3 scripts/analyze_sglang_torch_profile.py triage \
  --mapping-input /path/to/graph_off_profile_dir \
  --formal-input /path/to/graph_on_profile_dir

Use this when you need stronger overlap conclusions and cleaner kernel-to-source attribution.

4. Two-trace triage from running servers

python3 scripts/analyze_sglang_torch_profile.py triage \
  --mapping-url http://127.0.0.1:31025 \
  --formal-url http://127.0.0.1:31026 \
  --num-steps 5 \
  --profile-by-stage

profile_by_stage

profile_by_stage is not only for PD disaggregation.

• On ordinary non-PD serving, it is still useful because prefill and decode usually have very different bottlenecks.
• On the current profile-v2 path inside SGLang, stage-based profiling is effectively the normal path.
• PD-disaggregated serving adds one extra rule: prefill workers and decode workers must be profiled separately. That is stricter than ordinary profile_by_stage.

How To Choose The Triage Shape

Single-trace triage

Use when you want the lowest-friction report:

• one trace is already available
• you mainly want kernel share and fusion clues
• you are comparing two runs side by side by running triage once per trace

This is the recommended default.

Two-trace triage

Use when you need:

• a stronger answer about overlap headroom
• graph-off source mapping plus graph-on final behavior
• more trustworthy overlap recommendations in the middle table

1. mapping trace with --disable-cuda-graph --disable-piecewise-cuda-graph
2. formal trace with the real serving optimizations enabled

Do not call the mapping pass a "fast profile". It exists to recover kernel -> cpu_op -> python scope.

Workflow

Single-trace workflow

1. If the user only wants a quick diagnosis, one trace is enough.
2. Prefer rank-local TP-0 traces over merged traces.
3. For a live server, this skill can call sglang.profiler and automatically send a small probe request.
4. Prefer --profile-by-stage even on standard serving unless the user explicitly wants an all-stage mixed trace.

Two-trace workflow

1. Produce a mapping trace first with graph disabled.
2. Produce a formal trace second with graph enabled and the real serving flags kept on.
3. Run triage for the compact three-table report.
4. Read the results in this order:
   • kernel table
   • overlap-opportunity table
   • fuse-pattern table
5. Before calling something a "new" optimization idea, compare the top rows against both references/fuse-overlap-catalog.md and references/overlap-catalog.md. Check mainline rows first, then the PR-backed / in-flight sections for still-moving upstream work. Prefer reporting:
   • an existing fused or overlap path that should already apply here
   • an existing path that appears disabled, unsupported, or regressed in this trace
   • an upstream pattern that is mainline elsewhere but missing locally, or still open upstream
   • a truly new opportunity only when no catalog entry fits
6. If no exact pattern fully matches but the trace still looks semantically close to a known family, add one flat AI similarity judgment note after the tables. Use high, medium, or low only. Base that note on the full pattern shape, not on one kernel name alone. Prefer semantic cues such as producer-consumer chain, source locations, CPU op names, TP context, and model-specific structure. Do not rewrite the script table itself to include these heuristic judgments.

References

Load these only when needed:

• references/source-map.md
  • upstream SGLang profiler entrypoints and trace-writing source paths
• references/heuristics.md
  • overlap labels, dependency-risk interpretation, and limits
• references/fuse-overlap-catalog.md
  • mixed source-backed catalog of existing fuse and overlap patterns, including mainline rows plus PR-backed / in-flight rows
• references/overlap-catalog.md
  • overlap-only lookup table across LLM, VLM, diffusion, disaggregation, HiSparse, and speculative scheduling

Output Contract

Return:

• trace path or generated profile path
• model/server args when available
• kernel table
• overlap-opportunity table
• fuse-pattern table
• optional AI similarity judgment note with high / medium / low when exact matching is inconclusive
• one short conclusion about what dominates the run
• whether the overlap conclusion came from single-trace triage or mapping/formal two-trace triage