awfixers-stuff/flamegraphs

Flamegraphs

Purpose

Guide agents through the pipeline from profiler data to SVG flamegraph, and teach interpretation of flamegraphs to drive concrete optimisation decisions.

Triggers

• "How do I generate a flamegraph from perf data?"
• "How do I read a flamegraph?"
• "The flamegraph shows a wide frame — what does that mean?"
• "How do I generate a flamegraph from Callgrind?"
• "I want to compare two flamegraphs (before/after)"

Workflow

1. Install FlameGraph tools

git clone https://github.com/brendangregg/FlameGraph
# No install needed; scripts are in the repo
export PATH=$PATH:/path/to/FlameGraph

2. perf → flamegraph (most common path)

# Step 1: record
perf record -F 999 -g -o perf.data ./prog

# Step 2: generate script output
perf script -i perf.data > out.perf

# Step 3: collapse stacks
stackcollapse-perf.pl out.perf > out.folded

# Step 4: generate SVG
flamegraph.pl out.folded > flamegraph.svg

# Step 5: view
xdg-open flamegraph.svg     # Linux
open flamegraph.svg          # macOS

One-liner:

perf record -F 999 -g ./prog && perf script | stackcollapse-perf.pl | flamegraph.pl > fg.svg

3. Differential flamegraph (before/after)

# Collect two profiles
perf record -g -o before.data ./prog_old
perf record -g -o after.data ./prog_new

# Collapse
perf script -i before.data | stackcollapse-perf.pl > before.folded
perf script -i after.data  | stackcollapse-perf.pl > after.folded

# Diff (red = regressed, blue = improved)
difffolded.pl before.folded after.folded | flamegraph.pl > diff.svg

4. Callgrind → flamegraph

valgrind --tool=callgrind --callgrind-out-file=cg.out ./prog
stackcollapse-callgrind.pl cg.out | flamegraph.pl > fg.svg

5. Other profiler inputs

# Go pprof
go tool pprof -raw -output=prof.txt prog
stackcollapse-go.pl prof.txt | flamegraph.pl > fg.svg

# DTrace
dtrace -x ustackframes=100 -n 'profile-99 /execname=="prog"/ { @[ustack()] = count(); }' \
  -o out.stacks sleep 10
stackcollapse.pl out.stacks | flamegraph.pl > fg.svg

# Java (async-profiler)
async-profiler -d 30 -f out.collapsed PID
flamegraph.pl out.collapsed > fg.svg

6. Reading flamegraphs

A flamegraph is a call-stack visualisation:

• X axis: time on CPU (not time sequence) — wider = more time
• Y axis: call stack depth — taller = deeper call chain
• Color: random (no significance) — unless using differential mode

What to look for:

| Pattern | Meaning | Action | |---------|---------|--------| | Wide frame near bottom | Function itself is hot | Optimise that function | | Wide frame with tall narrow towers | Calling many different callees | Hot dispatch; reduce call overhead | | Very tall stack with wide base | Deep recursion | Check recursion depth; consider iterative approach | | Plateau at the top | Leaf function with no callees | This leaf is the actual hotspot | | Many narrow identical stacks | Many threads doing the same work | Consider parallelism or batching |

Identifying the actionable hotspot:

1. Find the widest top frame (a frame with no or narrow children above it)
2. That is where CPU time is actually spent
3. Trace down to understand what called it and why

Differential flamegraph:

• Red frames: more time in new profile (regression)
• Blue frames: less time in new profile (improvement)
• Frames only in one profile appear solid colored

7. flamegraph.pl options

flamegraph.pl --title "My App" \
              --subtitle "Release build, workload X" \
              --width 1600 \
              --height 16 \
              --minwidth 0.5 \
              --colors java \
              out.folded > fg.svg

| Option | Effect | |--------|--------| | --title | SVG title | | --width | Width in pixels | | --height | Frame height in pixels | | --minwidth | Omit frames < N% (reduces clutter) | | --colors | Palette: hot (default), mem, io, java, js, perl, red, green, blue | | --inverted | Icicle chart (roots at top) | | --reverse | Reverse stacks | | --cp | Consistent palette (same frame = same color across SVGs) |

References

For tool installation, stackcollapse scripts, and palette options, see references/tools.md.

Related skills

• Use skills/profilers/linux-perf to collect perf data
• Use skills/profilers/valgrind to collect Callgrind data
• Use skills/compilers/clang for LLVM PGO from sampling profiles