atopile/.claude/skills/sexp_bench

SEXP Benchmark Strategy

Goal

Measure and improve S-expression pipeline performance with a focus on:

• Throughput per stage
• Peak memory per stage
• End-to-end behavior on realistic KiCad PCB inputs

Pipeline Stages

Benchmark these layers separately:

• tokenizer
• ast
• parser (typed decode)
• encode (typed encode to raw SEXP)
• pretty (formatting)

Dataset Dimensions

Use a matrix over:

• depth: shallow vs deep nesting
• size: small, medium, large

Recommended size buckets:

• small: < 64 KiB
• medium: 64 KiB .. < 1 MiB
• large: 1 MiB .. < 10 MiB

Measurement Model

For each stage and sample:

• mean_ms and percentile latency (median, p80)
• mem_before, mem_after, mem_delta
• stage_peak_increment (stage-local high watermark increase)
• cumulative_pipeline_peak and cumulative_peak_over_start

Key interpretation:

• Negative/near-zero mem_delta can still coexist with high stage_peak_increment.
• For allocator-heavy code, peak metrics are more informative than final deltas.

Methodology

1. Warm caches with warmup runs.
2. Run at least one measured sample per cell (more for stable comparisons).
3. Keep run settings fixed when comparing commits:
   • same machine
   • same optimize mode
   • same dataset generator/inputs
4. Compare both synthetic matrix and a large real-world board.

E2E Roundtrip Benchmark (panel.kicad_pcb)

Use this when validating real end-to-end impact (load -> dump -> reload) on a large board.

Setup

1. Build pyzig_sexp.so in each checkout you want to compare:

source .venv/bin/activate
cd src/faebryk/core/zig
python -m ziglang build python-ext -Doptimize=ReleaseFast -Dpython-include=/usr/include/python3.14 -Dpython-lib=python3.14

2. For baseline vs current comparison, create a detached worktree for baseline:

git worktree add /tmp/atopile_pre_tokenizer_fix <baseline_commit>

3. Run the benchmark sequentially (not in parallel) to avoid cross-run CPU contention.

Runner Pattern

Use Python with direct importlib loading of pyzig_sexp.so from each checkout. This avoids accidental rebuilds and keeps the comparison tied to the compiled artifact in that checkout.

from pathlib import Path
from time import perf_counter
import gc, importlib.util, sys

spec = importlib.util.spec_from_file_location(
    "pyzig_sexp",
    "src/faebryk/core/zig/zig-out/lib/pyzig_sexp.so",
)
mod = importlib.util.module_from_spec(spec)
sys.modules["pyzig_sexp"] = mod
spec.loader.exec_module(mod)

text = Path("panel.kicad_pcb").read_text(encoding="utf-8")

# warmup
obj = mod.pcb.loads(text)
out = mod.pcb.dumps(obj)
obj2 = mod.pcb.loads(out)
del obj, out, obj2
gc.collect()

runs = []
for _ in range(5):
    gc.collect()
    t0 = perf_counter()
    obj = mod.pcb.loads(text)
    t1 = perf_counter()
    out = mod.pcb.dumps(obj)
    t2 = perf_counter()
    obj2 = mod.pcb.loads(out)
    t3 = perf_counter()
    runs.append((t1 - t0, t2 - t1, t3 - t2, t3 - t0))
    del obj, out, obj2

print("AVG", " ".join(f"{sum(r[i] for r in runs)/len(runs):.3f}" for i in range(4)))

Report Format

Report at minimum:

• load_avg_s
• dump_avg_s
• reload_avg_s
• total_roundtrip_avg_s
• relative delta vs baseline (%)

Optimization Approach

Prioritize stage-local wins first, then validate global effects:

1. Eliminate avoidable intermediate allocations.
2. Use streaming write paths for encode-heavy workloads.
3. Keep fast parse paths for success cases; fallback to richer diagnostics on error.
4. Re-check correctness via roundtrip and file-format tests after each change.

Safety Checks

After each optimization pass:

• Build release artifacts.
• Run KiCad file-format tests.
• Run representative load-transform-dump flows.
• Re-run matrix + panel benchmark snapshots.

Reporting

Summarize gains in two views:

• Matrix view: (depth, size) x stage
• Large-board view: stage timing + peak memory

Always report:

• absolute values
• speedup ratios
• peak-memory deltas vs baseline