jungho-git/benchmark

Benchmark

• Shared examples and formatting reference: references/EXAMPLE.md.
• Keep new guidance, snippets, and edits aligned with that file.

---

Scope

Use this rule when:

• comparing two or more agent configurations
• measuring token, cost, latency, cache, or tool-use impact
• choosing between models for a fixed task
• validating prompt or system-instruction changes
• checking whether a tool, context, or cache option changes performance

Benchmarking should isolate one variable and report measured evidence.

---

Fixed Task Contract

• Keep the task, input artifacts, allowed tools, and success criteria fixed.
• Change one variable per comparison.
• Run enough samples to reduce single-run noise.
• Record cache state for each run.
• Treat different tool-call counts or different task completion quality as a changed task surface.
• Do not compare costs across different task definitions.

Without a fixed task, token and latency deltas are not meaningful.

---

What To Measure

• input tokens
• output tokens
• cache creation tokens
• cache read tokens
• tool-call count and tool latency
• time to first token when streaming
• total wall time
• success/failure outcome
• answer quality or pass/fail score
• price table or pricing date used for cost calculation

Report raw usage before derived cost.

---

Protocol

• Define baseline, variant, task, inputs, and success criteria.
• Run a cold baseline and cold variant when cache matters.
• Run warm passes when prompt caching is part of the benchmark.
• Keep environment and network assumptions stable.
• Capture usage objects, timestamps, and command or harness version when available.
• Calculate per-run and aggregate deltas.
• Mark invalid runs when the model used different tools, failed the task, or changed output requirements.

Prefer a small reliable benchmark over broad anecdotal comparison.

---

Cost Rules

• Include token counts and pricing inputs.
• Separate cold cost, warm steady-state cost, and amortized break-even when cache is involved.
• Do not conflate cache creation with steady-state read cost.
• Report TTFT and total time separately for streaming experiences.
• State when pricing was unavailable or estimated.

Cost claims must be reproducible from reported numbers.

---

Output Shape

• benchmark name
• fixed task and input
• compared variable
• run table
• aggregate token, cost, and latency deltas
• validity notes
• decision and trade-off
• residual uncertainty

Keep conclusions tied to measured data.

---

Boundaries

• Do not use a single run as a final result unless the user explicitly accepts a smoke check.
• Do not report only cost without usage.
• Do not hide failed or invalid runs.
• Do not change model, prompt, tools, and context at the same time.
• Do not claim quality parity without a stated quality check.