raddue/temper-eval-calibrate

<!-- CANONICAL: shared/dispatch-convention.md -->

Temper Eval Harness — Calibrate Wrapper

Invocation:

/temper-eval-calibrate <run-id-prefix> [--k N] [--source SOURCE] [--fixture <id>]
                                       [--write-baseline] [--compare-baseline]
                                       [--trials-override N] [--max-parallel N]
                                       [--timeout N]

Pre-conditions:

• A Claude Code session is active
• <run-id-prefix> matches ^[A-Za-z0-9_][A-Za-z0-9_-]{0,28}$ (I-9; aligned with _runid.py's _PREFIX_RE; reserves 3 chars for -<i> suffix)

Post-conditions:

• For each i in 1..k, skills/temper/evals/.calibrate-state/<prefix>-<i>-complete exists AND skills/temper/evals/.calibrate-state/last_run-<prefix>-<i>.json exists with run_id == <prefix>-<i>
• A per-iteration summary line is emitted to stdout

Procedure

Step 0: Ordering precondition (2P-FE-6 R3 + F-R4-1, S-1 R5)

Task 13 (per-iter wiring) lands BEFORE this skill is invoked — verified by sequential task numbering. Per F-R4-1, Task 13 lands BOTH the --per-iter argparse declaration AND the main() wiring in a SINGLE atomic commit (Task 4 intentionally omits the argparse declaration to eliminate any silent-drop window). Because Task 13 is numbered ahead of Task 14, operators following the plan sequentially will have --per-iter wired before this skill's SKILL.md exists to be invoked. Defense-in-depth: if Task 13 is somehow skipped, score --per-iter will raise an argparse "unrecognized arguments" error (no silent drop) — the calibrate skill's Step 3e shell-out will fail loud rather than clobbering shared state.

Step 1: Validate prefix and k (I-9, M-2)

• <run-id-prefix> must match ^[A-Za-z0-9_][A-Za-z0-9_-]{0,28}$ (M-FE-3 R3); refuse with explicit error if not.
• --k (default 3) must be in [1, 99] inclusive; refuse if out-of-range. <!-- 2P-R4-2: k cap 99 ties to _PREFIX_RE's 3-char suffix reservation; k>=100 would overflow into a 4-char suffix and silently push the resulting run-id past the 32-char _RUN_ID_RE ceiling. -->

Step 2: Resolve calibrate-state directory

STATE_DIR = skills/temper/evals/.calibrate-state/. If it does not exist, create it (mkdir -p).

Step 3: For each iteration i in 1..k

3a. Compute RUN_ID

RUN_ID = "<prefix>-<i>".

3b. Resume idempotency check (SP-β, SP-4, AC-12)

If BOTH of the following are true, skip iteration entirely (no stage, no collect, no score, no billing):

• skills/temper/evals/.calibrate-state/<prefix>-<i>-complete exists
• skills/temper/evals/.calibrate-state/last_run-<prefix>-<i>.json exists AND its run_id field equals <prefix>-<i>

Print: [skip] iteration <i>: already complete.

3c. Stage

Shell out via Bash tool: python -m skills.temper.evals.run_evals stage "$RUN_ID" [--source ...] [--fixture ...] [--trials-override ...] [--timeout ...]

Pass through whatever flags the user supplied.

3d. Collect-dispatch behavior (inlined; see /temper-eval-collect spec)

Execute the full procedure documented in skills/temper-eval-collect/SKILL.md Steps 1–8 against RUN_ID.

Version pin (M1 R8): this inline-execution reference is pinned to the Steps-1–8 shape at calibrate-skill author time. If skills/temper-eval-collect/SKILL.md is later edited (e.g. a Step 4.5 is added, or steps are renumbered), this calibrate skill MUST be re-validated against the new step set — there is NO automatic inheritance. Treat collect SKILL.md edits as breaking changes for the calibrate skill until re-verified.

This includes:

• Run-id validation (I-9)
• Dispatch-dir stat (SP-3 step 1)
• Atomicity probe (SP-3 step 2, F-1)
• Stale .tmp cleanup
• Stage-manifest read with optional --timeout override
• Idempotency-aware wave dispatch (max_parallel default 6)
• Atomic per-seq result-file writes with DISPATCH_STATUS sentinels
• manifest.jsonl per-dispatch appends
• I-10 10 MB result-size ceiling
• .collect-status write with fsync (I-12)

The inlined behavior satisfies I-7, I-8, I-10, I-12 identically to standalone /temper-eval-collect.

3e. Score (with per-iteration output)

Shell out: python -m skills.temper.evals.run_evals score "$RUN_ID" --per-iter [--write-baseline] [--compare-baseline]

The --per-iter flag is REQUIRED here (F1): it tells score to write to skills/temper/evals/.calibrate-state/last_run-<prefix>-<i>.json instead of the shared last_run.json. Without --per-iter, iterations would clobber each other's output AND collide with operator-owned last_run.json artifacts. There is NO run-id-shape heuristic; the routing is explicit.

If score returns 2 (fatal), abort all remaining iterations and surface the error.

3f. Write completion sentinel

On successful score, write skills/temper/evals/.calibrate-state/<prefix>-<i>-complete with content complete-<ISO-8601>.

3g. Print per-iteration summary

[iter <i>/<k>] RUN_ID=<RUN_ID> score=<rc>

Step 4: Final consolidation (M-3)

After ALL k iterations succeed (every iteration's .calibrate-state/<prefix>-<i>-complete sentinel is present), copy skills/temper/evals/.calibrate-state/last_run-<prefix>-<k>.json to skills/temper/evals/last_run.json (the shared output the --compare-baseline workflow expects). This is unconditional on success — not optional.

Pre-copy verification (mirrors the sentinel-pair discipline in Step 3b):

1. Verify last_run-<prefix>-<k>.json exists. If absent, refuse to clobber last_run.json and exit with the fatal error "iteration k completion sentinel present but per-iter file missing at <path>; refusing to consolidate."
2. Parse the file as JSON. If parsing fails, refuse with "per-iter file at <path> is malformed JSON; refusing to consolidate."
3. Confirm the run_id field equals <prefix>-<k>. If it does not, refuse with "per-iter file at <path> carries run_id=<actual>, expected <prefix>-<k>; refusing to consolidate."
4. Only after all three checks pass, perform the copy.

If ANY iteration failed (sentinel missing, score returned 2, or stage refused), do NOTHING in this step: leave last_run.json untouched so the operator can inspect the per-iter files manually without a clobbered shared state.

Rationale: the original "optionally copy" wording produced two equally-valid downstream states (shared file = last iter vs. shared file = stale) with no operator signal to disambiguate. The all-or-nothing rule eliminates the ambiguity. Revisit only if best-or-median canonicalization is genuinely needed (defer to a follow-up ticket).

Failure Modes

• Stage fails: abort remaining iterations; surface the stage error.
• Collect crashes mid-wave: next invocation resumes from missing seqs in that iteration's dispatch dir; once that iteration completes, the calibrate sentinel is written and subsequent iterations proceed.
• Score fails (rc=1): continue to next iteration (a single iteration FAIL is data, not a blocker for k=3).
• Score returns rc=2 (fatal): abort all remaining iterations.

Invariants

This skill enforces: M-2 (k cap 1..99), I-9 (prefix regex), AC-12 (resume idempotency).