richfrem/os-eval-runner

Skill Improvement Evaluator

Stateless evaluation engine that scores and gates skill improvement iterations using headless Python evaluation scripts.

---

Ownership Boundary (Critical)

What os-eval-runner owns (permanent, version-controlled with this skill)

• Scoring scripts: ./scripts/evaluate.py, ./scripts/eval_runner.py
• Scaffold script: ./scripts/init_autoresearch.py
• Templates: ./assets/templates/autoresearch/ (program, evals, results, proposer prompt)

What lives with the target (deployed per experiment)

All experiment state deploys alongside the target (e.g. <experiment-dir>/references/program.md, <experiment-dir>/evals/evals.json, <experiment-dir>/evals/results.tsv). You MUST read the spec from <experiment-dir>/references/program.md and NOT fall back to engine-local config templates.

---

Phase 0: Intake Interview

Run this interview before starting any loop or evaluation. If enough information is provided in the initial prompt, skip the redundant questions.

1. Q1 — What target skill are you evaluating? (Provide path to skill folder)
2. Q2 — Where should the experiment files live? (Defaults to target skill directory)
3. Q2b — What metric are you optimizing? (quality_score, f1, precision, recall, or heuristic)
4. Q3 — What mode? (Loop mode for autonomous improvement vs QA mode for single diff validation)
5. Q4 — (Loop mode) How many iterations? (Default: NEVER STOP)
6. Q5 — Does evals.json exist? (If missing, scaffold from template)
7. Q6 — Does program.md exist? (If missing, scaffold from template)
8. Q7 — Does a baseline score exist? (If missing, run evaluate.py with --baseline)

---

Two Modes: Summarized

• Mode 1: Autoresearch Loop: Autonomous iterative improvement. The agent identifies failure types, requests mutations via external proposer CLI (Copilot/Gemini), and runs the eval gate iteratively until the budget or target score is met.
• Mode 2: Single-shot QA: Simple gate validation. Evaluates one specific proposed diff against the baseline and decides KEEP (exit 0) or DISCARD (revert, exit 1).

---

Stage Pointers & Reference Protocols

• Setup: Start a New Experiment — 4-step setup and re-baselining procedure.
• Mode 1: Autoresearch Loop Protocol — Proposer cycles, prompt mutations, and evaluation loop.
• Mode 2: Single-shot QA Protocol — Context acquisition, reverts, and reporting.
• Phase 2b: Overfitting Gate — Holdout set overfitting checks and forced discard logic.
• Phase 5: Self-Assessment Survey — Mandatory evaluator survey guidelines.

---

Smoke Test & Gotchas

Smoke Test

1. Scaffold an experiment: python3 ./scripts/init_autoresearch.py --experiment-dir temp/test-exp --mutation-target SKILL.md.
2. Establish baseline: python3 ./scripts/evaluate.py --skill temp/test-exp --baseline --desc "smoke test".
3. Validate exit code: Assert results.tsv is created, and running evaluate.py returns 0.

Gotchas

• Subjective Simulation: Avoid "mentally simulating" routing accuracy. Subjective audits are strictly banned; run Python evaluation scripts.
• Missing Holdout: Starting loops without holdout prompts. This bypasses the overfitting gate, rendering the results invalid.
• Keywords Footgun: Adding too many triggers to frontmatter. This dilutes semantic discrimination and degrades overall router precision.