chelch5/skill-eval-runner

Skill Eval Runner

Executes a skill's eval suite and produces a structured quality report.

Procedure

1. Locate test suite

Check for eval files in the skill directory:

• evals/triggers.yaml — trigger accuracy tests
• evals/outputs.yaml — behavior correctness tests
• evals/baselines.yaml — baseline comparison tests

Note which exist and which are missing.

2. Run trigger tests

For each case in triggers.yaml:

• Positive cases: Does the skill fire? (expected: yes)
• Negative cases: Does the skill NOT fire? (expected: no)

Record: prompt, expected, actual, Pass/Fail.

3. Run output tests

For each case in outputs.yaml:

• Run the skill with the given input
• Check against expected_sections, required_patterns, forbidden_patterns

Record: test name, checks passed/total, Pass/Fail.

4. Run baseline comparison

For each case in baselines.yaml:

• Run with skill active vs without skill
• Does the skill add value? (Yes if 2+ elements baseline would lack)

5. Aggregate results

Calculate:

• Trigger precision: TP / (TP + FP)
• Trigger recall: TP / (TP + FN)
• Output pass rate: passed / total checks
• Baseline win: Yes/No

6. Issue verdict

| Verdict | Criteria | |---------|----------| | Pass | All rates ≥80% AND baseline win | | Pass with issues | Any rate 60-79% | | Fail | Any rate <60% OR baseline lose |

Output contract

## Eval Report: [skill-name]
Date: [YYYY-MM-DD]

### Trigger Tests
| Prompt | Type | Expected | Actual | Result |
|--------|------|----------|--------|--------|
Precision: X%  Recall: Y%

### Output Tests
| Test | Checks Passed | Result |
|------|--------------|--------|
Pass rate: X%

### Baseline Comparison
Skill adds value: [Yes/No]

### Verdict: [Pass | Pass with issues | Fail]
Issues: [list or "None"]

Failure handling

• Silent skill failure (no output): Record Fail, halt remaining tests for that case
• Missing test files: Report which are missing, run available ones, note incomplete coverage
• Flaky results: Run 3x, use majority result, note flakiness in report
• All tests missing: Cannot evaluate — report "No eval suite found" and recommend building one

References

• Anthropic eval overview: https://docs.anthropic.com/en/docs/test-and-evaluate/eval-overview
• OpenAI evals: https://developers.openai.com/docs/evals/getting-started