mathews-tom/surrogate-verifier

Surrogate Verifier

Generate structured test assertions and failure diagnostics for skill packages through information-isolated verification. The verifier operates without access to the skill generator's reasoning — it sees only the skill definition, a task prompt, and the output artifacts. This isolation prevents confirmation bias and is the single largest contributor to skill quality in co-evolutionary generation (+30pp per EvoSkills).

Reference Files

| File | Contents | Load When | | ----------------------------------- | ---------------------------------------------------------------- | ------------------------------ | | references/assertion-patterns.md | Assertion catalog by skill category with weight guidance | Always | | references/diagnostic-templates.md| Failure diagnostic templates with root-cause categories | When producing failure reports |

Information Isolation Protocol

This is the most critical constraint. Violating isolation degrades verification quality.

The verifier MUST NOT access:

• The generator's conversation history or reasoning chain
• Prior evolution iterations or refinement context
• The generator's internal notes or decision rationale
• Any context beyond what is explicitly listed below

The verifier receives ONLY:

1. The skill's SKILL.md content (the definition file)
2. One or more task prompts representing intended use
3. The skill's output (when diagnosing failures)
4. The assertion results from scripts/eval_assertions.py (when diagnosing)

Implementation: When invoked by the test-engineer agent, this skill MUST be loaded into a separate Agent spawn using isolation: "worktree" or at minimum a fresh session with no shared context. The invoking agent passes artifacts as explicit text, not as conversation references.

Workflow

Mode 1: Assertion Generation

Generate assertions for a skill given its definition and task prompts.

Phase 1: Skill Analysis

Read the SKILL.md definition and extract:

1. Stated capabilities — what the skill claims to do (from description + workflow sections)
2. Output format — expected structure of the skill's output (markdown, JSON, tables, etc.)
3. Error handling — documented failure modes and recovery paths
4. Prerequisites — required tools, dependencies, or context
5. Trigger boundaries — what the skill does NOT handle (negative scope)

Phase 2: Assertion Design

For each task prompt, generate 5-10 assertions covering these dimensions:

| Dimension | Assertion Types to Use | Purpose | | --------------------- | ------------------------------- | ------------------------------------------ | | Output completeness | contains, matches_regex | All claimed sections/components present | | Format compliance | output_format, contains | Output matches declared structure | | Factual signals | contains, not_contains | Key domain terms present, hallmarks absent | | Tool usage | calls_tool | Expected tools were invoked | | Negative constraints | not_contains | Forbidden patterns absent |

Weight assignment:

• Output completeness assertions: weight 1.0 (must have)
• Format compliance: weight 0.8 (structural correctness)
• Factual signals: weight 0.6 (content quality)
• Tool usage: weight 0.5 (method verification)
• Negative constraints: weight 0.3 (absence checks are weaker signals)

See references/assertion-patterns.md for category-specific assertion catalogs.

Phase 3: Output

Produce assertions in the evals/cases.yaml schema format:

assertions:
  - type: contains
    target: "## Scalability"
    weight: 1.0
  - type: output_format
    target: markdown_table
    weight: 0.8
  - type: not_contains
    target: "TODO"
    weight: 0.3
  - type: calls_tool
    target: Read
    weight: 0.5

Context cap: Do not consume more than 70% of the available context window. If the skill definition is very long, focus assertion generation on the workflow phases and output format sections. Summarize rather than quote verbatim.

Mode 2: Failure Diagnostics

When an oracle returns fail, produce a structured diagnostic explaining why.

Input

• The skill's SKILL.md (same as Mode 1)
• The task prompt that was executed
• The output that failed
• The assertion results: which passed, which failed, with details

Phase 1: Failure Classification

Categorize each failed assertion into a root-cause category:

| Category | Signal | Severity | | --------------------- | -------------------------------------------------------------- | ---------- | | Missing capability | contains assertion failed for a claimed feature | HIGH | | Format mismatch | output_format assertion failed | HIGH | | Incomplete output | Multiple contains assertions failed in the same section | MEDIUM | | Hallucinated content | not_contains assertion failed (forbidden pattern present) | HIGH | | Wrong tool usage | calls_tool assertion failed | MEDIUM | | Partial success | Some assertions in a group pass, others fail | LOW |

Phase 2: Root-Cause Analysis

For each failed assertion:

1. Identify the specific section of SKILL.md that promises the missing capability
2. Compare what the skill definition instructs vs. what the output actually contains
3. Hypothesize why the gap exists (missing workflow step, ambiguous instruction, wrong tool choice)

Phase 3: Remediation Suggestions

For each root cause, produce a concrete, actionable fix:

• Missing capability: "Add a workflow step between Phase 2 and Phase 3 that explicitly generates [X]"
• Format mismatch: "Change the output format instruction from 'produce a summary' to 'produce a markdown table with columns: [A, B, C]'"
• Hallucinated content: "Add a negative constraint in the workflow: 'Do NOT include [X] unless [condition]'"
• Wrong tool usage: "Replace 'use Bash to read the file' with 'use the Read tool for file contents'"

Phase 4: Diagnostic Output

Produce a structured diagnostic string:

DIAGNOSTIC: [skill-name] failed on [task-prompt-summary]

FAILED ASSERTIONS (N/M):
  1. [SEVERITY] type=contains target="..." — Missing capability: [explanation]
  2. [SEVERITY] type=output_format target="..." — Format mismatch: [explanation]

ROOT CAUSES:
  - [category]: [specific explanation with SKILL.md section reference]

REMEDIATION:
  1. [Concrete change to SKILL.md with exact section and wording]
  2. [Concrete change to workflow with step numbers]

See references/diagnostic-templates.md for worked examples per root-cause category.

Budget Parameters

Per EvoSkills Algorithm 1:

• Context cap: 0.7 (70% of available context window)
• Max surrogate retries: 15 per oracle round
• Max oracle rounds: 5 (enforced by the orchestrating agent, not the verifier)

The verifier does not track its own budget — the test-engineer agent manages iteration limits.

Limitations

• No execution capability: The verifier generates assertions but does not execute them. Execution is handled by scripts/run_evals.py (the oracle).
• Text-only verification: Cannot verify visual outputs, interactive behaviors, or side effects. Assertions operate on the textual output only.
• Single-turn scope: Each verification is independent. The verifier does not remember prior rounds (the orchestrating agent feeds context as needed).
• Assertion granularity: The 5 assertion types cover common patterns but not all possible verification needs. Custom assertion types require extending scripts/eval_assertions.py.

Error Handling

| Error | Resolution | | ----------------------------- | --------------------------------------------------------------- | | Skill definition too large | Summarize to workflow phases + output format sections only | | No assertions generatable | Return empty assertions list with warning; skill may be too vague | | Ambiguous output format | Default to contains assertions; avoid output_format checks | | Context cap exceeded | Truncate diagnostic detail; preserve failed assertion list |