rbergman/evaluator

Evaluator Protocol

Separate the agent doing work from the agent judging it. This is more tractable than making one agent self-critical.

When to Invoke

The orchestrator calls the evaluator in these situations:

1. Post-subagent, pre-merge — after implementation passes mechanical gates AND either:
   • Browser-qa is available (CDT MCP connected + app running) — runtime evaluation
   • Acceptance criteria require runtime testing ("user can...", "page shows...", "form validates...")
2. On-demand — evaluate <bead-id> to test an existing feature against its criteria
3. Post-merge — /dm-work:post-merge runs evaluator against closed beads

Skip evaluator when:

• XS/S tasks without acceptance criteria
• Intent review returned COVERAGE: full, DRIFT: none, GAPS: none AND no browser-qa available
• Bead has no acceptance criteria (report to orchestrator, don't run empty evaluation)
• Task has no bead (ad-hoc work)

The intent review and evaluator have complementary scope:

• Intent review checks CODE COVERAGE — does the diff contain the right changes?
• Evaluator checks BEHAVIORAL CORRECTNESS — does the running app satisfy each criterion?

If no runtime testing is possible, the evaluator's value over intent review is minimal. Skip it.

Evaluator Agent Template

Task(subagent_type="general-purpose", model="opus", description="Evaluate against acceptance criteria", prompt="
# Use model="haiku" for code-only evaluation with simple criteria (no browser-qa)
ROLE: Evaluator. You judge work against acceptance criteria. You do NOT implement or fix.

BEAD: <id>
ACCEPTANCE CRITERIA (from bead --design field):
<numbered list of criteria>

CODE DIFF:
<git diff output or summary of changes>

EVALUATION PROCESS:

1. Classify each criterion:
   - RUNTIME: requires browser interaction to verify ("user can...", "page shows...", "form validates...")
   - CODE: verifiable from code inspection ("function exists", "type is correct", "test passes")

2. If browser-qa available (CDT MCP connected, app running at <url>):
   - Activate dm-work:browser-qa
   - For each RUNTIME criterion: navigate, interact, assert
   - For each CODE criterion: inspect the diff

3. If browser-qa NOT available:
   - For each CODE criterion: inspect the diff
   - For each RUNTIME criterion: mark UNTESTABLE with reason
   - If ALL criteria are UNTESTABLE: return early with overall: SKIP

4. Grade each criterion: PASS / FAIL / UNTESTABLE
   - PASS: criterion is satisfied (code or runtime evidence)
   - FAIL: criterion is not satisfied (describe what's wrong)
   - UNTESTABLE: cannot verify without runtime / missing prerequisite

SKILLS: dm-work:browser-qa (if CDT MCP available)

OUTPUT FORMAT (JSON to stdout):
{
  \"bead_id\": \"<id>\",
  \"criteria_results\": [
    {
      \"criterion\": 1,
      \"text\": \"User can navigate to /settings\",
      \"type\": \"RUNTIME\",
      \"result\": \"PASS\",
      \"detail\": \"Navigated to /settings, page loads with profile form visible\"
    },
    {
      \"criterion\": 2,
      \"text\": \"Email validates client-side\",
      \"type\": \"RUNTIME\",
      \"result\": \"FAIL\",
      \"detail\": \"Entered invalid email 'notanemail', no validation error shown\"
    }
  ],
  \"overall\": \"FAIL\",
  \"pass_count\": 1,
  \"fail_count\": 1,
  \"untestable_count\": 0,
  \"summary\": \"1/2 criteria pass. Email validation missing on client side.\"
}

RULES:
- Judge ONLY against the listed acceptance criteria. Do not invent requirements.
- PASS means the criterion is satisfied, not that the code is perfect.
- Report what you observed, not what you assumed.
- If a criterion is ambiguous, grade it and note the ambiguity in detail.
- Do NOT modify code, commit, or close beads.
")

Handling Evaluator Results

The orchestrator processes evaluator output:

overall: PASS → proceed to merge overall: SKIP → all criteria untestable, proceed (evaluator adds no value here) overall: FAIL →

1. Check fail count vs total:
   • 1-2 failures: send FAIL details back to original subagent for targeted fix

   • 50% failures: likely a spec problem — escalate to user, don't iterate

2. Check if failures are criteria bugs (criterion is impossible/ambiguous):
   • If so, update the bead criteria, don't blame the implementation
3. Create beads for persistent failures:

   bd create --title="Eval: <failed criterion>" --type=bug --priority=2
   bd dep add <new-bead> discovered-from:<parent-bead>
   

Circuit breaker: If evaluator fails twice on the same criterion after rework, escalate to user. Don't loop.

Cost and Timing

• Evaluator adds ~1-2 minutes per invocation (code-only) or ~2-4 minutes (with browser-qa)
• Skip aggressively when not needed (see skip conditions above)
• Use haiku model for code-only evaluation if criteria are simple
• Use opus for browser-qa evaluation (needs to drive CDT tools effectively)

Platform-Specific Verification

Not all projects use browser-qa. The evaluator should adapt:

| Project type | Verification method | Evaluator behavior | |-------------|--------------------|--------------------| | Standard web app | browser-qa (CDT MCP) | Full runtime evaluation | | WebGL / Canvas game | Manual screenshots + human verification | Mark runtime criteria UNTESTABLE; take screenshots if CDT available for visual reference, but can't assert on canvas content | | Native iOS/Android | Maestro or platform-specific tools | Mark runtime criteria UNTESTABLE unless project has automated UI test tooling wired | | CLI tool | Bash execution + output assertion | Code-only evaluation; test commands via bash, not browser | | API / backend | curl / httpie + response assertion | Code-only for endpoints; evaluate_script or direct API calls |

When runtime verification isn't possible, the evaluator should:

1. Grade all code-verifiable criteria normally
2. Mark platform-specific criteria as UNTESTABLE with the reason and recommended verification method
3. Include a note: "Manual verification recommended for: [list criteria]"

Integration Points

| Component | How evaluator connects | |-----------|----------------------| | Orchestrator | Calls evaluator as Step 1.5 in post-subagent verification | | Browser-qa | Evaluator activates browser-qa skill for standard web apps | | Beads | Reads acceptance criteria from bead; files new beads for failures | | Sprint contracts | Acceptance criteria in bead ARE the sprint contract | | Post-merge review | Post-merge command uses evaluator for closed beads | | Intent review | Complementary: intent checks code coverage, evaluator checks behavior |