dwmkerr/anthropic-evaluations

Anthropic Evaluations

Build rigorous evaluations for AI agents using Anthropic's proven patterns.

Quick Reference

You MUST read the reference files for detailed guidance:

• Grader Types - Code-based, model-based, human graders
• Agent Type Patterns - Coding, conversational, research, computer use
• Roadmap - Steps 0-8 for building evals from scratch
• Frameworks - Harbor, Promptfoo, Braintrust, etc.

YAML Templates:

• coding-agent-eval.yaml - Coding agent template
• conversational-agent-eval.yaml - Support agent template

Annotated Examples:

• Example: Coding Agent - Auth bypass fix walkthrough
• Example: Conversational - Refund handling walkthrough

Core Definitions

| Term | Definition | |------|------------| | Task | Single test with defined inputs and success criteria | | Trial | One attempt at a task (run multiple for consistency) | | Grader | Logic that scores agent performance; tasks can have multiple | | Transcript | Complete record of a trial (outputs, tool calls, reasoning) | | Outcome | Final state in environment (not just what agent said) | | Evaluation harness | Infrastructure that runs evals end-to-end | | Agent harness | System enabling model to act as agent (scaffold) | | Evaluation suite | Collection of tasks measuring specific capabilities |

Grader Types (Quick Reference)

| Type | Methods | Best For | |------|---------|----------| | Code-based | String match, unit tests, static analysis, state checks | Fast, cheap, objective verification | | Model-based | Rubric scoring, assertions, pairwise comparison | Nuanced, open-ended tasks | | Human | SME review, A/B testing, spot-check sampling | Gold standard calibration |

See Grader Types for detailed comparison.

Capability vs Regression Evals

| Type | Question | Target Pass Rate | |------|----------|------------------| | Capability | "What can this agent do well?" | Start low, hill-climb | | Regression | "Does it still handle what it used to?" | Near 100% |

Capability evals with high pass rates "graduate" to regression suites.

Non-Determinism Metrics

| Metric | Measures | Use When | |--------|----------|----------| | pass@k | At least 1 success in k attempts | One success matters (coding) | | pass^k | All k attempts succeed | Consistency essential (customer-facing) |

Example: 75% per-trial success rate

• pass@3 ≈ 98% (likely to get at least one)
• pass^3 ≈ 42% (0.75³ all succeed)

Tracked Metrics

tracked_metrics:
  - type: transcript
    metrics: [n_turns, n_toolcalls, n_total_tokens]
  - type: latency
    metrics: [time_to_first_token, output_tokens_per_sec, time_to_last_token]

Attribution

Based on Demystifying evals for AI agents by Anthropic (January 2026).