autohandai/agent-evaluation

Agent Evaluation (AI Agent Evals)

Based on Anthropic's "Demystifying evals for AI agents"

When to use this skill

• Designing evaluation systems for AI agents
• Building benchmarks for coding, conversational, or research agents
• Creating graders (code-based, model-based, human)
• Implementing production monitoring for AI systems
• Setting up CI/CD pipelines with automated evals
• Debugging agent performance issues
• Measuring agent improvement over time

Core Concepts

Eval Evolution: Single-turn → Multi-turn → Agentic

| Type | Turns | State | Grading | Complexity | |------|-------|-------|---------|------------| | Single-turn | 1 | None | Simple | Low | | Multi-turn | N | Conversation | Per-turn | Medium | | Agentic | N | World + History | Outcome | High |

7 Key Terms

| Term | Definition | |------|------------| | Task | Single test case (prompt + expected outcome) | | Trial | One agent run on a task | | Grader | Scoring function (code/model/human) | | Transcript | Full record of agent actions | | Outcome | Final state for grading | | Harness | Infrastructure running evals | | Suite | Collection of related tasks |

Instructions

Step 1: Understand Grader Types

Code-based Graders (Recommended for Coding Agents)

• Pros: Fast, objective, reproducible
• Cons: Requires clear success criteria
• Best for: Coding agents, structured outputs

# Example: Code-based grader
def grade_task(outcome: dict) -> float:
    """Grade coding task by test passage."""
    tests_passed = outcome.get("tests_passed", 0)
    total_tests = outcome.get("total_tests", 1)
    return tests_passed / total_tests

# SWE-bench style grader
def grade_swe_bench(repo_path: str, test_spec: dict) -> bool:
    """Run tests and check if patch resolves issue."""
    result = subprocess.run(
        ["pytest", test_spec["test_file"]],
        cwd=repo_path,
        capture_output=True
    )
    return result.returncode == 0

Model-based Graders (LLM-as-Judge)

• Pros: Flexible, handles nuance
• Cons: Requires calibration, can be inconsistent
• Best for: Conversational agents, open-ended tasks

# Example: LLM Rubric for Customer Support Agent
rubric:
  dimensions:
    - name: empathy
      weight: 0.3
      scale: 1-5
      criteria: |
        5: Acknowledges emotions, uses warm language
        3: Polite but impersonal
        1: Cold or dismissive

    - name: resolution
      weight: 0.5
      scale: 1-5
      criteria: |
        5: Fully resolves issue
        3: Partial resolution
        1: No resolution

    - name: efficiency
      weight: 0.2
      scale: 1-5
      criteria: |
        5: Resolved in minimal turns
        3: Reasonable turns
        1: Excessive back-and-forth

Human Graders

• Pros: Highest accuracy, catches edge cases
• Cons: Expensive, slow, not scalable
• Best for: Final validation, ambiguous cases

Step 2: Choose Strategy by Agent Type

2.1 Coding Agents

Benchmarks:

• SWE-bench Verified: Real GitHub issues (40% → 80%+ achievable)
• Terminal-Bench: Complex terminal tasks
• Custom test suites with your codebase

Grading Strategy:

def grade_coding_agent(task: dict, outcome: dict) -> dict:
    return {
        "tests_passed": run_test_suite(outcome["code"]),
        "lint_score": run_linter(outcome["code"]),
        "builds": check_build(outcome["code"]),
        "matches_spec": compare_to_reference(task["spec"], outcome["code"])
    }

Key Metrics:

• Test passage rate
• Build success
• Lint/style compliance
• Diff size (smaller is better)

2.2 Conversational Agents

Benchmarks:

• τ2-Bench: Multi-domain conversation
• Custom domain-specific suites

Grading Strategy (Multi-dimensional):

success_criteria:
  - empathy_score: >= 4.0
  - resolution_rate: >= 0.9
  - avg_turns: <= 5
  - escalation_rate: <= 0.1

Key Metrics:

• Task resolution rate
• Customer satisfaction proxy
• Turn efficiency
• Escalation rate

2.3 Research Agents

Grading Dimensions:

1. Grounding: Claims backed by sources
2. Coverage: All aspects addressed
3. Source Quality: Authoritative sources used

def grade_research_agent(task: dict, outcome: dict) -> dict:
    return {
        "grounding": check_citations(outcome["report"]),
        "coverage": check_topic_coverage(task["topics"], outcome["report"]),
        "source_quality": score_sources(outcome["sources"]),
        "factual_accuracy": verify_claims(outcome["claims"])
    }

2.4 Computer Use Agents

Benchmarks:

• WebArena: Web navigation tasks
• OSWorld: Desktop environment tasks

Grading Strategy:

def grade_computer_use(task: dict, outcome: dict) -> dict:
    return {
        "ui_state": verify_ui_state(outcome["screenshot"]),
        "db_state": verify_database(task["expected_db_state"]),
        "file_state": verify_files(task["expected_files"]),
        "success": all_conditions_met(task, outcome)
    }

Step 3: Follow the 8-Step Roadmap

Step 0: Start Early (20-50 Tasks)

# Create initial eval suite structure
mkdir -p evals/{tasks,results,graders}

# Start with representative tasks
# - Common use cases (60%)
# - Edge cases (20%)
# - Failure modes (20%)

Step 1: Convert Manual Tests

# Transform existing QA tests into eval tasks
def convert_qa_to_eval(qa_case: dict) -> dict:
    return {
        "id": qa_case["id"],
        "prompt": qa_case["input"],
        "expected_outcome": qa_case["expected"],
        "grader": "code" if qa_case["has_tests"] else "model",
        "tags": qa_case.get("tags", [])
    }

Step 2: Ensure Clarity + Reference Solutions

# Good task definition
task:
  id: "api-design-001"
  prompt: |
    Design a REST API for user management with:
    - CRUD operations
    - Authentication via JWT
    - Rate limiting
  reference_solution: "./solutions/api-design-001/"
  success_criteria:
    - "All endpoints documented"
    - "Auth middleware present"
    - "Rate limit config exists"

Step 3: Balance Positive/Negative Cases

# Ensure eval suite balance
suite_composition = {
    "positive_cases": 0.5,    # Should succeed
    "negative_cases": 0.3,    # Should fail gracefully
    "edge_cases": 0.2         # Boundary conditions
}

Step 4: Isolate Environments

# Docker-based isolation for coding evals
eval_environment:
  type: docker
  image: "eval-sandbox:latest"
  timeout: 300s
  resources:
    memory: "4g"
    cpu: "2"
  network: isolated
  cleanup: always

Step 5: Focus on Outcomes, Not Paths

# GOOD: Outcome-focused grader
def grade_outcome(expected: dict, actual: dict) -> float:
    return compare_final_states(expected, actual)

# BAD: Path-focused grader (too brittle)
def grade_path(expected_steps: list, actual_steps: list) -> float:
    return step_by_step_match(expected_steps, actual_steps)

Step 6: Always Read Transcripts

# Transcript analysis for debugging
def analyze_transcript(transcript: list) -> dict:
    return {
        "total_steps": len(transcript),
        "tool_usage": count_tool_calls(transcript),
        "errors": extract_errors(transcript),
        "decision_points": find_decision_points(transcript),
        "recovery_attempts": find_recovery_patterns(transcript)
    }

Step 7: Monitor Eval Saturation

# Detect when evals are no longer useful
def check_saturation(results: list, window: int = 10) -> dict:
    recent = results[-window:]
    return {
        "pass_rate": sum(r["passed"] for r in recent) / len(recent),
        "variance": calculate_variance(recent),
        "is_saturated": all(r["passed"] for r in recent),
        "recommendation": "Add harder tasks" if saturated else "Continue"
    }

Step 8: Long-term Maintenance

# Eval suite maintenance checklist
maintenance:
  weekly:
    - Review failed evals for false negatives
    - Check for flaky tests
  monthly:
    - Add new edge cases from production issues
    - Retire saturated evals
    - Update reference solutions
  quarterly:
    - Full benchmark recalibration
    - Team contribution review

Step 4: Integrate with Production

CI/CD Integration

# GitHub Actions example
name: Agent Evals
on: [push, pull_request]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run Evals
        run: |
          python run_evals.py --suite=core --mode=compact
      - name: Upload Results
        uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: results/

Production Monitoring

# Real-time eval sampling
class ProductionMonitor:
    def __init__(self, sample_rate: float = 0.1):
        self.sample_rate = sample_rate

    async def monitor(self, request, response):
        if random.random() < self.sample_rate:
            eval_result = await self.run_eval(request, response)
            self.log_result(eval_result)
            if eval_result["score"] < self.threshold:
                self.alert("Low quality response detected")

A/B Testing

# Compare agent versions
def run_ab_test(suite: str, versions: list) -> dict:
    results = {}
    for version in versions:
        results[version] = run_eval_suite(suite, agent_version=version)
    return {
        "comparison": compare_results(results),
        "winner": determine_winner(results),
        "confidence": calculate_confidence(results)
    }

Best Practices

Do's ✅

1. Start with 20-50 representative tasks
2. Use code-based graders when possible
3. Focus on outcomes, not paths
4. Read transcripts for debugging
5. Monitor for eval saturation
6. Balance positive/negative cases
7. Isolate eval environments
8. Version your eval suites

Don'ts ❌

1. Don't over-rely on model-based graders without calibration
2. Don't ignore failed evals (false negatives exist)
3. Don't grade on intermediate steps
4. Don't skip transcript analysis
5. Don't use production data without sanitization
6. Don't let eval suites become stale

Success Patterns

Pattern 1: Graduated Eval Complexity

Level 1: Unit evals (single capability)
Level 2: Integration evals (combined capabilities)
Level 3: End-to-end evals (full workflows)
Level 4: Adversarial evals (edge cases)

Pattern 2: Eval-Driven Development

1. Write eval task for new feature
2. Run eval (expect failure)
3. Implement feature
4. Run eval (expect pass)
5. Add to regression suite

Pattern 3: Continuous Calibration

Weekly: Review grader accuracy
Monthly: Update rubrics based on feedback
Quarterly: Full grader audit with human baseline

Troubleshooting

Problem: Eval scores at 100%

Solution: Add harder tasks, check for eval saturation (Step 7)

Problem: Inconsistent model-based grader scores

Solution: Add more examples to rubric, use structured output, ensemble graders

Problem: Evals too slow for CI

Solution: Use toon mode, parallelize, sample subset for PR checks

Problem: Agent passes evals but fails in production

Solution: Add production failure cases to eval suite, increase diversity

References

• Anthropic: Demystifying evals for AI agents
• SWE-bench
• WebArena
• τ2-Bench

Examples

Example 1: Simple Coding Agent Eval

# Task definition
task = {
    "id": "fizzbuzz-001",
    "prompt": "Write a fizzbuzz function in Python",
    "test_cases": [
        {"input": 3, "expected": "Fizz"},
        {"input": 5, "expected": "Buzz"},
        {"input": 15, "expected": "FizzBuzz"},
        {"input": 7, "expected": "7"}
    ]
}

# Grader
def grade(task, outcome):
    code = outcome["code"]
    exec(code)  # In sandbox
    for tc in task["test_cases"]:
        if fizzbuzz(tc["input"]) != tc["expected"]:
            return 0.0
    return 1.0

Example 2: Conversational Agent Eval with LLM Rubric

task:
  id: "support-refund-001"
  scenario: |
    Customer wants refund for damaged product.
    Product: Laptop, Order: #12345, Damage: Screen crack
  expected_actions:
    - Acknowledge issue
    - Verify order
    - Offer resolution options
  max_turns: 5

grader:
  type: model
  model: claude-3-5-sonnet-20241022
  rubric: |
    Score 1-5 on each dimension:
    - Empathy: Did agent acknowledge customer frustration?
    - Resolution: Was a clear solution offered?
    - Efficiency: Was issue resolved in reasonable turns?