greyhaven-ai/evaluation

Evaluation Skill

Evaluate LLM outputs systematically with rubrics, handle non-determinism, and implement LLM-as-judge patterns.

Core Insight: The 95% Variance Finding

Research shows 95% of output variance comes from just two sources:

• 80% from prompt tokens (wording, structure, examples)
• 15% from random seed/sampling

Temperature, model version, and other factors account for only 5%.

Implication: Focus evaluation on prompt quality, not model tweaking.

What's Included

Examples (examples/)

• Prompt comparison - A/B testing prompts with rubrics
• Model evaluation - Comparing outputs across models
• Regression testing - Detecting output degradation

Reference Guides (reference/)

• Rubric design - Multi-dimensional evaluation criteria
• LLM-as-judge - Using LLMs to evaluate LLM outputs
• Statistical methods - Handling non-determinism

Templates (templates/)

• Rubric templates - Ready-to-use evaluation criteria
• Judge prompts - LLM-as-judge prompt templates
• Test case format - Structured test case templates

Checklists (checklists/)

• Evaluation setup - Before running evaluations
• Rubric validation - Ensuring rubric quality

Key Concepts

1. Multi-Dimensional Rubrics

Don't use single scores. Break down evaluation into dimensions:

| Dimension | Weight | Criteria | |-----------|--------|----------| | Accuracy | 30% | Factually correct, no hallucinations | | Completeness | 25% | Addresses all requirements | | Clarity | 20% | Well-organized, easy to understand | | Conciseness | 15% | No unnecessary content | | Format | 10% | Follows specified structure |

2. Handling Non-Determinism

LLMs are non-deterministic. Handle with:

Strategy 1: Multiple Runs
- Run same prompt 3-5 times
- Report mean and variance
- Flag high-variance cases

Strategy 2: Seed Control
- Set temperature=0 for reproducibility
- Document seed for debugging
- Accept some variation is normal

Strategy 3: Statistical Significance
- Use paired comparisons
- Require 70%+ win rate for "better"
- Report confidence intervals

3. LLM-as-Judge Pattern

Use a judge LLM to evaluate outputs:

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Prompt    │────▶│  Test LLM   │────▶│   Output    │
└─────────────┘     └─────────────┘     └─────────────┘
                                               │
                                               ▼
                    ┌─────────────┐     ┌─────────────┐
                    │   Rubric    │────▶│ Judge LLM   │
                    └─────────────┘     └─────────────┘
                                               │
                                               ▼
                                        ┌─────────────┐
                                        │   Score     │
                                        └─────────────┘

Best Practice: Use stronger model as judge (Opus judges Sonnet).

4. Test Case Design

Structure test cases with:

interface TestCase {
  id: string
  input: string              // User message or context
  expectedBehavior: string   // What output should do
  rubric: RubricItem[]       // Evaluation criteria
  groundTruth?: string       // Optional gold standard
  metadata: {
    category: string
    difficulty: 'easy' | 'medium' | 'hard'
    createdAt: string
  }
}

Evaluation Workflow

Step 1: Define Rubric

rubric:
  dimensions:
    - name: accuracy
      weight: 0.3
      criteria:
        5: "Completely accurate, no errors"
        4: "Minor errors, doesn't affect correctness"
        3: "Some errors, partially correct"
        2: "Significant errors, mostly incorrect"
        1: "Completely incorrect or hallucinated"

Step 2: Create Test Cases

test_cases:
  - id: "code-gen-001"
    input: "Write a function to reverse a string"
    expected_behavior: "Returns working reverse function"
    ground_truth: |
      function reverse(s: string): string {
        return s.split('').reverse().join('')
      }

Step 3: Run Evaluation

# Run test suite
python evaluate.py --suite code-generation --runs 3

# Output
# ┌─────────────────────────────────────────────┐
# │ Test Suite: code-generation                 │
# │ Total: 50 | Pass: 47 | Fail: 3              │
# │ Accuracy: 94% (±2.1%)                       │
# │ Avg Score: 4.2/5.0                          │
# └─────────────────────────────────────────────┘

Step 4: Analyze Results

Look for:

• Low-scoring dimensions - Target for improvement
• High-variance cases - Prompt needs clarification
• Regression from baseline - Investigate changes

Grey Haven Integration

With TDD Workflow

1. Write test cases (expected behavior)
2. Run baseline evaluation
3. Modify prompt/implementation
4. Run evaluation again
5. Compare: new scores ≥ baseline?

With Pipeline Architecture

acquire → prepare → process → parse → render → EVALUATE
                                                  │
                                          ┌───────┴───────┐
                                          │ Compare to    │
                                          │ ground truth  │
                                          │ or rubric     │
                                          └───────────────┘

With Prompt Engineering

Current prompt → Evaluate → Score: 3.2
Apply principles → Improve prompt
New prompt → Evaluate → Score: 4.1 ✓

Use This Skill When

• Testing new prompts before production
• Comparing prompt variations (A/B testing)
• Validating model outputs meet quality bar
• Detecting regressions after changes
• Building evaluation datasets
• Implementing automated quality gates

Related Skills

• prompt-engineering - Improve prompts based on evaluation
• testing-strategy - Overall testing approaches
• llm-project-development - Pipeline with evaluation stage

Quick Start

# Design your rubric
cat templates/rubric-template.yaml

# Create test cases
cat templates/test-case-template.yaml

# Learn LLM-as-judge
cat reference/llm-as-judge-guide.md

# Run evaluation checklist
cat checklists/evaluation-setup-checklist.md

---

Skill Version: 1.0 Key Finding: 95% variance from prompts (80%) + sampling (15%) Last Updated: 2025-01-15