jaktestowac/testing-llm-features

Testing LLM Features

Use this skill when a product's behaviour depends on a model, and the current verification method is a person looking at output and deciding it seems fine.

The problem is not that LLM output is hard to assert on. It is that the same input produces different output, and a prompt or model change can degrade quality without breaking anything. No test in a normal suite fails. Coverage stays green. The feature just gets worse, and you find out from users.

An eval suite is the unit-test layer for this: a fixed set of cases, run on every change that could affect model behaviour, scored consistently enough that a regression is visible.

When to Use

• a feature calls a model and there is no automated check on output quality
• a prompt, model version, temperature, tool list, or retrieval config is about to change
• a model upgrade is proposed and nobody can say what would regress
• an agent or chain has grown beyond what manual spot-checks can cover
• RAG answers are drifting and it is unclear whether retrieval or generation is at fault
• a CI gate is needed for AI behaviour, not just AI code

Operating Principles

• Golden cases, not vibes. A case has an input, an expectation, and a scoring rule. "Looks good" is not an expectation.
• Determinism where you can get it. Pin the model version, set temperature to 0, fix seeds, freeze retrieval snapshots. Non-determinism is a property of the model; it is not an excuse for a non-reproducible test.
• Test the assertable layer first. Schema conformance, required fields, refusal behaviour, citation presence, tool-call shape, latency and cost. Most feature-breaking regressions are structural, and structural checks are cheap and exact.
• Rubrics before judges. If an LLM judges the output, it needs a written rubric with a scale, and the judge itself needs validating against human labels. An unrubriced judge is a random number with an API bill.
• A regression is a new failure on a case that used to pass. That is the gate. An absolute score threshold on a hard case set gates nothing useful.
• Separate retrieval failures from generation failures. A RAG answer can be wrong because the right document was never retrieved. Score retrieval independently or you will tune the prompt to fix an index problem.
• Cases are a product artifact. They encode what the feature promises. Review them like code and grow them from real failures.
• Cost and latency are results. A 12% quality gain for 4× the cost and 3× the latency is a product decision, so report all three.

Workflow

Phase 0: Map the surface

Find everything that can change model behaviour, because each is a trigger for the suite:

| Element | Where to look | | --- | --- | | Prompt templates | prompts/, *.prompt.*, template literals passed to the SDK | | System prompts | client construction, agent definitions | | Model config | model id, temperature, top_p, max tokens, stop sequences, thinking budget | | Tool definitions | tool/function schemas, MCP servers, tool grants | | Retrieval | chunking, embedding model, index, topK, reranking, filters | | Output contract | Zod/schema on the response, structured-output config, parsing code | | Orchestration | chain and agent-loop structure, retries, fallbacks between models |

Write down what the feature promises - the user-visible contract. "Summarise the ticket in ≤3 sentences, never invent a customer name, always cite the ticket id." That sentence is where the cases come from.

Phase 1: Design the case set

./resources/eval-case-design.md has the taxonomy. The minimum viable set per prompt template:

| Class | Purpose | Count to start | | --- | --- | --- | | Golden | canonical inputs with known-good outputs | 5–10 | | Edge | empty, huge, multilingual, malformed, contradictory input | 5 | | Negative | must refuse, must not answer, must not leak | 3–5 | | Regression | one case per bug ever found in production | grows forever | | Adversarial | injection and jailbreak attempts | see testing-llm-guardrails |

Cases carry the metadata that makes a failure diagnosable: which prompt template, which capability, why the expectation is what it is. A case whose expectation nobody can justify is a case nobody will maintain.

The regression class is the one that compounds. Every production incident becomes a permanent case. After six months this set is the most valuable test asset the feature has, and it costs one case per incident to build.

Phase 2: Choose scoring per case

./resources/scoring-strategies.md covers each in detail. Pick the cheapest one that can actually fail:

| Strategy | Use when | Cost | | --- | --- | --- | | Exact / regex match | classification, extraction, IDs, enum outputs | free, exact | | Schema validation | any structured output | free, exact | | Required / forbidden substrings | must cite the ticket id, must never say "as an AI" | free | | Deterministic property | length limits, valid JSON, no PII pattern, citation resolves to a real source | free | | Embedding similarity | paraphrase-tolerant equivalence | cheap, fuzzy | | Rubric-based LLM judge | tone, helpfulness, faithfulness | expensive, needs validation | | Human review | the final arbiter on a sample | expensive, definitive |

The ordering is the advice. Teams reach for an LLM judge first because it feels closest to "quality", and end up with a noisy score they cannot debug. Most real regressions trip a free check.

Phase 3: Implement the suite

Two viable shapes, both in ./resources/promptfoo-setup.md:

• promptfoo - config-driven, good matrix support across prompts and models, built-in assertion types, decent reporting. The default choice for a suite that will grow.
• Vitest + the SDK - the eval suite as ordinary tests. Better when evals must share fixtures with the app, or when the team will not adopt another tool.

Non-negotiables whichever you pick:

• pin the exact model version (claude-sonnet-4-5-20250929, not latest) - a floating alias makes every run unreproducible
• temperature 0 unless the feature's value depends on variation
• freeze retrieval: snapshot the documents, do not query a live index
• record model, prompt hash, and case-set version with every result
• cache responses so a re-run of unchanged cases is free
• run N times on cases where variance matters, and report the distribution, not one sample

Phase 4: Gate on regressions

The gate is comparative:

| Condition | Verdict | | --- | --- | | A case that passed on the baseline now fails | BLOCK - this is the whole point | | Aggregate score drops beyond the noise band | WARN - investigate before merging | | New cases fail | INFO - they were added because they fail | | Cost or latency regresses beyond budget | WARN | | Flaky case (passes and fails across N runs) | fix the case or the determinism, do not average it away |

Establish the noise band empirically: run the suite three times on an unchanged system and see how much the score moves. Anything inside that band is not a signal, and gating on it produces the flakiest CI job in the repo.

Store the baseline as a committed artifact so the comparison is reviewable in the diff. See ./resources/promptfoo-setup.md for the CI job, the trigger paths, and the API-key-free path for forks.

Phase 5: Report

• Regressions - case, what it asserted, baseline versus now, the actual output
• Score by capability - not one global number; "citation accuracy 94%, refusal 100%, summary faithfulness 81%" is actionable
• Retrieval versus generation - for RAG, always split
• Cost and latency - per case and total, against budget
• Variance - which cases are unstable across runs
• Coverage gaps - prompt templates with no cases, capabilities with no negative case

Common Failure Modes

• Evals only in a notebook. They exist, they are never run in CI, and the regression ships anyway.
• LLM judge with no rubric. Produces a number that moves for unrelated reasons and cannot be debugged.
• Judging with the model under test. Self-preference is real; use a different model, or a deterministic check.
• Floating model alias. latest silently changes the system under test between runs.
• One aggregate score. Masks a 40% drop in citation accuracy behind a 3% total move.
• Golden outputs copied from model output. Recording what the model did rather than what it should do freezes today's bugs into the expectations. Golden outputs need a human's approval.
• No negative cases. The feature that must refuse is never tested for refusing, and the first jailbreak is found by a user.
• Tuning the prompt to fix a retrieval miss. Weeks of prompt work on an indexing bug.
• Cases that grow only from imagination. Every production failure should become a case; that is where the good cases come from.

Resource Map

• ./resources/eval-case-design.md - case taxonomy, how to write a golden expectation, case metadata, growing the set from incidents
• ./resources/scoring-strategies.md - each scoring method, when to use it, LLM-judge rubrics and how to validate a judge
• ./resources/promptfoo-setup.md - promptfoo and Vitest implementations, determinism settings, CI job, baseline comparison, cost control

Related Skills

• testing-llm-guardrails - runtime output validation and the adversarial suite: injection, jailbreaks, tool abuse
• reviewing-ai-output-groundedness - the human review protocol for factuality, and the attestation an eval suite cannot replace
• designing-test-data - building the input corpora these cases consume
• analyzing-quality-metrics - defining and trending eval pass rate, variance, and cost as real metrics
• unslop-tests - the same discipline applied to test code: an eval that cannot fail is worse than no eval
• scoping-change-relevance - which changes should trigger the suite (modified-prompt, model config, tool grants, retrieval)

Definition of Done

This skill is complete when:

• every prompt template has golden, edge, and negative cases with justified expectations
• scoring is the cheapest method that can actually fail, and any LLM judge has a written rubric and validation against human labels
• the model version, temperature, seeds, and retrieval snapshot are pinned and recorded with results
• the suite runs in CI on prompt, model, tool, and retrieval changes, and gates on regressions against a committed baseline
• the noise band is measured, so the gate does not fire on variance
• retrieval and generation failures are reported separately for RAG features
• cost and latency are reported alongside quality
• every past production failure has a permanent regression case