mohitagw15856/eval-rubric-designer

Eval Rubric Designer Skill

You can't improve what you can't score. The hard part of evaluating AI output isn't running the judge — it's defining dimensions that are specific, observable, and independent, with anchors concrete enough that two people (or two judge runs) agree. This skill turns "is the output good?" into a rubric and a judge prompt you can run today.

Working from a brief

Given just "I need to eval my summariser", produce the full rubric anyway — infer the task, the output type, and the dimensions that matter for it, and label inferred choices. Never hand back a list of dimension names with no anchors; the anchors are where the rubric earns its keep.

Required Inputs

Ask for these only if they aren't already provided (else infer and label):

• The task — what the AI is supposed to produce, and for whom.
• A sample output (or two) — ideally one good and one weak, to calibrate anchors.
• What "good" means here — the quality bar and any non-negotiables (e.g. must be grounded, must follow format).
• How it'll be scored — human review, LLM-as-judge, or both; and whether you need a single score or per-dimension.

Output Format

Eval Rubric: [task]

1. Dimensions — 3–6 independent dimensions, each with a one-line definition and a weight. Default set, tailored to the task: structure, completeness, correctness/grounding, usefulness, safety/tone.

2. Anchors — for each dimension, concrete descriptions at 1, 3, and 5 (what a poor / acceptable / excellent answer looks like for this task). Anchors must be observable, not "feels good".

| Dimension (weight) | 1 — poor | 3 — acceptable | 5 — excellent | |---|---|---|---| | Grounding (×2) | invents facts not in the source | mostly grounded, minor drift | every claim traceable to the source |

3. Judge prompt — a ready-to-run LLM-as-judge prompt in a fenced block: the task description, the rubric, an instruction to score each dimension 1–5, and a strict JSON output contract ({"dimension":N,...}) so scores parse reliably. Include a one-line "return only JSON" reinforcement.

4. Labelling guide — short rules for tie-breaks and common edge cases, so repeat runs stay consistent.

5. Judge reliability notes — known biases (length, position, self-preference), and how to mitigate: a cheaper judge for scale vs. a stronger judge for the rubric, sampling N runs, and spot-checking judge scores against a few human labels before trusting the leaderboard.

Quality Checks

• [ ] Dimensions are independent — a single flaw doesn't tank three of them at once
• [ ] Every dimension has concrete 1/3/5 anchors specific to this task, not generic adjectives
• [ ] The judge prompt has a strict, parseable output contract (JSON), with a retry/repair note
• [ ] Weights reflect what actually matters for the task (grounding usually > prose polish)
• [ ] The rubric is calibrated against at least one good and one weak sample
• [ ] Judge biases are named with a concrete mitigation, not just listed

Anti-Patterns

• [ ] Do not ship dimension names without anchors — names alone don't make scores reproducible
• [ ] Do not let one quality issue load onto multiple dimensions — keep them orthogonal
• [ ] Do not trust an LLM judge blind — calibrate against a handful of human labels first
• [ ] Do not use a vague "overall quality 1–10" — it hides which part is broken
• [ ] Do not ignore the negative case — a rubric must distinguish "wrong" from "thin", not just "great" from "okay"

Based On

LLM-as-judge evaluation practice — orthogonal weighted dimensions, anchored scales, structured judge prompts, and judge-bias mitigation.