RonanCodes/skills/braintrust-vision-evals

Braintrust vision / OCR document-extraction evals

The key framing: an OCR or document-intake eval is a JSON-vs-JSON eval where the input happens to be an image. There is no special OCR monitor or named recipe. You feed the document in, the task runs the vision model and returns parsed JSON, and you score that JSON against a golden sample with the same recursive scorers you would use for any structured output.

Read /ro:braintrust first for install + auth.

Dataset: doc in, golden JSON out

Put the document in input and the hand-checked field JSON in expected. Use a Braintrust Attachment for the doc, it dodges span-size limits that base64 in the span would hit. A base64 data URL or an external URL also work for small docs.

import { Attachment } from "braintrust";
import { readFileSync } from "node:fs";

const data = () => [
  {
    input: new Attachment({
      data: readFileSync("fixtures/payslip-001.pdf"),
      filename: "payslip-001.pdf",
      contentType: "application/pdf",
    }),
    expected: { grossMonthly: 4250.0, employer: "Acme BV", iban: "NL00BANK0000000000" },
  },
];

Task: fetch bytes, call the model, parse

task: async (input) => {
  const bytes = await input.data();            // Attachment.data() → the raw bytes
  const res = await extractWithVisionModel(bytes); // your model call, structured-output mode
  return JSON.parse(res);                       // return the parsed object, not a string
},

Scorers

import { ValidJSON, JSONDiff, NumericDiff, Levenshtein } from "autoevals";

• ValidJSON — did it parse and match the expected shape. The floor.

• JSONDiff — the headline number. Recursively compares objects (per-key, averaged), arrays (element-wise) and leaves: strings get Levenshtein partial credit, numbers get tolerant NumericDiff. So a near-miss on a name or a rounding wobble on an amount scores high instead of zero. Parameterize the leaf scorers:

  JSONDiff({ output, expected, stringScorer: Levenshtein, numberScorer: NumericDiff });
  

• Custom per-field F1 — there is no built-in. Iterate the expected keys, apply a per-field comparator (exact for IBAN / dates / IDs, NumericDiff for amounts, Levenshtein or embeddings for names), average to one score and emit per-field sub-scores in metadata so a regression points at the field that broke.

const fieldF1 = ({ output, expected }) => {
  const cmp = { iban: exact, grossMonthly: numeric, employer: fuzzy };
  const subs = Object.keys(expected).map((k) => ({ k, s: (cmp[k] ?? exact)(output?.[k], expected[k]) }));
  const mean = subs.reduce((a, b) => a + b.s, 0) / subs.length;
  return { name: "field_f1", score: mean, metadata: Object.fromEntries(subs.map((x) => [x.k, x.s])) };
};

Run each document N times (trialCount)

A vision model is as nondeterministic as a chat model: the JSON it extracts from the same scan can wobble between runs. Pass trialCount to Eval() so every document runs N times and the field scores aggregate to mean plus variance, which measures consistency rather than a single lucky extraction.

Eval("makely-doc-extraction", {
  data,
  task,
  scores: [ValidJSON, JSONDiff, fieldF1],
  trialCount: 3, // each doc extracted 3x; read the mean + variance per field
  maxConcurrency: 5,
});

Keep trialCount low (3) when each trial is a paid vision call. A field that flips between trials is flaky extraction even if its mean looks acceptable.

EU-resident vision model choice

For Dutch documents the data must stay in the EU. As of mid-2026:

• Mistral OCR 4 — top pick. EU company, self-hostable container, returns structured JSON annotations.
• Gemini 3.x via Vertex AI (EU region) — native JSON Schema output, strong on messy or low-quality scans.
• Claude vision — only via Bedrock or Vertex in an EU region.
• Azure Document Intelligence — prebuilt ID / receipt models if you want a turnkey extractor.

Re-validate any 2026 benchmark numbers on your own documents before committing, and confirm zero-retention plus self-host licensing before any document carrying a BSN.

The hard privacy rules (non-negotiable)

• Never persist the BSN. Extract it, validate it, then redact or discard the raw value. Keep only a derived verified boolean or a salted token, never the digits.
• Eval data uses masked or synthetic BSNs only. The committed dataset and any Attachment in it must not contain a real citizen-service number. Generate synthetic ones for fixtures.
• Keep the Braintrust project in the EU data plane (api-eu.braintrust.dev) so traces and dataset rows stay EU-resident, matching the EU Sentry and PostHog setup.

npm scripts + CI

"eval:extract": "bt eval evals/doc-extraction.eval.ts"

bt eval evals/ --no-input --json for CI; braintrustdata/eval-action@v1 with fail_on_regression: true to gate. Vision evals cost model spend, so run them on-demand or nightly as uploaded experiments rather than in the pre-push gate; keep pre-push to the deterministic tool-call gate.

See also

• /ro:braintrust — install, auth, Eval() basics, the Vercel-vs-Workers flush note.
• /ro:braintrust-tool-evals — tool-selection + argument evals.
• /ro:braintrust-trace-surfaces — trace the production document-intake calls.