RonanCodes/braintrust-tool-evals

Braintrust tool-call evals

Measure whether an LLM picks the right tool with the right arguments for a user message. Braintrust ships no canned "tool-call scorer", so this is hand-rolled but small: an Eval() whose task forces a tool call, returns a normalized { tool, args }, and two code scorers assert the tool name and the argument values.

Read /ro:braintrust first for install + auth (BRAINTRUST_API_KEY, braintrust + autoevals + zod).

The dataset shape

Each row is a user message and the tool call you expect. Keep args minimal: only the keys that must be correct.

const data = () => [
  { input: "Find houses in Utrecht under 400k",
    expected: { tool: "search_homes", args: { location: "Utrecht", maxPrice: 400000 } } },
  { input: "What can the buyer earning 80k afford?",
    expected: { tool: "estimate_affordability", args: { grossIncome: 80000 } } },
];

The eval

Force a call with toolChoice: "required", read result.toolCalls[0], normalize, score.

// evals/tool-calls.eval.ts
import { Eval } from "braintrust";
import { generateText, tool } from "ai";
import { openai } from "@ai-sdk/openai";
import { toolsFor } from "@/lib/tools";

const registry = toolsFor("adviser");

// Capture the chosen call via a closure instead of running the real tool.
// The eval measures tool selection + args, so `execute` is a no-op: it never
// touches Supabase. Mutating the real DB from an eval is the trap to avoid.
let lastCall: { tool: string; args: unknown } | undefined;
const tools = Object.fromEntries(
  registry.map((t) => [
    t.name,
    tool({
      description: t.description,
      inputSchema: t.schema, // AI SDK v6: `inputSchema`, not v4's `parameters`
      execute: async (args) => {
        lastCall = { tool: t.name, args };
        return { ok: true }; // no-op; no DB write
      },
    }),
  ]),
);

const toolSelected = ({ output, expected }) => ({
  name: "tool_selected",
  score: output?.tool === expected?.tool ? 1 : 0,
});

const argsMatch = ({ output, expected }) => {
  if (output?.tool !== expected?.tool) return { name: "args_match", score: 0 };
  const exp = expected.args ?? {}, got = output.args ?? {};
  const keys = Object.keys(exp);
  const hits = keys.filter((k) => JSON.stringify(got[k]) === JSON.stringify(exp[k])).length;
  return { name: "args_match", score: keys.length ? hits / keys.length : 1 };
};

Eval("makely-tool-calls", {
  data,
  task: async (input) => {
    const res = await generateText({
      model: openai("gpt-4o-mini"), tools, toolChoice: "required", prompt: input,
    });
    const call = res.toolCalls[0]; // AI SDK v5/v6: { toolName, input }
    return { tool: call?.toolName, args: call?.input };
  },
  scores: [toolSelected, argsMatch],
  trialCount: 3, // re-run each row 3x; scores aggregate to mean + variance
  maxConcurrency: 5,
});

Run each row N times (trialCount)

Evals should run the same case several times, because an LLM is nondeterministic: one pass tells you it worked once, not that it works reliably. Eval() takes a trialCount option that re-runs every dataset row N times (3-5 is plenty for tool selection) and aggregates the scores, so you read the mean plus the variance across trials. The mean is the accuracy number; the variance is a consistency signal in its own right. A tool that picks correctly on 2 of 3 trials is a flaky tool, even though the single-pass run might have looked green.

Confirm the call shape against the installed ai version: v5/v6 give { toolName, input }, v4 gave { toolName, args }. Source the tools from the real registry so the eval drifts with the app, not a copy.

A single toolChoice: "required" call needs no step limit. For a multi-step routing eval (let the model call, observe a result, call again), use the AI SDK v6 names: stopWhen: stepCountIs(n) (imported from ai), not v4's maxSteps.

Scorers worth reaching for

argsMatch above is exact per key. For richer scoring, swap in autoevals:

• ValidJSON — args parse and match the schema.
• JSONDiff — recursive, tolerant per-field comparison (Levenshtein on strings, NumericDiff on numbers); the best default when args have free text or amounts.
• NumericDiff — price/income within tolerance instead of exact.
• ExactMatch — tool name as a one-liner.

For multi-step routing, return an ordered array of tool names and score with an ordered or Jaccard comparison. Use an LLM judge (Factuality, LLMClassifierFromTemplate) only to judge intent, never for the deterministic tool/args assertion.

Deterministic pre-push gate (the important bit)

GitHub Actions billing is blocked here, so the real gate is local pre-push and it must be free and deterministic. Mirror apps/web/scripts/eval-matching.ts: a plain tsx script with a handful of pinned cases, a committed baseline JSON, and process.exit(1) on regression. The only model spend is the tool-selection calls themselves; skip log upload.

// scripts/eval-tools.ts — run: tsx scripts/eval-tools.ts
import { readFileSync, writeFileSync, existsSync } from "node:fs";
// ...build `tools`, run the ~6 pinned cases with toolChoice:"required"...

const meanToolSelected = results.filter((r) => r.tool === r.expectedTool).length / results.length;
const baselinePath = "scripts/eval-tools-baseline.json";

if (process.argv.includes("--update")) {
  writeFileSync(baselinePath, JSON.stringify({ tool_selected: meanToolSelected }, null, 2));
  process.exit(0);
}
const baseline = existsSync(baselinePath)
  ? JSON.parse(readFileSync(baselinePath, "utf8")).tool_selected : 1;
if (meanToolSelected < baseline) {
  console.error(`tool_selected ${meanToolSelected} < baseline ${baseline}`);
  process.exit(1);
}

If the gate runs each case more than once, gate on the mean tool_selected across trials, not a single pass, so one lucky run cannot mask a regression. High variance across trials is its own warning sign even when the mean stays above baseline; surface it rather than swallow it.

Wire tsx scripts/eval-tools.ts into the pre-push hook. Run with --update to re-baseline after an intentional change, and commit eval-tools-baseline.json. To avoid uploading logs from the gate, run bt eval ... --no-send-logs or use the plain tsx + conditional initLogger pattern from eval-matching.ts.

npm scripts + CI

"eval:tools":      "bt eval evals/tool-calls.eval.ts",
"eval:tools:gate": "tsx scripts/eval-tools.ts"

bt eval evals/ --no-input --json is the CI-friendly form (no prompts, machine-readable). First-party GitHub Action when billing allows:

- uses: braintrustdata/eval-action@v1
  with:
    api_key: ${{ secrets.BRAINTRUST_API_KEY }}
    runtime: node
    root: apps/web/evals

Add fail_on_regression: true and a min_score to block a merge on a drop. Reserve the full LLM-judge runs for on-demand or nightly; keep the pre-push gate deterministic.

See also

• /ro:braintrust — install, auth, Eval() basics, the Vercel-vs-Workers flush note.
• /ro:braintrust-vision-evals — OCR/document-extraction evals.
• /ro:braintrust-trace-surfaces — trace the real tool calls these evals model.