RonanCodes/braintrust

Braintrust

Wire Braintrust into a TanStack Start + Vercel AI SDK + Cloudflare Workers app. Braintrust is AI eval + observability: it traces LLM/agent calls in production and runs structured offline evals (datasets, scorers, experiments, CI deploy-gating). The two feed each other, turn real traces into eval datasets, gate deploys on scores.

Its edge over Langfuse is the eval harness (Langfuse is observability-first + OSS/self-host). Both trace the AI SDK. Reach for Braintrust when you want to measure prompt/model changes, not just watch them.

Usage

/ro:braintrust install        # CLI + env + initLogger
/ro:braintrust add-tracing     # wrapAISDK over streamText/generateText
/ro:braintrust add-evals       # *.eval.ts with autoevals scorers + bt eval
/ro:braintrust add-proxy       # OpenAI-compatible proxy (many providers, one key)
/ro:braintrust ops             # CLI cheat-sheet

Prerequisites

• A Braintrust account. Key in ~/.claude/.env: BRAINTRUST_API_KEY (starts with sk-).
• Packages: braintrust (tracing + Eval()), autoevals (scorers). braintrust v2+ has zod as a peer dep — install it yourself.
• For the AI SDK app: /ro:vercel-ai-sdk already wired (ai + a provider).

Install flow

1. CLI (optional auto-instrument)

curl -fsSL https://braintrust.dev/wizard/setup.sh | sh   # installs the `bt` CLI
bt setup                                                  # detects stack, installs SDK, instruments, verifies with a trace

bt setup will mint + print export BRAINTRUST_API_KEY=... if none is set. The Node --import braintrust/hook.mjs auto-instrument does not run on the Workers runtime — on Workers wire initLogger + wrapAISDK explicitly (below).

2. Env

# .dev.vars (gitignored)
BRAINTRUST_API_KEY=sk-...

wrangler secret put BRAINTRUST_API_KEY

On Workers pass the key from env, not process.env (not populated on workerd without nodejs_compat).

3. Install

pnpm add braintrust autoevals zod

add-tracing (AI SDK, the key part)

Current integration is wrapAISDK (wrap the whole ai module; v5+ gets per-tool-call child spans). wrapAISDKModel (wrap one model) still exists; wrapAISDK is the documented path.

import { initLogger, wrapAISDK, wrapTraced, traced, currentSpan } from 'braintrust'
import * as ai from 'ai'
import { openai } from '@ai-sdk/openai'

// once at startup (pass env.* on Workers, not process.env)
const logger = initLogger({ projectName: 'smart-cart', apiKey: env.BRAINTRUST_API_KEY })

const { generateText, streamText } = wrapAISDK(ai)

export async function chat(request: Request) {
  return traced(
    async () => {
      const { messages } = await request.json()
      const result = await streamText({ model: openai('gpt-4o'), messages })
      return result.toDataStreamResponse()
    },
    { type: 'function', name: 'POST /api/chat' },
  )
}

// custom spans for non-LLM work (tools, helpers)
const lookupPrice = wrapTraced(
  async function lookupPrice({ sku }: { sku: string }) {
    currentSpan().log({ metadata: { sku } })
    return fetchPrice(sku)
  },
  { type: 'tool', name: 'lookupPrice' },
)

API names (verified): initLogger, wrapAISDK, wrapAISDKModel, wrapTraced, traced, currentSpan, plus wrapOpenAI / wrapAnthropic for raw clients. wrapAISDK traces top-level call I/O + tool calls (v5+); use wrapTraced for intermediary steps it won't auto-capture.

Cloudflare Workers flush (the #1 footgun)

Logs POST over HTTP (fetch), so the SDK runs on the edge. But asyncFlush (default true) flushes in the background via waitUntil — if the isolate ends first, logs silently drop.

• Workers HAS waitUntil, so default asyncFlush: true mostly works. Belt-and-braces:

  // inside fetch(request, env, ctx)
  const logger = initLogger({ projectName: 'smart-cart', apiKey: env.BRAINTRUST_API_KEY })
  // ...work...
  ctx.waitUntil(logger.flush())   // ensure logs leave before freeze
  return response
  

• Runtimes without waitUntil: set asyncFlush: false and await logger.flush() before returning.
• Missing traces → it's almost always this. Wire ctx.waitUntil(logger.flush()).

"Works ootb with Vite/Next" is true via the bundler plugin / --import hook. On Workers that hook doesn't run, so wire initLogger + wrapAISDK explicitly (a few lines) and pass the key via env. Braintrust names Workers as supported, but there's no first-party Workers-runtime page (the /cloudflare doc is about AI Gateway, not workerd) — smoke-test a real trace from a deployed Worker.

Vercel / Node vs Cloudflare Workers (flush)

The asyncFlush + ctx.waitUntil(logger.flush()) dance above is Workers-only. On Vercel / Node there is no ctx.waitUntil and the isolate-dying-early failure mode is different: a short-lived serverless function can return before a background flush completes. So on Vercel/Node, await logger.flush() before the function returns. Use process.env.BRAINTRUST_API_KEY there (it is populated), unlike Workers where you thread env. Pick the rule by runtime: Workers → ctx.waitUntil(logger.flush()); Vercel/Node → await logger.flush().

add-evals

Eval(project, { data, task, scores }) in a *.eval.ts file; run with bt eval.

// scenarios.eval.ts
import { Eval } from 'braintrust'
import { Factuality, Levenshtein } from 'autoevals'
import { generateText } from 'ai'
import { openai } from '@ai-sdk/openai'

Eval('smart-cart', {
  data: () => [
    { input: 'eating out Wednesday', expected: 'Wednesday removed from the week' },
    { input: 'no fish', expected: 'fish dishes swapped out' },
  ],
  task: async (input) => {
    const { text } = await generateText({ model: openai('gpt-4o-mini'), prompt: input })
    return text
  },
  scores: [Factuality, Levenshtein],
  maxConcurrency: 5,
})

bt eval scenarios.eval.ts        # one file
bt eval                          # all *.eval.ts in cwd
bt eval --watch scenarios.eval.ts
bt eval --no-input               # CI-friendly (no prompts)
bt eval --no-send-logs           # dry run, don't upload

autoevals scorers: LLM-judge (Factuality, ClosedQA, Moderation, Security, Summary, Sql, Translation, Battle), RAG (context precision/recall/relevancy, faithfulness, answer correctness/similarity), heuristic (Levenshtein, ExactMatch, NumericDiff, JSONDiff, ValidJSON, ListContains), embedding (EmbeddingSimilarity).

Custom scorer (return { name, score }, score 0..1):

const NamesADay = ({ output }: { output: string }) => ({
  name: 'names_a_day',
  score: /monday|tuesday|wednesday|thursday|friday|saturday|sunday/i.test(output) ? 1 : 0,
})
// scores: [Factuality, NamesADay]

Custom LLM-judge: LLMClassifierFromTemplate({ name, promptTemplate, choiceScores }). Managed datasets: initDataset(project, { dataset }).

add-proxy

OpenAI-compatible proxy at https://api.braintrust.dev/v1/proxy: one key for many providers + automatic logging + caching.

import OpenAI from 'openai'
const client = new OpenAI({
  baseURL: 'https://api.braintrust.dev/v1/proxy',
  apiKey: env.BRAINTRUST_API_KEY,
})

Use the proxy for "log everything + many providers behind one key" with a raw client. For an AI-SDK app prefer direct provider + wrapAISDK (keeps the typed provider, streaming, tool loops, richer spans). The proxy doc page is marked deprecated but the endpoint is live + open-source (braintrustdata/braintrust-proxy).

Gotchas

1. zod peer dep in braintrust v2+ — install it or resolution fails.
2. Edge flush timing — default asyncFlush:true relies on waitUntil; isolate dying early drops logs. ctx.waitUntil(logger.flush()).
3. process.env on Workers — thread env through to initLogger/proxy instead.
4. API drift — wrapAISDKModel → wrapAISDK; proxy doc deprecated while endpoint lives. Trust current docs over old blog snippets.
5. AI SDK v6 tool/agent names: when an eval task defines tools, use v6's inputSchema (not v4's parameters) and, for multi-step tasks, stopWhen: stepCountIs(n) (not maxSteps). v5/v6 tool calls read as { toolName, input }. The eval siblings build on this.

Braintrust vs Langfuse vs PostHog

• Braintrust: evals central (datasets, scorers, experiment diffs, CI deploy-gating) + tracing feeding the loop. Measure changes, not just watch.
• Langfuse: observability-first, OSS/self-host (data residency, no per-seat cost), lighter evals. (Acquired by ClickHouse Jan 2026, capabilities unchanged.)
• PostHog LLM analytics: product-analytics-native LLM cost/latency/usage + session-replay tie-in; not a real eval harness. Use if you already run /ro:posthog.

For Souso (and the megathon): Braintrust for evals (the synthetic-scenario tests), Langfuse if/when you want OSS self-hosted observability. They can coexist; Braintrust alone covers both.

Env var summary

BRAINTRUST_API_KEY=sk-...    # server-only secret

Safety

Server-side only. On Workers pass the key via env and ensure ctx.waitUntil(logger.flush()) so traces aren't dropped. Evals can call real models, mind cost with maxConcurrency and cheap models for fast iterations.

See also

• Docs: AI-SDK integration https://www.braintrust.dev/docs/integrations/sdk-integrations/vercel, tracing quickstart, autoevals reference.
• Eval-family siblings (Makely-shaped, built on this generic installer):
  • /ro:braintrust-tool-evals — eval tool selection + arguments (input → expected tool + args), deterministic pre-push gate.
  • /ro:braintrust-vision-evals — OCR / document-extraction evals (extracted JSON vs golden, EU vision models, BSN rules).
  • /ro:braintrust-trace-surfaces — trace chat + Vapi + MCP through one wrapTraced wrapper, log→dataset loop, cost-by-model BTQL.
• Siblings: /ro:vercel-ai-sdk (the SDK Braintrust wraps), /ro:posthog, /ro:sentry. Observability alternative: Langfuse.