RonanCodes/braintrust-trace-surfaces

Braintrust trace surfaces

Trace production tool calls from all three AI front doors through a single chokepoint. The win: every surface, in-app chat (Vercel AI SDK), Vapi voice webhook, and the MCP server, funnels through one shared ToolSpec.execute registry, so wrapping that once with wrapTraced traces all three. Chat additionally gets per-tool child spans for free from wrapAISDK.

Read /ro:braintrust first for install + auth.

Init once

// lib/braintrust.ts
import { initLogger, wrapAISDK } from "braintrust";
import * as ai from "ai";

initLogger({ projectName: "makely", apiKey: process.env.BRAINTRUST_API_KEY });

export const { generateText, streamText } = wrapAISDK(ai);

On the chat route, call the wrapped generateText / streamText and the SDK emits a child span per tool call automatically (span type function).

The one wrapper for all three surfaces

Vapi and MCP do not go through the AI SDK, so wrapAISDK cannot see them. Wrap the shared execute instead, once, and map every adapter's tools through it.

// lib/tools-traced.ts
import { wrapTraced, currentSpan } from "braintrust";

export function traceTool(spec, surface /* "chat" | "vapi" | "mcp" */) {
  const execute = wrapTraced(
    async (args, ctx) => {
      currentSpan().log({
        metadata: {
          tool: spec.name, surface,
          role: ctx.role, agencyId: ctx.agencyId, trace_id: ctx.traceId,
        },
      });
      return spec.execute(args, ctx);
    },
    { name: spec.name, type: "function" },
  );
  return { ...spec, execute };
}

Wire it per adapter by mapping toolsFor(role) through traceTool:

• Chat — api/chat/route.ts: tools mapped through traceTool(spec, "chat") (or rely on wrapAISDK for the chat spans and use this for the non-AI-SDK surfaces only).
• Vapi — api/vapi/webhook/route.ts: traceTool(spec, "vapi") where the call's tool requests are dispatched.
• MCP — api/[transport]/route.ts: traceTool(spec, "mcp") inside the handler builder.

Logging trace_id keeps these spans aligned with the repo's FE → BE → Sentry → PostHog correlation, so one id ties a request across all observability tools.

Vapi + MCP need a manual root span (no AI SDK call to wrap)

wrapAISDK gives the chat route a root span for free because the LLM call runs locally there. Voice and MCP have no local LLM call: the model lives in Vapi (or in the client connecting to the MCP server), so there is no AI-SDK call to wrap and nothing to anchor the tool spans under. Open a root span yourself per request, then the traceTool spans nest inside it.

// Vapi webhook or MCP handler: one root span per request
import { initLogger } from "braintrust";

const logger = initLogger({ projectName: "makely", apiKey: process.env.BRAINTRUST_API_KEY });

export async function POST(req: Request) {
  return logger.traced(
    async (span) => {
      span.log({ metadata: { surface: "vapi", trace_id } });
      // ...dispatch the call's tool requests through traceTool(spec, "vapi")...
      // those tool spans nest under this root span
    },
    { name: "POST /api/vapi/webhook", type: "function" },
  );
}

Use the same pattern in the MCP [transport] handler with surface: "mcp". wrapAISDK stays the chat route's anchor only; the other two front doors get this hand-opened root span.

Flush on Vercel / Node

This app is Next.js on Vercel (Node), so await logger.flush() before the serverless function returns. The asyncFlush + ctx.waitUntil(logger.flush()) dance is a Cloudflare Workers concern only; it does not apply here. See the Vercel-vs-Workers note in /ro:braintrust.

EU OTLP option

To send spans over OpenTelemetry instead of the SDK transport (keeps everything in one OTEL pipeline), use the Braintrust exporter against the EU data plane:

// instrumentation.ts
import { BraintrustExporter } from "@braintrust/otel";
// OTEL_EXPORTER_OTLP_ENDPOINT=https://api-eu.braintrust.dev/otel
// header: x-bt-parent=project_id:<PROJECT_ID>
// set filterAISpans: false so hand-built tool spans are not dropped

Add @braintrust/otel + @vercel/otel. Use the EU endpoint to match EU Sentry and PostHog.

Log → dataset loop

Traced production calls become eval data. In the UI, the "Log" button on a span promotes it to a dataset row. In code:

import { initDataset } from "braintrust";
const ds = initDataset("makely", { dataset: "real-tool-calls" });
ds.insert({ input: span.input, expected: span.output });

Curate real traces (especially the ones that went wrong) into the dataset that /ro:braintrust-tool-evals runs against, so the eval suite tracks what users actually ask.

Cost + tokens by model (BTQL)

Braintrust records estimated_cost and token metrics on every LLM span natively, so no custom instrumentation is needed to roll up spend per model:

SELECT metadata.model,
       sum(estimated_cost())        AS cost,
       sum(metrics.prompt_tokens)   AS in_tok,
       sum(metrics.completion_tokens) AS out_tok
FROM project_logs('<PROJECT_ID>')
WHERE span_attributes.type = 'llm'
  AND created > now() - interval 7 day
GROUP BY metadata.model
ORDER BY cost DESC

Run it in the BTQL console or via the SQL API to see weekly spend split by model across all three surfaces.

See also

• /ro:braintrust — install, auth, wrapAISDK / wrapTraced basics, Vercel-vs-Workers flush.
• /ro:braintrust-tool-evals — eval the tool calls you are tracing here.
• /ro:braintrust-vision-evals — OCR/document-extraction evals.