glebis/rag-eval

rag-eval

Purpose

Replace the "tweak → squint → swap model → burn credits" loop with a single command that runs a grid of eval variants on the user's gold-set, ranks them by a cost-aware score, and returns structured feedback on architecture, stack, and likely-issues. Draws on evidence-based RAG practices and learns from the user's past runs.

When to use

Trigger on: "help me test a RAG", "tune my RAG", "my RAG is bad", "compare retrieval prompts", "how do I eval this", "what's the best embedding model for X", "my RAG eval is expensive". Also trigger when the user reports burning OpenRouter / OpenAI credits with no clear signal of improvement.

Prerequisites — gather before running

Collect these from the user before the first sweep. Many are optional with sensible defaults; always confirm the ones that gate cost.

1. RAG codebase root — path to the repo/module under test.
2. Gold-set — at least 10 Q&A pairs. If missing, offer to generate a starter gold-set from the user's dataset (LLM-synthesized, human-reviewed). See references/best-practices.md.
3. Dataset — the corpus the RAG retrieves over.
4. Budget cap — hard dollar limit per run (default: $2 if user doesn't specify). Always confirm before any sweep.
5. Provider keys — OPENROUTER_API_KEY or OPENAI_API_KEY (read from env).
6. Vector-store config — collection name, embedding model, chunk size (read from repo; confirm if ambiguous).
7. Eval history path (optional) — defaults to .rag-eval/history.jsonl in the repo root.

Workflow

Follow this order. Refer to references/best-practices.md for the canonical checklist and references/evidence-base.md for the research-backed defaults.

Step 0 — (Optional) Ingest a prior iteration session

When the user provides a session ID (Claude Code transcript, skill-studio session, or a Fathom meeting), run the deterministic ingest first — no LLM calls. This extracts only the useful signals (models tried, prompt variants, cost events, eval results) as compact JSON, so the rest of the skill works off a tiny structured bundle instead of a long raw transcript.

python scripts/session_ingest.py <session_id> > /tmp/rag-eval-bundle.json
# or with a direct path:
python scripts/session_ingest.py --path /path/to/transcript.jsonl > /tmp/rag-eval-bundle.json

The bundle includes: models_tried, prompts_tried (hashes only), iterations, total_cost_usd, summary_stats. Feed this into Step 1 — do not paste the raw transcript.

Why this matters: transcripts can be 100k+ tokens of noise. The ingest script does regex extraction only, keeping the LLM budget for the actual audit + sweep planning. This is a hard requirement, not an optimization.

Step 1 — Audit the stack

Read references/best-practices.md and inspect the user's repo + vector-store config. Produce a structured report covering:

• Architecture (retrieval type: dense / hybrid / rerank; chunking strategy; prompt structure)
• Tech stack (embedding model, LLM, vector store)
• Resources (dataset size, gold-set size, prior eval runs)
• Risks (known anti-patterns, missing pieces)

Present the report to the user and ask which issues to address first.

Step 2 — Propose a sweep plan

Based on the audit, propose 3–8 variants to test. Keep the grid small on the first run (default: 2 prompts × 2 models × 1 retrieval variant = 4 cells). Estimate cost using gold-set size × variants × avg tokens × provider pricing. Present the cost estimate and wait for user confirmation before running.

Step 3 — Run the sweep

Use scripts/eval_sweep.py (see the script header for invocation). It reads a config YAML, runs each variant against the gold-set, records per-variant cost and answer quality, and appends to history.jsonl.

Guardrails:

• Never exceed the budget cap — halt mid-sweep if reached.
• Never mutate the user's repo. Write all artifacts under .rag-eval/ (gitignore it).
• Confirm before any sweep estimated to exceed the user's cap.

Step 4 — Rank and report

After the sweep, rank variants by a cost-aware score: quality × (1 / log(1 + cost)). Present:

• Top 3 variants with quality metrics and cost
• What changed vs the previous best
• Concrete next experiment to try

Write the full report to .rag-eval/reports/<timestamp>.md.

Step 5 — Self-improve

Before each subsequent run, read history.jsonl and factor in what the user has already tried. Avoid re-testing rejected variants. Surface patterns ("models A, B, C all underperformed on multi-hop queries — next try a reranker").

Reusable resources

• scripts/eval_sweep.py — grid-search runner. Reads eval_config.yaml, writes results to history.jsonl.
• references/best-practices.md — evidence-based RAG checklist the agent uses as an anchor.
• references/evidence-base.md — pointers to recent RAG research and when each technique helps.
• assets/eval_config.template.yaml — starter config to copy into the user's repo.
• assets/gold_set.template.jsonl — 3 example Q&A pairs to show the gold-set format.

Notes

• Cost is the main failure mode. Never run without a confirmed budget. Err on the side of smaller sweeps; users can always run again.
• No repo mutation. All outputs go under .rag-eval/ in the target repo.
• When uncertain about best practices, do web research. Use tavily-search or firecrawl-research to pull current evidence, then synthesize into the audit report.
• Defer to the user. Before changing any file in the target repo, always confirm.