mathews-tom/rag-auditor

RAG Auditor

Systematic RAG pipeline evaluation across the full retrieval-generation chain: designs evaluation query sets, measures retrieval metrics (Precision@K, Recall@K, MRR), evaluates generation quality (groundedness, completeness, hallucination rate), diagnoses component-level failures, and recommends targeted improvements.

Reference Files

| File | Contents | Load When | | ---------------------------------- | -------------------------------------------------------------------------- | ---------------------------- | | references/retrieval-metrics.md | Precision@K, Recall@K, MRR, NDCG definitions and calculation | Always | | references/generation-metrics.md | Groundedness, completeness, hallucination detection methods | Generation evaluation needed | | references/failure-taxonomy.md | RAG failure categories: retrieval, generation, chunking, embedding | Failure diagnosis needed | | references/diagnostic-queries.md | Designing evaluation query sets, known-answer questions, difficulty levels | Evaluation setup |

Prerequisites

• Access to the RAG pipeline (or its outputs for post-hoc evaluation)
• A set of test queries with known-correct answers
• Understanding of the pipeline components (embedding model, retriever, generator)

Workflow

Phase 1: Pipeline Inventory

Document the RAG pipeline configuration:

1. Document source — What documents are indexed? Format, count, size.
2. Chunking — Strategy (fixed-size, semantic, paragraph), chunk size, overlap.
3. Embedding — Model name and version, dimensionality.
4. Vector store — Type (FAISS, Pinecone, Chroma, pgvector), index type.
5. Retrieval — Method (similarity, hybrid, reranking), top-K parameter.
6. Generation — Model, prompt template, context window usage.

Phase 2: Design Evaluation Queries

Create a diverse set of test queries:

| Query Type | Purpose | Count | | ---------------------- | ------------------------------------------- | ----- | | Known-answer (factoid) | Measure retrieval + generation accuracy | 10+ | | Multi-hop | Require combining info from multiple chunks | 5+ | | Unanswerable | Not in the corpus — should abstain | 3+ | | Ambiguous | Multiple valid interpretations | 3+ | | Recent/updated | Test freshness | 2+ |

For each query, document the expected answer and the source chunk(s).

Phase 3: Evaluate Retrieval

For each test query, measure:

1. Precision@K — Of the K retrieved chunks, how many are relevant?
2. Recall@K — Of all relevant chunks in the corpus, how many were retrieved?
3. MRR (Mean Reciprocal Rank) — How high is the first relevant chunk ranked?
4. Chunk relevance — Score each retrieved chunk: Relevant, Partially Relevant, Irrelevant.

Phase 4: Evaluate Generation

For each test query with retrieved context:

1. Groundedness — Is every claim in the response supported by the retrieved context? Score: 0 (hallucinated) to 1 (fully grounded).
2. Completeness — Does the response use all relevant information from the context? Score: 0 (ignored context) to 1 (complete).
3. Hallucination detection — Identify specific claims not supported by context.
4. Abstention — For unanswerable queries, does the model correctly say "I don't know"?

Phase 5: Diagnose Failures

For every incorrect or low-quality response, classify the root cause:

| Failure Type | Diagnosis | Indicator | | -------------------- | -------------------------------------------------- | ---------------------------------------- | | Retrieval failure | Relevant chunks not retrieved | Low Recall@K | | Ranking failure | Relevant chunk retrieved but ranked low | Low MRR, high Recall | | Chunk boundary issue | Answer split across chunk boundaries | Partial matches in multiple chunks | | Embedding mismatch | Query semantics don't match chunk embeddings | Relevant chunk has low similarity score | | Generation failure | Correct context but wrong answer | High retrieval scores, low groundedness | | Hallucination | Model invents facts not in context | Claims not traceable to any chunk | | Over-abstention | Model refuses to answer when context is sufficient | Unanswered with relevant context present |

Phase 6: Recommendations

Based on failure analysis, recommend specific improvements:

| Failure Pattern | Recommendation | | --------------------- | -------------------------------------------------------------- | | Chunk boundary issues | Increase overlap, try semantic chunking | | Low Precision@K | Reduce K, add reranking stage | | Low Recall@K | Increase K, try hybrid search | | Embedding mismatch | Try different embedding model, add query expansion | | Hallucination | Strengthen grounding instruction in prompt, reduce temperature | | Over-abstention | Soften abstention criteria in prompt |

Output Format

## RAG Audit Report

### Pipeline Configuration
| Component | Value |
|-----------|-------|
| Documents | {N} ({format}) |
| Chunking | {strategy}, {size} tokens, {overlap}% overlap |
| Embedding | {model} ({dimensions}d) |
| Retrieval | {method}, K={N} |
| Generation | {model}, temperature={T} |

### Evaluation Dataset
- **Total queries:** {N}
- **Known-answer:** {N}
- **Multi-hop:** {N}
- **Unanswerable:** {N}

### Retrieval Quality

| Metric | Score | Target | Status |
|--------|-------|--------|--------|
| Precision@{K} | {score} | {target} | {Pass/Fail} |
| Recall@{K} | {score} | {target} | {Pass/Fail} |
| MRR | {score} | {target} | {Pass/Fail} |

### Generation Quality

| Metric | Score | Target | Status |
|--------|-------|--------|--------|
| Groundedness | {score} | {target} | {Pass/Fail} |
| Completeness | {score} | {target} | {Pass/Fail} |
| Hallucination rate | {score} | {target} | {Pass/Fail} |
| Abstention accuracy | {score} | {target} | {Pass/Fail} |

### Failure Analysis

| # | Query | Failure Type | Root Cause | Recommendation |
|---|-------|-------------|------------|----------------|
| 1 | {query} | {type} | {cause} | {fix} |

### Recommendations (Priority Order)
1. **{Recommendation}** — addresses {N} failures, expected impact: {description}
2. **{Recommendation}** — addresses {N} failures, expected impact: {description}

### Sample Failures

#### Query: "{query}"
- **Expected:** {answer}
- **Retrieved chunks:** {chunk summaries with relevance scores}
- **Generated:** {response}
- **Issue:** {diagnosis}

Calibration Rules

1. Component isolation. Evaluate retrieval and generation independently. A great retriever with a bad generator looks like retrieval failure if you only check end output.
2. Known answers first. Start with factoid questions where the correct answer is unambiguous. Multi-hop and ambiguous queries are harder to evaluate.
3. Quantify, don't qualify. "Retrieval is bad" is not a finding. "Precision@5 is 0.3 (target: 0.8) with 70% of failures due to chunk boundary splits" is actionable.
4. Sample failures deeply. Aggregate metrics identify WHERE the problem is. Individual failure analysis identifies WHY.

Error Handling

| Problem | Resolution | | --------------------------------- | ------------------------------------------------------------------------------------------------------- | | No known-answer queries available | Help design them from the document corpus. Pick 10 facts and formulate questions. | | Pipeline access not available | Work from recorded inputs/outputs. Post-hoc evaluation is possible with query-context-response triples. | | Corpus is too large to review | Sample-based evaluation. Select representative documents and generate queries from them. | | Multiple failure types co-exist | Address retrieval failures first. Generation quality cannot exceed retrieval quality. |

When NOT to Audit

Push back if:

• The pipeline hasn't been built yet — design it first, audit after
• The corpus has fewer than 10 documents — too small for meaningful retrieval evaluation
• The user wants to compare embedding models — that's a benchmark task, not an audit