curiositech/khattab-2023-dspy

DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines

Decision Points

When to Choose Optimization Strategy

Input: LLM pipeline with N modules, training data size T, target metric M
├── T < 50 examples AND N ≤ 2 modules
│   └── Use manual prompt engineering → DSPy overhead not justified
├── T ≥ 50 examples AND no labeled intermediate steps
│   ├── N ≤ 3 modules → Use BootstrapFewShot teleprompter
│   └── N > 3 modules → Use BootstrapFewShotWithRandomSearch
├── T ≥ 200 examples AND have some labeled intermediates
│   ├── Simple accuracy metric → Use MIPRO (optimizes instructions + demos)
│   └── Complex composite metric → Use BootstrapFinetune if budget allows
└── T ≥ 1000 examples AND production system
    └── Use ensemble of compiled programs → vote/rank outputs

Module Design Decision Tree

Need: Transform input X to output Y
├── Single reasoning step (X → Y)
│   └── Use Predict(signature="X -> Y") 
├── Multi-step reasoning required
│   ├── Steps are sequential (A→B→C→Y) → Use ChainOfThought("X -> reasoning, Y")
│   ├── Steps need external tools → Use ReAct("X -> thought, action, observation, Y")
│   └── Need confidence estimation → Use majority vote ensemble
├── Retrieval + generation needed
│   ├── Simple QA → Use Retrieve(k=3) + Generate("context, question -> answer")
│   └── Complex reasoning over docs → Use ColBERTv2 + ChainOfThought
└── Multiple valid approaches exist
    └── Use ProgramOfThought for mathematical reasoning

Signature Complexity Guidelines

Field count in signature:
├── 1-2 fields → Start here, expand only if metrics plateau
├── 3-4 fields → Acceptable for complex tasks, monitor token usage
├── 5+ fields → Likely over-specified, consider decomposition
└── If adding fields doesn't improve validation metrics → Remove them

Failure Modes

1. Metric Misalignment Trap

Symptom: High training accuracy, poor real-world performance
Detection: training_metric >> validation_metric OR users report "system gives perfect but useless answers"
Root cause: Metric optimizes for wrong objective (exact string match vs semantic correctness)
Fix: Redesign metric to capture actual user success criteria, recompile with composite metrics

2. Bootstrapping Collapse

Symptom: Compiler generates repetitive or nonsensical demonstrations
Detection: len(set(demonstrations)) < 0.3 * len(demonstrations) OR average_demo_length < 5 tokens
Root cause: Training set too narrow, causing trace filtering to select degenerate examples
Fix: Expand training diversity, lower trace filtering threshold, or add minimum quality constraints

3. Signature Explosion

Symptom: Modules produce malformed outputs, compilation time increases exponentially
Detection: signature_fields > 5 OR parsing_errors > 20% OR compilation_time > 2x baseline
Root cause: Over-specified signatures make reliable generation harder
Fix: Decompose into simpler modules or merge related fields (reasoning_step_1, reasoning_step_2 → reasoning)

4. Manual Prompt Injection

Symptom: Performance doesn't improve with compilation, prompts contain hard-coded instructions
Detection: str("Let's think step by step") in module.forward() OR instructions manually set in __init__
Root cause: Bypassing DSPy abstraction by injecting imperative prompts
Fix: Remove hard-coded strings, let compiler optimize instructions, trust the abstraction

5. Validation Leak

Symptom: Perfect compiled performance that doesn't generalize to deployment
Detection: compiled_accuracy = 100% OR validation_set overlaps training_set
Root cause: Compiler overfit to training traces, no held-out validation
Fix: Create true validation split, use cross-validation during compilation, monitor test metrics

Worked Examples

Example 1: Multi-Hop QA System

Scenario: Build a system that answers questions requiring 2-3 reasoning hops over a knowledge base.

Initial Manual Approach (what novices do):

# Brittle manual prompting
prompt = f"Given context: {context}\nQuestion: {question}\nLet's think step by step:\n1. First, I need to..."

DSPy Expert Approach:

# 1. Define signature (interface, not implementation)
class MultiHopQA(dspy.Module):
    def __init__(self):
        self.retrieve = dspy.Retrieve(k=5)
        self.reason = dspy.ChainOfThought("context, question -> reasoning, answer")
    
    def forward(self, question):
        contexts = self.retrieve(question)
        return self.reason(context=contexts, question=question)

# 2. Set up compilation
train_set = [...] # 100 question/answer pairs
metric = lambda example, prediction: example.answer.lower() in prediction.answer.lower()

# 3. Compile (this is where the magic happens)
teleprompter = BootstrapFewShot(metric=metric, max_bootstrapped_demos=8)
compiled_qa = teleprompter.compile(MultiHopQA(), trainset=train_set)

Key Decision Points Navigated:

• Used ChainOfThought signature because multi-hop requires explicit reasoning
• Chose BootstrapFewShot because no intermediate reasoning labels available
• Simple contains-answer metric appropriate for factual QA

What Expert Catches vs Novice Misses:

• Expert: Signature drives both training and inference behavior consistently
• Novice: Manually writes "step 1, step 2" prompts that vary by hand
• Expert: Bootstrapping finds task-specific demonstrations from successful traces
• Novice: Uses generic few-shot examples that may not match pipeline context

Example 2: Document Summarization with Constraints

Scenario: Summarize technical papers with length/style constraints while preserving key findings.

DSPy Implementation with Trade-offs:

class ConstrainedSummarizer(dspy.Module):
    def __init__(self):
        # Trade-off: More fields = better control but harder optimization
        self.summarize = dspy.ChainOfThought(
            "document, style_guide, max_words -> key_findings, summary"
        )
    
    def forward(self, document, style_guide="academic", max_words=150):
        result = self.summarize(
            document=document, 
            style_guide=style_guide,
            max_words=max_words
        )
        return result

# Composite metric balancing multiple objectives
def summary_quality(example, prediction):
    word_count = len(prediction.summary.split())
    length_ok = word_count <= example.max_words * 1.1  # 10% tolerance
    
    # Trade-off: More sophisticated metrics = better quality but slower compilation
    key_present = any(finding in prediction.summary for finding in example.key_findings)
    
    return length_ok and key_present

# Compilation decision: MIPRO optimizes both demonstrations AND instructions
teleprompter = MIPRO(metric=summary_quality, num_candidates=10)

Trade-offs Shown:

• 3-field signature enables fine control but may slow compilation
• Composite metric ensures length constraints but adds complexity
• MIPRO teleprompter improves quality but requires more compute than BootstrapFewShot

Quality Gates

Validation checklist for DSPy pipeline deployment:

• [ ] Signature validation: All module signatures parse correctly on 100% of validation inputs
• [ ] Metric plateau detection: Compilation metric improvement < 2% over last 3 iterations
• [ ] Cross-model consistency: Compiled program achieves >80% of best performance when swapped to different LM
• [ ] Demonstration quality: Manual inspection shows ≥80% of bootstrapped examples are sensible
• [ ] Validation holdout: Validation set completely separate from compilation training data
• [ ] Latency constraint: 95th percentile response time meets production SLA requirements
• [ ] Cost constraint: Token usage per request within acceptable budget limits
• [ ] Failure graceful: Pipeline handles malformed inputs without crashing (try 10 edge cases)
• [ ] Metric alignment: Manual evaluation of 50 outputs confirms metric captures real quality
• [ ] Overfit detection: Validation performance within 10% of training performance

NOT-FOR Boundaries

Do NOT use DSPy for:

• Single LM calls with abundant labeled examples → Use standard fine-tuning instead
• Real-time systems requiring <100ms response → Use cached/pre-computed approaches
• Tasks where prompt content is legally regulated → Use verified manual prompts instead
• Simple classification with clear training data → Use traditional ML classifiers instead
• One-off queries or prototypes → Use manual prompting, DSPy overhead not justified

Delegate to other skills:

• For training custom models → Use model-fine-tuning skill instead
• For retrieval system design → Use rag-architecture skill instead
• For LM serving optimization → Use inference-optimization skill instead
• For prompt security → Use prompt-injection-defense skill instead
• For LM evaluation methodology → Use llm-evaluation-frameworks skill instead

DSPy is for: Multi-module pipelines, automatic optimization, model-agnostic systems, compositional reasoning chains, and when you need programs that improve from their own execution traces.