lebsral/dspy-bootstrap-rs

Optimize Few-Shot Demos with dspy.BootstrapFewShotWithRandomSearch

Guide the user through using DSPy's BootstrapFewShotWithRandomSearch optimizer to find the best set of few-shot demonstrations for their program. This optimizer runs BootstrapFewShot multiple times with different random seeds and keeps the candidate program that scores highest on a metric.

What it is

BootstrapFewShotWithRandomSearch (also known as BootstrapRS) is a prompt optimizer that searches over multiple candidate sets of few-shot demonstrations to find the best one. It wraps BootstrapFewShot and runs it repeatedly with different random subsets of training examples, then evaluates each candidate program on a held-out portion of the trainset.

trainset ──> [ BootstrapFewShot run 1 ] ──> candidate program 1 ──┐
         ──> [ BootstrapFewShot run 2 ] ──> candidate program 2 ──┤
         ──> [ BootstrapFewShot run 3 ] ──> candidate program 3 ──┼──> evaluate all ──> best program
         ──> ...                                                  │
         ──> [ BootstrapFewShot run N ] ──> candidate program N ──┘

How it improves on BootstrapFewShot

BootstrapFewShot runs once: it bootstraps demonstrations from your training data, picks a fixed set, and returns a single optimized program. The result depends heavily on which examples happened to be selected and which traces succeeded. You might get lucky or unlucky.

BootstrapFewShotWithRandomSearch removes that luck factor. It runs the bootstrap process multiple times (controlled by num_candidate_programs), each time with a different random sample of training examples. Each candidate program gets scored on a validation set, and the optimizer returns the highest-scoring one.

The trade-off is straightforward: more compute for more reliable results.

| | BootstrapFewShot | BootstrapFewShotWithRandomSearch | |---|-----------------|----------------------------------| | Bootstrap runs | 1 | num_candidate_programs (default 16) | | Selection | Returns the single result | Evaluates all candidates, returns the best | | Reliability | Results vary between runs | More consistent, higher-quality results | | Cost | 1x | ~Nx (N = num_candidate_programs) | | When to use | Quick iteration, <50 examples | You want the best few-shot demos, 50-200+ examples |

Basic usage

import dspy
from dspy.evaluate import Evaluate

lm = dspy.LM("openai/gpt-4o-mini")  # or any LiteLLM-supported provider
dspy.configure(lm=lm)

# 1. Define your program
qa = dspy.ChainOfThought("question -> answer")

# 2. Prepare training data (50-200+ examples recommended)
trainset = [
    dspy.Example(question="What is the capital of France?", answer="Paris").with_inputs("question"),
    dspy.Example(question="What is 2 + 2?", answer="4").with_inputs("question"),
    # ... more examples
]

# 3. Define a metric
def metric(example, prediction, trace=None):
    return prediction.answer.strip().lower() == example.answer.strip().lower()

# 4. Optimize with random search
optimizer = dspy.BootstrapFewShotWithRandomSearch(
    metric=metric,
    max_bootstrapped_demos=4,
    max_labeled_demos=4,
    num_candidate_programs=16,
)
optimized_qa = optimizer.compile(qa, trainset=trainset)

# 5. Use the optimized program
result = optimized_qa(question="What is the capital of Germany?")
print(result.answer)

# 6. Save for later
optimized_qa.save("optimized_qa.json")

Key parameters

dspy.BootstrapFewShotWithRandomSearch(
    metric,                          # Scoring function: (example, prediction, trace) -> float|bool
    max_bootstrapped_demos=4,        # Max demos generated by running the program on training examples
    max_labeled_demos=16,            # Max demos taken directly from labeled training data
    num_candidate_programs=16,       # How many random bootstrap runs to try
    num_threads=None,                # Threads for parallel evaluation of candidates
    stop_at_score=None,              # Early-stop if a candidate reaches this score
    metric_threshold=None,           # Min metric score for a bootstrapped demo to be kept
)

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | metric | Callable | required | Scoring function (example, prediction, trace=None) -> float\|bool | | max_bootstrapped_demos | int | 4 | Maximum bootstrapped (program-generated) demos per predictor | | max_labeled_demos | int | 16 | Maximum labeled (from trainset) demos per predictor | | num_candidate_programs | int | 16 | Number of random bootstrap attempts to evaluate | | num_threads | int \| None | None | Threads for evaluating candidates. Falls back to dspy.settings.num_threads. | | stop_at_score | float \| None | None | Early-stop search if a candidate reaches this score | | metric_threshold | float \| None | None | Minimum metric score for a bootstrapped demo to be included | | teacher_settings | dict \| None | None | LM config for the teacher model (e.g., {"lm": big_model}) | | max_rounds | int | 1 | Bootstrap rounds per candidate (>1 generates diverse traces at temperature=1.0) | | max_errors | int \| None | None | Error tolerance before aborting |

max_bootstrapped_demos vs max_labeled_demos

These two parameters control where demonstrations come from:

• Bootstrapped demos are generated by running your program on training examples and keeping the traces where the metric passes. These are powerful because they show the LM its own successful reasoning patterns, including intermediate steps like chain-of-thought reasoning.

• Labeled demos are taken directly from your training data as input-output pairs. They don't include intermediate reasoning steps, but they're reliable because they use your gold-standard answers.

The optimizer includes up to max_bootstrapped_demos bootstrapped demos plus up to max_labeled_demos labeled demos in each candidate program's prompt.

Guidance:

• Start with max_bootstrapped_demos=4, max_labeled_demos=4 for most tasks.
• Increase max_labeled_demos (up to 8-16) if you have high-quality labeled data and your model benefits from more examples.
• Increase max_bootstrapped_demos (up to 4-8) if your task involves chain-of-thought or multi-step reasoning where seeing worked examples helps.
• Keep the total number of demos reasonable -- too many demos bloat the prompt and can hurt performance or exceed context limits.

How random search works

Each candidate program is built by a separate BootstrapFewShot run. The randomness comes from:

1. Shuffled training data: Each run sees a different random ordering of training examples, so different examples get bootstrapped.
2. Different demo subsets: The random ordering means each candidate ends up with a different combination of bootstrapped and labeled demos.

After all candidate programs are generated, the optimizer evaluates each one on a validation set (a portion of your trainset that was held out). The candidate with the highest validation score wins.

This is conceptually similar to hyperparameter random search: instead of searching over learning rates or layer sizes, you're searching over which few-shot demos to include in the prompt.

Computational cost

The cost scales linearly with num_candidate_programs:

| num_candidate_programs | Approximate cost multiplier | When to use | |------------------------|----------------------------|-------------| | 4-8 | 4-8x base BootstrapFewShot | Quick search, limited budget | | 16 (default) | 16x | Good balance for most tasks | | 25-50 | 25-50x | Maximum quality, budget allows |

Each candidate program requires:

1. One BootstrapFewShot run (bootstrapping demos from trainset)
2. One evaluation pass over the validation set

Cost estimate: If a single BootstrapFewShot run costs ~$0.50, then 16 candidate programs costs ~$8. With a larger trainset or more expensive model, plan for $5-$20.

Tip: Start with num_candidate_programs=8 to get a quick sense of how much random search helps, then increase to 16 or 25 if the improvement justifies the cost.

When to use BootstrapFewShotWithRandomSearch

Use BootstrapFewShotWithRandomSearch when:

• You have 50-200+ training examples
• Basic BootstrapFewShot gives inconsistent results across runs
• You want better few-shot demos without optimizing instructions
• You have budget for 10-20x the cost of a single BootstrapFewShot run
• You want a solid middle ground between BootstrapFewShot and MIPROv2

Use BootstrapFewShot instead when:

• You have fewer than 50 examples
• You want the fastest possible optimization
• Budget is very tight
• You're just prototyping and will optimize more later

Use MIPROv2 instead when:

• You want to optimize instructions and demos together (BootstrapRS only optimizes demos)
• You have 200+ examples and budget for a thorough search
• You've already tried BootstrapRS and want to push further
• You want the best prompt optimization DSPy offers

Quick & cheap          Solid middle ground         Best quality
BootstrapFewShot  -->  BootstrapFewShotWithRS  -->  MIPROv2
~$0.50                 ~$5-20                       ~$5-50
Few-shot demos only    Few-shot demos (searched)    Instructions + few-shot demos
1 candidate            N candidates                 Bayesian optimization

Using a teacher model

Use a larger model to generate high-quality bootstrapped demos, then deploy with a cheaper student model:

teacher_lm = dspy.LM("openai/gpt-4o")  # or any LiteLLM-supported provider
student_lm = dspy.LM("openai/gpt-4o-mini")  # or any LiteLLM-supported provider

dspy.configure(lm=student_lm)

optimizer = dspy.BootstrapFewShotWithRandomSearch(
    metric=metric,
    max_bootstrapped_demos=4,
    num_candidate_programs=16,
    teacher_settings={"lm": teacher_lm},
)
optimized = optimizer.compile(my_program, trainset=trainset)
# optimized runs on student_lm but uses demos generated by teacher_lm

Early stopping with stop_at_score

Skip evaluating remaining candidates once a "good enough" program is found:

optimizer = dspy.BootstrapFewShotWithRandomSearch(
    metric=metric,
    num_candidate_programs=25,
    stop_at_score=95.0,  # stop as soon as a candidate scores >= 95%
)

This is useful when you set num_candidate_programs high but want to save cost if an early candidate is already excellent.

Passing an optimized program to further optimization

You can stack optimizers. Run BootstrapRS first to find great demos, then pass the result to MIPROv2 to refine instructions on top:

# Step 1: Find best demos with random search
bootstrap_optimizer = dspy.BootstrapFewShotWithRandomSearch(
    metric=metric,
    max_bootstrapped_demos=4,
    max_labeled_demos=4,
    num_candidate_programs=16,
)
bootstrapped = bootstrap_optimizer.compile(my_program, trainset=trainset)

# Step 2: Refine instructions with MIPROv2
mipro_optimizer = dspy.MIPROv2(metric=metric, auto="medium")
final = mipro_optimizer.compile(bootstrapped, trainset=trainset)

Gotchas

1. Claude uses the same data for training and validation. BootstrapRS evaluates candidates on a held-out validation set. If you pass all data as trainset without a separate valset, the optimizer splits it internally, but you get no control over the split. Pass valset explicitly for reproducible results: optimizer.compile(program, trainset=trainset, valset=devset).
2. Claude sets num_candidate_programs too low. With num_candidate_programs=3 the random search barely explores the space. The default of 16 is a good starting point. Fewer than 8 rarely finds materially better demos than plain BootstrapFewShot.
3. Claude sets max_labeled_demos=16 with multi-step pipelines. Each predictor in the pipeline gets up to max_labeled_demos + max_bootstrapped_demos demos. A 3-step pipeline with 16+4 demos per step = 60 demos total, which can blow past context limits. Use 2-4 demos per type for multi-step pipelines.
4. Claude forgets the candidate_programs attribute on the result. The optimized program has a candidate_programs attribute containing all scored candidates. This is useful for inspecting how much variance exists and whether more search would help.
5. Claude runs BootstrapRS with fewer than 50 training examples. With fewer than ~50 examples, the random search has too little data to meaningfully differentiate candidates. Use plain BootstrapFewShot instead, or collect more data.

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• BootstrapFewShot for the simpler single-run version -- see /ai-improving-accuracy
• MIPROv2 for instruction + demo optimization -- see /ai-improving-accuracy
• Evaluate for measuring quality with metrics and devsets -- see /dspy-evaluate
• Data handling for preparing training sets -- see /dspy-data
• Improving accuracy for the full optimization decision framework -- see /ai-improving-accuracy
• For worked examples, see examples.md
• Install /ai-do if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

Additional resources

• dspy.BootstrapFewShotWithRandomSearch API docs
• DSPy optimizer selection guide
• For constructor signatures and method reference, see reference.md
• For worked examples (QA optimization, multi-step pipeline), see examples.md