lebsral/dspy-best-of-n

Pick the Best Output with dspy.BestOfN

Guide the user through using DSPy's BestOfN module to run a program multiple times and keep the highest-scoring result. This is rejection sampling -- generate N candidates, score each one, return the winner.

What is BestOfN

dspy.BestOfN wraps any DSPy module and calls it up to N times with temperature=1.0 (each attempt uses a different rollout ID to get diverse outputs). A reward function scores every result, and BestOfN returns the single best prediction.

If any attempt hits a score threshold you set, execution stops early -- no need to burn through all N attempts when you already have a great result.

Your module ──> Run N times ──> Score each with reward_fn ──> Return best

When to use BestOfN

• You have a cheap, fast metric that can score outputs automatically (test suite passes, regex match, word count check, etc.)
• Quality variance is high -- the same prompt sometimes produces great output and sometimes doesn't
• You'd rather spend tokens than engineering time -- BestOfN is the simplest way to boost quality without optimization
• You need a quick quality boost before investing in full prompt optimization with MIPROv2 or BootstrapFewShot

Do not use BestOfN when:

• You have no way to automatically score outputs (you need a metric)
• Latency matters more than quality (N calls take N times longer, unless you can parallelize)
• Cost is a hard constraint and N is large

Basic usage

import dspy

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

# 1. Define your module
qa = dspy.ChainOfThought("question -> answer")

# 2. Define a reward function
def short_answer(args, pred):
    """Prefer concise single-word answers."""
    return 1.0 if len(pred.answer.split()) == 1 else 0.0

# 3. Wrap with BestOfN
best_qa = dspy.BestOfN(
    module=qa,
    N=3,
    reward_fn=short_answer,
    threshold=1.0,
)

# 4. Call it like any module
result = best_qa(question="What is the capital of Belgium?")
print(result.answer)

Constructor parameters

dspy.BestOfN(
    module,       # Any dspy.Module to run repeatedly
    N,            # Number of attempts (int)
    reward_fn,    # Scoring function: (args_dict, prediction) -> float
    threshold,    # Early-stop threshold: stop as soon as a score >= threshold
    fail_count=None,  # Max failures before raising an error (defaults to N)
)

| Parameter | Type | Description | |-----------|------|-------------| | module | dspy.Module | The module to run N times | | N | int | Maximum number of attempts | | reward_fn | Callable[[dict, Prediction], float] | Scores each prediction; higher is better | | threshold | float | If any attempt scores >= this value, return immediately | | fail_count | int \| None | How many attempts can fail (raise exceptions) before BestOfN itself raises. Defaults to N (all can fail before error) |

The reward function

The reward function is the core of BestOfN. It receives two arguments:

def reward_fn(args: dict, prediction: dspy.Prediction) -> float:
    # args: the keyword arguments you passed to the BestOfN call
    # prediction: the output from one attempt of the wrapped module
    # Return: a scalar score (higher = better)
    ...

Key differences from a dspy.Evaluate metric:

• Signature: (args_dict, prediction) not (example, prediction, trace)
• No gold labels: args contains only the inputs you passed, not expected outputs
• No trace parameter: BestOfN doesn't use traces

Reward function examples

Binary pass/fail:

def passes_tests(args, pred):
    """Score 1.0 if generated code passes all tests, 0.0 otherwise."""
    try:
        exec(pred.code)
        return 1.0
    except Exception:
        return 0.0

Graded score:

def quality_score(args, pred):
    """Score summaries on length and keyword coverage."""
    score = 0.0
    # Prefer summaries under 100 words
    if len(pred.summary.split()) <= 100:
        score += 0.5
    # Reward covering key topics
    keywords = ["revenue", "growth", "forecast"]
    covered = sum(1 for kw in keywords if kw in pred.summary.lower())
    score += 0.5 * (covered / len(keywords))
    return score

Using an LM as judge inside the reward:

class JudgeQuality(dspy.Signature):
    """Rate the answer quality from 0.0 to 1.0."""
    question: str = dspy.InputField()
    answer: str = dspy.InputField()
    score: float = dspy.OutputField(desc="Quality score from 0.0 to 1.0")

judge = dspy.Predict(JudgeQuality)

def llm_reward(args, pred):
    result = judge(question=args["question"], answer=pred.answer)
    return result.score

Note: Using an LM as judge inside the reward function costs additional tokens per attempt. Reserve this for cases where programmatic scoring isn't feasible.

Tuning N

| N | Trade-off | |---|-----------| | 2-3 | Low cost, modest quality gain. Good starting point. | | 5 | Solid improvement for tasks with high variance. Sweet spot for most uses. | | 10+ | Diminishing returns unless your metric is very selective (e.g., <10% pass rate). |

Rule of thumb: if your base module succeeds ~50% of the time, N=3 gives you a ~87.5% chance of at least one success. If it succeeds ~20% of the time, you need N=8 for ~83%.

The math: probability of at least one success in N tries = 1 - (1 - p)^N where p is the single-attempt success rate.

How selection works internally

1. BestOfN calls your module with temperature=1.0 and a unique rollout ID for each attempt
2. Each attempt produces a dspy.Prediction
3. The reward function scores the prediction
4. If the score >= threshold, return immediately (early stopping)
5. If the attempt raises an exception, increment the failure counter
6. After all N attempts (or early stopping), return the prediction with the highest score
7. If failures exceed fail_count, raise an exception

The unique rollout IDs ensure the LM produces diverse outputs even with the same input. Temperature is fixed at 1.0 to maximize diversity.

Cost considerations

BestOfN multiplies your token usage by up to N times (fewer if early stopping kicks in). Budget accordingly:

| Base cost per call | N | Max cost | |--------------------|---|----------| | $0.01 | 3 | $0.03 | | $0.01 | 5 | $0.05 | | $0.01 | 10 | $0.10 |

Ways to manage cost:

• Set a tight threshold so good results stop early (often after 1-2 attempts)
• Use a cheap model as the base module and a stronger model only for the reward function
• Start with N=3 and increase only if your metric shows it helps
• Use programmatic reward functions (regex, test execution, length checks) instead of LM-based judges to avoid extra LM calls per attempt

BestOfN vs MultiChainComparison

Both BestOfN and dspy.MultiChainComparison aim to pick the best output from multiple candidates, but they work differently:

| | BestOfN | MultiChainComparison | |---|---------|---------------------| | Selection method | Your reward function scores each candidate | An LM reads all candidates and picks the best | | Metric required | Yes -- you must provide a reward_fn | No -- the LM decides what "best" means | | Token cost | N calls to your module (+ reward fn) | Multiple chain calls + one comparison call | | Best when | You have a clear, automatable scoring criterion | Quality is subjective or hard to score programmatically | | Optimizable | The wrapped module can be optimized | The comparison module can be optimized |

Use BestOfN when you can write a reward function. Use MultiChainComparison when you want the LM to judge quality using its own understanding.

Combining BestOfN with optimization

BestOfN works well as a complement to DSPy optimizers. Optimize your module first, then wrap the optimized version with BestOfN for an additional quality boost:

# Optimize the base module
optimizer = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
optimized_qa = optimizer.compile(qa, trainset=trainset)

# Wrap the optimized module with BestOfN
best_qa = dspy.BestOfN(
    module=optimized_qa,
    N=3,
    reward_fn=my_reward,
    threshold=1.0,
)

This stacks two quality improvements: better prompts from the optimizer, and rejection sampling from BestOfN.

Gotchas

• Claude writes the reward function with (example, prediction, trace=None) signature. BestOfN reward functions take (args_dict, prediction), not the (example, prediction, trace) signature used by dspy.Evaluate metrics. The args dict contains only the inputs you passed to the call, not labeled examples with gold outputs.
• Claude sets N too high without considering cost. Each attempt is a full LM call at temperature=1.0. N=10 means 10x the token cost. Start with N=3 and increase only if your metric shows improvement — diminishing returns kick in quickly above N=5.
• Claude uses BestOfN when the reward function is as expensive as the module itself. If your reward function calls an LM (e.g., LM-as-judge), each BestOfN attempt costs 2x tokens (one for the module, one for the judge). For N=5, that is 10 LM calls total. Use programmatic reward functions (test execution, regex, length checks) whenever possible.
• Claude forgets to set threshold to enable early stopping. Without a meaningful threshold, BestOfN always runs all N attempts even when the first one is perfect. Set threshold to a value that represents "good enough" (e.g., 1.0 for binary pass/fail, 0.9 for graded metrics) to save tokens on easy inputs.
• Claude wraps an already-optimized module but does not evaluate the incremental gain. BestOfN on top of an optimized module costs N times more per call at inference time. Always measure the quality gain from BestOfN separately to confirm the extra cost is justified — if the optimized module already hits 95%+, BestOfN may not add enough to be worth it.

Additional resources

• dspy.BestOfN API docs
• reference.md — constructor parameters, forward() method, key behaviors
• examples.md — code generation with test-based selection, summarization with graded metric

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• MultiChainComparison for LM-based candidate selection -- see /dspy-multi-chain-comparison
• Evaluate for measuring quality with metrics and devsets -- see /dspy-evaluate
• Improving accuracy for the full optimization workflow -- see /ai-improving-accuracy
• Install /ai-do if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do