ADu2021/deep-search-mcts-rlvr-training

DeepSearch: MCTS Integration During RLVR Training

DeepSearch addresses training plateaus in reasoning models by integrating MCTS directly into the RL training loop. Rather than waiting for standard RLVR to explore, structured search during training accelerates discovery of correct solutions, achieving superior results with 5.7x fewer GPU hours.

Core Architecture

• Training-time search: MCTS during training, not just inference
• Global frontier selection: Identify most promising incomplete solution paths
• Entropy-guided sampling: Target confident errors for maximal learning signal
• Asymmetric Q-value updates: Differentiate correct/incorrect trajectories
• Replay buffer optimization: Prioritize valuable training experiences

Implementation Steps

Setup MCTS-augmented RLVR trainer:

# Initialize DeepSearch training framework
from deepsearch import MCTSTrainer, MCTSConfig, FrontierSelector

mcts_config = MCTSConfig(
    num_simulations=100,
    max_depth=50,
    exploration_constant=1.414,
    temperature=1.0
)

frontier_selector = FrontierSelector(
    selection_strategy="global_frontier",
    num_frontier_nodes=50,
    confidence_threshold=0.5
)

trainer = MCTSTrainer(
    model=your_reasoning_llm,
    verifier=your_verifier,
    mcts_config=mcts_config,
    frontier_selector=frontier_selector,
    algorithm="GRPO"
)

Execute MCTS-augmented training:

# Training loop with integrated MCTS
for epoch in range(num_epochs):
    for batch in training_dataloader:
        prompts = batch["prompt"]

        # Stage 1: MCTS exploration during training
        search_trees = []
        for prompt in prompts:
            # Run MCTS from this prompt
            tree = trainer.mcts.search(
                root_prompt=prompt,
                num_simulations=mcts_config.num_simulations,
                temperature=mcts_config.temperature
            )
            search_trees.append(tree)

        # Stage 2: Global frontier selection across all search trees
        frontier_nodes = frontier_selector.select(
            trees=search_trees,
            num_nodes=len(prompts) * 5  # 5 frontier nodes per prompt
        )

        # Stage 3: Entropy-guided sampling from frontier
        # Prioritize confident errors (high model confidence but wrong answer)
        sampled_trajectories = []
        for node in frontier_nodes:
            trajectory = trainer.trajectory_from_node(node)

            # Compute entropy/confidence
            entropy = compute_entropy(node.model_confidence)

            # Entropy-guided selection: confident but incorrect
            if node.is_error and entropy < entropy_threshold:
                sampled_trajectories.append(trajectory)

        # Stage 4: Compute rewards and asymmetric updates
        rewards = trainer.verifier.evaluate(sampled_trajectories)

        # Asymmetric Q-value updates
        for trajectory, reward in zip(sampled_trajectories, rewards):
            if reward > 0:  # Correct solution
                # Strong positive signal
                q_value = 1.0 + bonus_correct
            else:  # Incorrect solution
                # Negative signal for learning
                q_value = -1.0 + penalty_entropy * (entropy / max_entropy)

        # Stage 5: RLVR policy update on selected trajectories
        loss = trainer.compute_grpo_loss(
            trajectories=sampled_trajectories,
            rewards=rewards,
            q_values=q_values
        )

        # Backward pass
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

        # Stage 6: Update replay buffer with MCTS trajectories
        for trajectory, reward in zip(sampled_trajectories, rewards):
            trainer.replay_buffer.add(
                trajectory=trajectory,
                reward=reward,
                source="mcts_training"
            )

        # Logging
        if epoch % 10 == 0:
            success_rate = (torch.tensor(rewards) > 0.5).float().mean()
            print(f"Epoch {epoch}: Success={success_rate:.1%}")

Practical Guidance

When to use DeepSearch:

• Reasoning models plateauing under standard RLVR
• Verification signals available (MCTS requires reward feedback)
• Sufficient compute for training-time search overhead
• Complex reasoning domains (math, theorem proving, code)

When NOT to use:

• Non-verifiable tasks (MCTS requires ground truth)
• Compute-constrained settings (search overhead significant)
• Real-time training required (MCTS adds wall-clock latency)
• Domains where standard RLVR already converges well

Hyperparameters:

• Num simulations (100): Increase to 200 for deeper exploration; 50 for speed
• Max depth (50): Solution length limit; adjust to problem complexity
• Num frontier nodes (50 per prompt): More nodes explore better; increase to 100 for small batches
• Entropy threshold: Tune to focus on confident errors
• Correct bonus: Weight for correct solutions (1.0 + 0.5 typical)
• Error penalty: Scale by entropy (prioritize high-confidence mistakes)

Search Strategy

Global frontier selection:

• Identifies most promising incomplete paths across all search trees
• Prioritizes paths where model confident but yet to be verified
• Avoids redundant exploration of already-successful paths

Entropy guidance:

• Low entropy: High model confidence
• High entropy: Uncertain predictions
• Sweet spot: High confidence + incorrect (most informative for learning)

Performance Metrics

• 5.7x GPU-hour reduction: Compared to standard RLVR
• Performance improvement: 62.95% vs 61.70% baseline (higher-quality solutions)
• Convergence speed: Reaches plateau 3x faster
• Final accuracy: Continues improving beyond standard RLVR plateau

Computational Cost

Training overhead:

• MCTS simulation: ~30% of training time
• Frontier selection: ~5% overhead
• Asymmetric updates: <1% overhead
• Total: ~35% overhead partially amortized by better solutions

GPU hour reduction:

• 5.7x reduction suggests marginal cost is negative
• Faster convergence outweighs per-step overhead

Replay Buffer Integration

Trajectories discovered via MCTS stored and replayed:

• Prioritize MCTS-discovered solutions (high value)
• Mix with standard RLVR trajectories for diversity
• Enable transfer to other reasoning tasks

Architecture Notes

Key insight: "MCTS during training acts as oracle exploration, revealing high-value regions policy might miss." Unlike inference-only MCTS (expensive, deployment-time), training-time MCTS can be amortized during learning.

Comparison to Standard RLVR

| Aspect | Standard RLVR | DeepSearch | |--------|---------------|-----------| | Exploration | Random sampling | Structured MCTS | | Training time | Baseline | +35% per-step | | Convergence | 3K steps plateau | Continues beyond 5K | | GPU hours to 62.95% | ~1000 | ~175 | | Final performance | 61.70% | 62.95% |

References

Builds on MCTS theory, curriculum learning, and verifiable reward signals for RL.