ADu2021/cure-coevolving-llm-testing

Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning

Core Concept

CURE introduces "mutual supervision"—a mechanism where code generators and unit test generators co-evolve through RL without requiring ground-truth solutions. Rather than static code-test pairs, the framework generates candidate code, evaluates it against generated tests, and optimizes both components based on test discriminative power. The innovation is theoretically grounded: reward precision (probability generated tests rank correct code above incorrect code) converges to 1 as more tests are generated, enabling self-supervised learning.

This eliminates dependency on expensive labeled datasets and achieves 5.3% improvement in one-shot accuracy and 9.0% in Best-of-N performance while reducing inference by 64.8% on long-CoT variants.

Architecture Overview

• Joint Optimization: Simultaneously trains code generators and test generators using shared reward signals
• Test Discriminativeness: Rewards unit tests based on their ability to distinguish correct from incorrect code solutions
• Theoretical Foundation: Derives individual-level rewards from first principles using reward precision convergence
• Long-CoT Efficiency: Response-length-guided reward transformations penalize verbosity while preserving solution quality
• Self-Supervision: Trained unit test generators transfer as reward models to other base models

Implementation

1. Reward Function Design: Ground test rewards in discriminative ability

# Pseudo-code for reward calculation
def compute_joint_rewards(candidate_codes, candidate_tests, ground_truth_solution):
    """
    Score code solutions and unit tests based on mutual evaluation.
    Tests rewarded by discriminative power; code by correctness and test agreement.
    """
    # Execute all code solutions against all tests
    execution_matrix = []  # [num_solutions, num_tests]

    for solution in candidate_codes:
        results = []
        for test in candidate_tests:
            try:
                passed = execute_test(test, solution)
                results.append(passed)
            except:
                results.append(False)
        execution_matrix.append(results)

    # Compute code rewards: correct solutions + those passing more tests
    code_rewards = []
    for i, solution in enumerate(candidate_codes):
        passes_ground_truth = solution_correct(solution, ground_truth_solution)
        num_passing_tests = sum(execution_matrix[i])

        code_reward = 1.0 if passes_ground_truth else 0.0
        code_reward += 0.1 * (num_passing_tests / len(candidate_tests))
        code_rewards.append(code_reward)

    # Compute test rewards: reward based on discriminative power
    # Higher reward if test separates correct from incorrect solutions
    test_rewards = []
    for j, test in enumerate(candidate_tests):
        test_results = [execution_matrix[i][j] for i in range(len(candidate_codes))]

        # Reward precision: probability test ranks correct above incorrect
        correct_indices = [i for i, code in enumerate(candidate_codes)
                          if solution_correct(code, ground_truth_solution)]
        incorrect_indices = [i for i, code in enumerate(candidate_codes)
                            if not solution_correct(code, ground_truth_solution)]

        discrimination_score = 0.0
        if correct_indices and incorrect_indices:
            correct_pass_rate = sum(test_results[i] for i in correct_indices) / len(correct_indices)
            incorrect_pass_rate = sum(test_results[i] for i in incorrect_indices) / len(incorrect_indices)
            discrimination_score = correct_pass_rate - incorrect_pass_rate

        test_rewards.append(max(0.0, discrimination_score))

    return code_rewards, test_rewards

2. Long-CoT Response Length Penalty: Prevent excessive verbosity in reasoning chains

def apply_length_penalty(reward, response_tokens, target_length=None):
    """
    Penalize overly long responses while preserving correctness signal.
    Useful for chain-of-thought models.
    """
    if target_length is None:
        target_length = 512  # Typical reasoning length

    if response_tokens > target_length:
        # Exponential penalty for excessive length
        excess = response_tokens - target_length
        length_penalty = 1.0 - min(0.5, excess / (2 * target_length))
    else:
        # Slight bonus for conciseness
        length_penalty = 1.0 + 0.05 * (1 - response_tokens / target_length)

    adjusted_reward = reward * length_penalty
    return max(0.0, adjusted_reward)

3. Training Loop: GRPO-based optimization with dual rollouts

def training_step(coder, tester, task_batch, optimizer_code, optimizer_test):
    """
    Single training iteration co-optimizing code and test generators.
    Generates multiple rollouts per task for stable estimates.
    """
    total_code_loss = 0.0
    total_test_loss = 0.0

    for task in task_batch:
        # Generate candidate solutions and tests
        code_rollouts = coder.sample(task, num_samples=16, temperature=1.0)
        test_rollouts = tester.sample(task, num_samples=16, temperature=1.0)

        # Compute mutual rewards
        code_rewards, test_rewards = compute_joint_rewards(
            code_rollouts, test_rollouts, task['solution']
        )

        # GRPO losses: relative policy optimization
        code_loss = grpo_loss(coder, code_rollouts, code_rewards)
        test_loss = grpo_loss(tester, test_rollouts, test_rewards)

        # Apply KL penalty for stability
        code_loss += 0.01 * kl_divergence(coder, reference_model)
        test_loss += 0.01 * kl_divergence(tester, reference_model)

        total_code_loss += code_loss
        total_test_loss += test_loss

    # Update both models
    optimizer_code.zero_grad()
    total_code_loss.backward()
    optimizer_code.step()

    optimizer_test.zero_grad()
    total_test_loss.backward()
    optimizer_test.step()

    return total_code_loss.item(), total_test_loss.item()

4. Configuration: Recommended hyperparameters for stable training
   • Batch size: 16 code rollouts, 16 test rollouts per task
   • Learning rate: 1×10⁻⁶ with KL coefficient 0.01
   • Training steps: 350 for standard models, 50 for long-CoT variants
   • Temperature: 1.0 for diverse rollouts

Practical Guidance

When to Apply:

• Training code generation models without expensive labeled test suites
• Need to improve both code quality and test effectiveness simultaneously
• Working with long-CoT models where verbosity is problematic

Setup Requirements:

• Base LLM supporting code generation (Qwen2.5 7B+, Llama 3.1+)
• Code execution sandbox for safe test evaluation
• Multiple coding benchmarks for validation

Expected Results:

• One-shot accuracy improvement: 5.3% over base models
• Best-of-N performance: 9.0% improvement
• Inference efficiency: 64.8% reduction on 4B long-CoT variants
• Test quality: Discriminative power improved through RL signal

Key Tuning Decisions:

• Number of rollouts: Balance between sample efficiency and gradient stability (16 typically good)
• KL coefficient: Higher values (0.05) prevent distribution drift, lower (0.005) allow faster improvement
• Long-CoT penalty: Adjust target_length based on task complexity (512-2048 typical range)
• Reference model: Pre-trained code LLM provides better KL anchoring than random initialization

Transferability:

• Trained unit test generator transfers to other base models as reward model
• Test reward signal validates across model architectures
• Code patterns learned may be model-specific—re-train tester for new base models

Reference

Implemented on Qwen2.5 (7B, 14B) and Qwen3-4B. Evaluated across LiveBench, MBPP, LiveCodeBench, CodeContests, and CodeForces. Training uses 16 A100 GPUs with GRPO algorithm and mutual supervision. Demonstrates label-free optimization without dependency on proprietary model outputs.