ADu2021/dupo-dual-preference-optimization

DuPO: Dual Preference Optimization for Reliable LLM Self-Verification

Core Concept

DuPO enables large language models to self-verify outputs without manual feedback by leveraging dual learning frameworks. The technique decomposes task inputs into known and unknown components, then constructs complementary reconstruction tasks. The model's ability to reconstruct hidden information from its primary output serves as an intrinsic reward signal. This approach eliminates dependency on expensive labeled data while improving reasoning quality across mathematical, translation, and general reasoning domains.

Architecture Overview

• Input Decomposition: Split task inputs into observable and hidden components
• Dual Task Construction: Create complementary reconstruction tasks (e.g., reverse engineering solutions to recover variables)
• Reconstruction Quality Scoring: Use reconstruction fidelity as self-supervised reward signal
• Preference Optimization: Apply reward signals to optimize model outputs via preference learning
• Non-Invertible Handling: Extend framework to problems without direct mathematical inverses through learned approximations

Implementation Steps

1. Decompose Task Inputs

Create paired representations separating known and unknown components:

def decompose_task(task_input: str, task_type: str) -> tuple[str, str]:
    """
    Decompose task into observable and hidden components.

    For math: full equation becomes observable setup + hidden variables
    For translation: source+target becomes observable pairs + masked segments
    """
    if task_type == "math":
        # Extract equation structure, mask coefficients/variables
        observable = extract_equation_skeleton(task_input)
        hidden = extract_hidden_variables(task_input)
    elif task_type == "translation":
        # Preserve source, mask target segments
        observable = task_input.split("|||")[0]  # source language
        hidden = task_input.split("|||")[1]      # target language
    return observable, hidden

2. Generate Dual Task Reconstructions

Create complementary tasks to reconstruct the hidden information:

def create_dual_task(observable: str, task_type: str) -> str:
    """
    Construct dual task that reconstructs hidden information from primary output.
    """
    if task_type == "math":
        # Task: Given solution, recover hidden variables
        prompt = f"Solution: {observable}\nRecover the original variables/coefficients"
    elif task_type == "translation":
        # Task: Given translated output, reconstruct source
        prompt = f"Translated text: {observable}\nRetrieve original source language"
    return prompt

3. Compute Reconstruction Quality

Evaluate how well the model reconstructs hidden information:

def compute_reconstruction_score(
    original_hidden: str,
    reconstructed: str,
    similarity_metric: str = "exact_match"
) -> float:
    """
    Measure reconstruction quality as self-supervised reward.
    """
    if similarity_metric == "exact_match":
        return 1.0 if reconstructed.strip() == original_hidden.strip() else 0.0
    elif similarity_metric == "semantic":
        # Use embedding similarity or parsing-based comparison
        original_embedding = encode(original_hidden)
        reconstructed_embedding = encode(reconstructed)
        return cosine_similarity(original_embedding, reconstructed_embedding)
    return 0.0

4. Apply Preference Optimization

Use reconstruction scores as reward signals in preference learning:

def apply_preference_optimization(
    model: LLM,
    training_examples: list[dict],
    reconstruction_scores: list[float],
    learning_rate: float = 1e-5
) -> LLM:
    """
    Optimize model preferences based on reconstruction quality rewards.
    """
    # Sort examples by reconstruction score
    ranked_pairs = create_preference_pairs(
        training_examples,
        reconstruction_scores
    )

    # Apply DPO-style loss: prefer higher-scored outputs
    for batch in ranked_pairs:
        preferred_output = batch["high_score"]
        dispreferred_output = batch["low_score"]

        loss = compute_dpo_loss(
            model,
            preferred_output,
            dispreferred_output,
            beta=0.5
        )

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    return model

5. Handle Non-Invertible Problems

Extend framework to tasks without direct mathematical inverses:

def handle_non_invertible_task(
    primary_output: str,
    task_type: str,
    learned_inverter: LLM
) -> tuple[str, float]:
    """
    For non-invertible tasks, use learned approximation to reconstruct hidden info.
    """
    # Use auxiliary model trained to approximate the inverse
    reconstructed = learned_inverter.generate(
        f"Given output: {primary_output}\nApproximate original input:"
    )

    # Score based on output validity and consistency checks
    validity_score = check_output_consistency(reconstructed, task_type)
    return reconstructed, validity_score

Practical Guidance

When to Use DuPO

• Mathematical reasoning tasks where solutions enable variable recovery
• Machine translation with source-target decomposition
• Code generation with problem-solution duality
• Any domain where complementary reconstruction tasks can be formulated

When NOT to Use DuPO

• Tasks without clear input decomposition (e.g., open-ended generation)
• Domains where reconstruction is computationally expensive
• Single-step decision problems without intermediate structure

Key Hyperparameters

• Beta (β): Controls preference learning strength; typical range 0.3-1.0
• Reconstruction Threshold: Minimum quality score to count as valid reward
• Dual Task Complexity: Balance between informativeness and computational cost
• Training Ratio: Proportion of training data using dual feedback vs. other signals

Performance Expectations

• Translation: +2.13 COMET points across language pairs
• Mathematical reasoning: +6.4 points on AIME/Math benchmarks
• Inference-time reranking: +9.3 point improvement when selecting best-of-N outputs
• Token efficiency: Maintains or improves accuracy while reducing generation costs

Reference

She, S., Bao, Y., Lu, Y., Xu, L., Li, T., Zhu, W., Huang, S., Cheng, S., Lu, L., & Wang, Y. (2024). DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimization. arXiv preprint arXiv:2508.14460.