ADu2021/f-grpo-divergence-alignment

f-GRPO: Divergence-Based RL for LLM Alignment

Popular LLM alignment methods optimize divergence between aligned and unaligned distributions, yet lack a unified framework. f-GRPO grounds GRPO in f-divergence theory, enabling selection of divergence measures matching your alignment objectives. For verifiable rewards (correct/incorrect), f-GRPO concentrates probability on high-reward responses. For preference-based alignment, f-HAL balances on-policy exploration with off-policy preference learning to prevent reward hacking.

Core Concept

f-GRPO generalizes GRPO by parameterizing the divergence: min_π f(π || π_ref) where f is any f-divergence (KL, Jensen-Shannon, Hellinger, etc.). Different f-divergences yield different concentration behaviors:

• KL divergence: Punishes low-probability modes in π_ref; stays close to reference
• Jensen-Shannon: Symmetric; balances exploration and exploitation
• Hellinger: More aggressive concentration on high-reward modes
• Reverse KL: Avoids modes with low rewards in π_ref

For verifiable rewards: f-GRPO estimates divergence between above-average and below-average reward distributions. For preference-based: f-HAL uses on-policy RL (exploration) combined with off-policy preference signals (exploitation).

Architecture Overview

• f-Divergence Family: Choose divergence matching alignment goals (concentration vs. exploration)
• Reward-Based f-GRPO: For verifiable rewards (math, code); converges to max-reward policies
• f-HAL: Hybrid on-policy + off-policy for preference alignment; prevents reward hacking
• Monotonic Improvement: All variants guarantee reward improvement until convergence
• Theoretical Guarantees: Alignment consistency proofs for each divergence choice

Implementation

Implement f-GRPO for verifiable rewards:

import torch
import torch.nn.functional as F

def compute_f_divergence(p_logits, q_logits, divergence_type='kl'):
    """
    Compute f-divergence between two distributions.
    Args:
        p_logits: Logits from policy distribution [batch, vocab_size]
        q_logits: Logits from reference distribution [batch, vocab_size]
        divergence_type: 'kl', 'js', 'hellinger', etc.
    Returns:
        divergence: Scalar divergence value
    """
    p = F.softmax(p_logits, dim=-1)
    q = F.softmax(q_logits, dim=-1)

    if divergence_type == 'kl':
        # KL(p || q) = sum p * log(p/q)
        return (p * (torch.log(p) - torch.log(q))).sum(dim=-1).mean()

    elif divergence_type == 'js':
        # Jensen-Shannon: symmetric divergence
        m = 0.5 * (p + q)
        return 0.5 * (p * (torch.log(p) - torch.log(m))).sum(dim=-1).mean() + \
               0.5 * (q * (torch.log(q) - torch.log(m))).sum(dim=-1).mean()

    elif divergence_type == 'hellinger':
        # Hellinger distance (squared)
        return (torch.sqrt(p * q + 1e-8)).sum(dim=-1).mean()

    else:
        raise ValueError(f"Unknown divergence: {divergence_type}")

def f_grpo_advantage(rewards, divergence_strength=1.0):
    """Compute advantages for f-GRPO with reward-based concentration."""
    # Separate above-average and below-average rewards
    mean_reward = rewards.mean()
    above_avg = (rewards > mean_reward).float()

    # Advantage: how much better than average
    advantage = rewards - mean_reward

    return advantage * divergence_strength

def f_grpo_loss(policy_logits, reference_logits, rewards, divergence_type='kl',
                 divergence_strength=1.0, entropy_coef=0.01):
    """Compute f-GRPO loss combining divergence and reward optimization."""
    # Compute divergence penalty
    divergence = compute_f_divergence(policy_logits, reference_logits, divergence_type)

    # Compute advantages for reward optimization
    advantages = f_grpo_advantage(rewards, divergence_strength)

    # Policy gradient
    log_probs = F.log_softmax(policy_logits, dim=-1)
    policy_loss = -(log_probs.detach() * advantages.unsqueeze(-1)).mean()

    # Entropy regularization (prevent mode collapse)
    entropy = -(log_probs * F.softmax(policy_logits, dim=-1)).sum(dim=-1).mean()

    # Combined loss
    loss = policy_loss + divergence + entropy_coef * entropy

    return loss

def f_grpo_training_step(policy, reference_model, batch, optimizer, divergence_type='js'):
    """Single f-GRPO training step."""
    inputs, rewards = batch

    # Get logits
    policy_logits = policy(inputs).logits
    with torch.no_grad():
        ref_logits = reference_model(inputs).logits

    # Compute loss
    loss = f_grpo_loss(policy_logits, ref_logits, rewards, divergence_type=divergence_type)

    # Update
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    return loss.item()

Implement f-HAL for preference-based alignment:

def f_hal_loss(policy_logits, reference_logits, preference_pairs, on_policy_rewards,
               divergence_type='js', on_policy_weight=0.5, off_policy_weight=0.5):
    """
    f-HAL: Hybrid on-policy + off-policy preference alignment.
    Args:
        policy_logits: [batch, vocab_size]
        reference_logits: [batch, vocab_size]
        preference_pairs: [(preferred_seq, dispreferred_seq), ...] from offline data
        on_policy_rewards: Rewards from on-policy rollouts
        on_policy_weight: Balance between on-policy and off-policy
        off_policy_weight: Weight for preference learning
    """
    # On-policy component: standard f-GRPO with on-policy rewards
    on_policy_loss = f_grpo_loss(policy_logits, reference_logits, on_policy_rewards,
                                  divergence_type=divergence_type)

    # Off-policy component: preference learning from offline pairs
    off_policy_loss = 0.0
    for preferred, dispreferred in preference_pairs:
        # Bradley-Terry preference model: log(π(preferred)) - log(π(dispreferred))
        pref_logprob = F.log_softmax(policy_logits, dim=-1)[preferred].sum()
        dispref_logprob = F.log_softmax(policy_logits, dim=-1)[dispreferred].sum()

        # Loss: maximize log odds of preferred
        off_policy_loss += -(pref_logprob - dispref_logprob)

    off_policy_loss = off_policy_loss / len(preference_pairs)

    # Combine with weight balance
    total_loss = on_policy_weight * on_policy_loss + off_policy_weight * off_policy_loss

    return total_loss

Practical Guidance

| Parameter | Recommendation | Notes | |-----------|-----------------|-------| | Divergence type | Start with 'js' | Jensen-Shannon balances exploration; tune if needed. | | Divergence strength | 0.5-2.0 | Higher values strengthen alignment pressure. | | On-policy weight | 0.5-0.7 | Balance exploration (on-policy) vs. preference (off-policy). | | Off-policy weight | 0.3-0.5 | Too high causes reward hacking; too low loses preference signal. | | Entropy coefficient | 0.01-0.05 | Prevents mode collapse; adjust if needed. |

When to Use

• Alignment with verifiable rewards (math, code) → use f-GRPO
• Preference-based alignment with safety concerns → use f-HAL
• Need flexible divergence choices for domain-specific goals
• Want theoretical guarantees on reward improvement

When NOT to Use

• Simple reward signals (standard GRPO sufficient)
• Extremely limited offline data (off-policy weight too high)

Common Pitfalls

• Divergence weight too high → policy stops learning meaningful patterns
• Mixing on-policy and off-policy improperly → reward hacking reappears
• Not monitoring entropy; mode collapse if too low
• Choosing wrong divergence for task (test JS first, then adjust)

Reference

See https://arxiv.org/abs/2602.05946 for theoretical analysis of f-divergence, convergence proofs, and empirical validation on math reasoning and safety alignment benchmarks.