ADu2021/entropy-ratio-clipping-stable-rl

Overview

Entropy Ratio Clipping (ERC) addresses training instabilities in LLM post-training by introducing a global metric measuring relative policy exploration changes. While PPO-Clip helps locally, it overlooks the global distributional shift of actions. ERC provides bidirectional constraints on entropy ratio to stabilize policy updates globally.

When to Use

• Training language models with reinforcement learning post-training (DAPO, GPPO)
• Entropy fluctuations and unstable gradients appear during off-policy training
• Probability shifts of unsampled actions cause performance degradation
• Local clipping (PPO-Clip) alone is insufficient for stability

When NOT to Use

• Models already achieving stable training with PPO-Clip
• On-policy training scenarios without distribution shift concerns
• Applications where some degree of exploration variance is beneficial
• Tasks where entropy constraints reduce desired diversity

Core Technique

Entropy Ratio Clipping uses entropy ratio between successive policies as a constraint mechanism:

# Entropy Ratio Clipping for stable RL
class EntropyRatioClipping:
    def __init__(self, epsilon_lower=0.9, epsilon_upper=1.1):
        self.epsilon_lower = epsilon_lower
        self.epsilon_upper = epsilon_upper

    def compute_entropy_ratio(self, current_policy, previous_policy):
        """
        Measures relative exploration changes between policies.
        Quantifies distributional shifts at the aggregate level.
        """
        # Entropy of current policy
        current_entropy = self.compute_policy_entropy(current_policy)
        # Entropy of previous policy
        prev_entropy = self.compute_policy_entropy(previous_policy)

        # Entropy ratio
        ratio = current_entropy / (prev_entropy + 1e-8)
        return ratio

    def apply_erc_loss(self, ratio, base_loss):
        """
        Imposes bidirectional bounds on entropy ratio.
        Prevents excessive distributional divergence during training.
        """
        # Clip ratio to maintain global stability
        clipped_ratio = torch.clamp(
            ratio,
            self.epsilon_lower,
            self.epsilon_upper
        )

        # Combine with base policy loss
        erc_penalty = torch.mean((ratio - clipped_ratio) ** 2)
        total_loss = base_loss + 0.1 * erc_penalty

        return total_loss

    def compute_policy_entropy(self, logits):
        """Compute entropy of policy distribution."""
        probs = torch.softmax(logits, dim=-1)
        entropy = -torch.sum(probs * torch.log(probs + 1e-8), dim=-1)
        return entropy.mean()

ERC integrates into DAPO and GPPO by imposing constraints on global policy divergence while allowing local flexibility.

Key Results

• Consistent performance improvements across multiple benchmarks
• Stabilizes training when PPO-Clip alone is insufficient
• Maintains safety against unsampled action probability shifts
• Integration with DAPO and GPPO algorithms validated

Implementation Notes

• Bidirectional bounds (lower and upper) prevent both collapse and divergence
• Acts as soft constraint rather than hard clipping
• Complements rather than replaces PPO-Clip
• Applicable to any off-policy RL algorithm for LLM post-training

References

• Original paper: https://arxiv.org/abs/2512.05591
• Focus: Training stability in RL post-training
• Domain: Language model alignment, reinforcement learning