ADu2021/derf-normalization-free-transformers

Overview

The paper demonstrates that simpler point-wise functions can replace traditional normalization while achieving superior generalization. Derf, based on rescaled Gaussian CDF, provides normalization-free transformers with improved performance across diverse domains.

When to Use

• Training transformers without normalization layers
• Scenarios where generalization is critical
• Vision, speech, and sequence modeling tasks
• Need for improved training stability
• Replacing LayerNorm or RMSNorm

When NOT to Use

• Applications already achieving good normalization
• Scenarios where norm-dependent features matter
• Models requiring specific normalization properties

Core Technique

Derf activation function design:

# Normalization-free transformers with Derf
class DerfNormalizationFree(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.alpha = nn.Parameter(torch.ones(1))
        self.s = nn.Parameter(torch.zeros(1))

    def forward(self, x):
        """Derf(x) = erf(α*x + s)"""
        from scipy.special import erf as scipy_erf

        # Apply Derf activation
        output = torch.erf((self.alpha * x + self.s) / math.sqrt(2.0))

        return output

class TransformerBlock(nn.Module):
    def __init__(self, dim, num_heads):
        super().__init__()
        self.attn = nn.MultiheadAttention(dim, num_heads)
        # NO LayerNorm, use Derf instead
        self.derf = DerfNormalizationFree(dim)
        self.ffn = nn.Sequential(
            nn.Linear(dim, 4 * dim),
            self.derf,
            nn.Linear(4 * dim, dim)
        )

    def forward(self, x):
        # Self-attention
        attn_output = self.attn(x, x, x)[0]
        # Apply Derf instead of LayerNorm
        x = x + self.derf(attn_output)

        # Feed-forward
        ffn_output = self.ffn(x)
        # Derf-based combination
        x = x + self.derf(ffn_output)

        return x

Key Results

• Derf outperforms LayerNorm on generalization
• Effective across vision, speech, DNA
• Dynamic Tanh surpassed
• Improved training stability

References

• Original paper: https://arxiv.org/abs/2512.10938
• Focus: Normalization-free transformer training
• Domain: Transformer architectures, activation functions