wenmin-wu/nlp-weight-decay-bias-exclusion

Weight Decay Bias Exclusion

Overview

Weight decay (L2 regularization) should only apply to weight matrices, not to bias terms or LayerNorm parameters. Regularizing biases pushes them toward zero unnecessarily, and regularizing LayerNorm weights distorts the learned scale. Standard practice for fine-tuning transformers — split parameters into two groups with different decay rates.

Quick Start

from torch.optim import AdamW

def get_optimizer_grouped_parameters(model, lr=2e-5, weight_decay=0.01):
    """Split parameters into decay and no-decay groups."""
    no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']
    
    return [
        {
            'params': [p for n, p in model.named_parameters()
                       if not any(nd in n for nd in no_decay)],
            'weight_decay': weight_decay
        },
        {
            'params': [p for n, p in model.named_parameters()
                       if any(nd in n for nd in no_decay)],
            'weight_decay': 0.0
        }
    ]

optimizer = AdamW(
    get_optimizer_grouped_parameters(model, lr=2e-5, weight_decay=0.01)
)

Key Decisions

• weight_decay=0.01: standard for BERT/transformer fine-tuning; increase for stronger regularization
• LayerNorm.weight included: this is a scale parameter, not a typical weight — decaying it hurts normalization
• String matching: any(nd in n ...) checks parameter names — verify naming convention matches your model
• Also applies to embeddings: some practitioners exclude embedding weights too — experiment

References

• Source: toxic-bert-plain-vanila
• Competition: Jigsaw Unintended Bias in Toxicity Classification