wenmin-wu/nlp-cyclical-lr-triangular

Overview

Cyclical Learning Rate (Smith 2017) avoids the manual LR tuning problem by sweeping between base_lr and max_lr repeatedly during training. The triangular pattern often converges faster than static LR and helps the model escape sharp minima. Three modes: triangular (constant amplitude), triangular2 (amplitude halves each cycle), exp_range (amplitude decays exponentially).

Quick Start

import numpy as np
import keras.backend as K
from keras.callbacks import Callback

class CyclicLR(Callback):
    def __init__(self, base_lr=1e-3, max_lr=6e-3, step_size=2000,
                 mode='triangular', gamma=1.0):
        super().__init__()
        self.base_lr = base_lr
        self.max_lr = max_lr
        self.step_size = step_size
        self.mode = mode
        self.gamma = gamma
        self.iterations = 0.
        if mode == 'triangular':
            self.scale_fn = lambda x: 1.
        elif mode == 'triangular2':
            self.scale_fn = lambda x: 1 / (2. ** (x - 1))
        elif mode == 'exp_range':
            self.scale_fn = lambda x: gamma ** x

    def clr(self):
        cycle = np.floor(1 + self.iterations / (2 * self.step_size))
        x = np.abs(self.iterations / self.step_size - 2 * cycle + 1)
        return self.base_lr + (self.max_lr - self.base_lr) * max(0, 1 - x) * self.scale_fn(cycle)

    def on_train_begin(self, logs=None):
        K.set_value(self.model.optimizer.lr, self.base_lr)

    def on_batch_end(self, batch, logs=None):
        self.iterations += 1
        K.set_value(self.model.optimizer.lr, self.clr())

# Usage
clr = CyclicLR(base_lr=1e-3, max_lr=6e-3, step_size=2000, mode='triangular2')
model.fit(X, y, callbacks=[clr])

Workflow

1. Run a LR range test first: train for ~1 epoch sweeping LR from 1e-7 to 1, plot loss. base_lr = start of decrease, max_lr = start of divergence.
2. Set step_size to 2-8 times num_batches_per_epoch (a full cycle = 2 * step_size iterations).
3. Attach the callback to model.fit — LR updates every batch, no extra code required.
4. Mode triangular2 decays amplitude over cycles for fine-tuning; exp_range for very long training.

Key Decisions

• step_size = 2-8 × epoch: Smaller means more cycles per epoch, larger means longer exploration per minimum.
• mode: triangular for short runs, triangular2 for longer ones (decays amplitude), exp_range when you want smooth decay.
• vs. fixed LR: Typically faster convergence and often comparable or better final accuracy with no LR tuning.

References

• Single RNN with 4 folds (CLR)