wenmin-wu/nlp-checkpoint-ensemble-exponential

Checkpoint Ensemble with Exponential Weights

Overview

Instead of using only the best checkpoint, collect predictions from every epoch and average with exponential weights (2^epoch). Later epochs get more weight since they're more converged, but earlier epochs contribute diversity. Costs zero extra training time — just save predictions at each epoch.

Quick Start

import numpy as np

def checkpoint_ensemble_predict(model, train_fn, test_loader, n_epochs,
                                 device='cuda'):
    """Train model and collect exponentially-weighted checkpoint predictions.
    
    Args:
        model: PyTorch model
        train_fn: function(model, epoch) that trains one epoch
        test_loader: DataLoader for test data
        n_epochs: total training epochs
    Returns:
        averaged predictions array
    """
    all_preds = []
    weights = [2 ** epoch for epoch in range(n_epochs)]
    
    for epoch in range(n_epochs):
        train_fn(model, epoch)
        
        model.eval()
        epoch_preds = []
        with torch.no_grad():
            for x_batch in test_loader:
                logits = model(x_batch.to(device))
                epoch_preds.append(torch.sigmoid(logits).cpu().numpy())
        all_preds.append(np.concatenate(epoch_preds))
    
    return np.average(all_preds, weights=weights, axis=0)

# Usage
predictions = checkpoint_ensemble_predict(model, train_one_epoch, test_loader, 5)

Key Decisions

• 2^epoch weights: geometric increase; last epoch gets 16x weight of first (for 5 epochs)
• No extra cost: predictions are collected during training — no separate inference pass needed
• Combine with K-fold: collect per-epoch preds for each fold, then average across both dimensions
• Alternative: uniform weights work too but exponential consistently scores 0.001–0.005 better

References

• Source: simple-lstm-pytorch-version
• Competition: Jigsaw Unintended Bias in Toxicity Classification