wenmin-wu/nlp-pairwise-ranking-validation

Pairwise Ranking Validation

Overview

When the task is ranking (not classification), standard metrics like accuracy or AUC don't apply directly. Pairwise ranking accuracy measures the fraction of (item_a, item_b) pairs where the model correctly assigns a higher score to the preferred item. For toxicity ranking: given pairs of (less_toxic, more_toxic) texts, check if score(more_toxic) > score(less_toxic). This metric directly reflects competition evaluation and is the right validation signal for margin ranking or regression-based ranking models.

Quick Start

import numpy as np

def pairwise_accuracy(model, vectorizer, val_df):
    """Fraction of pairs correctly ordered by model scores."""
    X_less = vectorizer.transform(val_df['less_toxic'])
    X_more = vectorizer.transform(val_df['more_toxic'])

    score_less = model.predict(X_less)
    score_more = model.predict(X_more)

    return (score_less < score_more).mean()

# For transformer models
def pairwise_accuracy_nn(model, val_loader):
    scores_less, scores_more = [], []
    for batch in val_loader:
        with torch.no_grad():
            s_less = model(batch['less_ids'], batch['less_mask'])
            s_more = model(batch['more_ids'], batch['more_mask'])
        scores_less.append(s_less.cpu())
        scores_more.append(s_more.cpu())
    less = torch.cat(scores_less)
    more = torch.cat(scores_more)
    return (less < more).float().mean().item()

acc = pairwise_accuracy(ridge_model, tfidf, val_df)
print(f"Pairwise accuracy: {acc:.4f}")

Workflow

1. Score both items in each validation pair independently
2. Compare: does the preferred item get a higher score?
3. Average across all pairs to get pairwise accuracy
4. Use to compare models, tune hyperparameters, select thresholds

Key Decisions

• Ties: Ties count as incorrect by default; can count as 0.5 with (less < more).mean() + 0.5 * (less == more).mean()
• vs Kendall's tau: Pairwise accuracy on given pairs is simpler; Kendall's tau considers all possible pairs
• Baseline: Random ranking gives 0.50; a good model should be 0.70+
• Per-fold: Compute per CV fold and average for stable estimates

References

• Jigsaw ensemble best public sub
• Jigsaw Incredibly Simple Naive Bayes