wenmin-wu/nlp-pearson-correlation-metric

Pearson Correlation Metric

Overview

For regression tasks where the target represents similarity or relatedness scores, Pearson correlation measures linear agreement between predictions and labels — invariant to scale and shift. This makes it better than MSE for model selection: a model with correct ranking but wrong scale scores poorly on MSE but perfectly on Pearson r. Standard metric for STS benchmarks and similarity competitions.

Quick Start

import numpy as np
from scipy import stats

def pearson_score(y_true, y_pred):
    return stats.pearsonr(y_true, y_pred)[0]

# HuggingFace Trainer integration
def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    predictions = predictions.reshape(-1)
    return {'pearson': np.corrcoef(predictions, labels)[0][1]}

trainer = Trainer(
    model=model, args=args,
    train_dataset=train_ds, eval_dataset=val_ds,
    compute_metrics=compute_metrics,
)

Workflow

1. Define compute_metrics returning Pearson r from predictions and labels
2. Pass to HuggingFace Trainer or compute manually in a training loop
3. Use Pearson r for checkpoint selection: if score > best_score: save_model()
4. Report final CV score as mean Pearson r across folds

Key Decisions

• Pearson vs Spearman: Pearson measures linear correlation; Spearman measures rank correlation. Use Pearson when the relationship is approximately linear
• Scale invariance: Pearson r doesn't penalize predictions that are scaled/shifted — pair with MSE loss during training to maintain calibration
• p-value: scipy.stats.pearsonr also returns p-value — useful for small validation sets
• Numpy shortcut: np.corrcoef(x, y)[0][1] is faster than scipy for large arrays

References

• Getting started with NLP for absolute beginners
• PPPM / Deberta-v3-large baseline w/ W&B [train]