wenmin-wu/nlp-pairwise-softmax-rank-aggregation

Pairwise Softmax Rank Aggregation

Overview

In pairwise ranking, a model predicts P(A > B) for every pair (A, B). To recover a global ordering, apply temperature-scaled softmax over pairwise scores for each query item, then compute a weighted sum of anchor positions. The temperature controls sharpness: high temperature produces a soft average, low temperature approaches argmax. This converts N binary predictions into a single continuous rank value.

Quick Start

import numpy as np

def pairwise_to_rank(pairwise_preds, anchor_positions, temperature=20):
    """Convert pairwise scores to a continuous rank.

    Args:
        pairwise_preds: array of P(query > anchor_i) for each anchor
        anchor_positions: known positions of anchor items (e.g., code cell ranks)
        temperature: higher = sharper softmax (default 20)
    """
    centered = pairwise_preds - np.mean(pairwise_preds)
    weights = np.exp(centered * temperature)
    weights /= weights.sum()
    return np.sum(weights * anchor_positions)

# For each markdown cell, predict rank relative to all code cells
for i, md_preds in enumerate(all_pairwise_preds):
    ranks[i] = pairwise_to_rank(md_preds, code_cell_ranks)

Workflow

1. Train a binary classifier: P(item_A should come before item_B)
2. At inference, predict pairwise scores for each query against all anchors
3. Center predictions (subtract mean) for numerical stability
4. Apply temperature-scaled softmax to get attention weights
5. Weighted sum of anchor positions gives the continuous rank

Key Decisions

• Temperature: 10-30 typical; higher makes ranking sharper, lower gives smoother averages
• Anchor set: Use items with known positions (e.g., code cells with fixed relative order)
• Centering: Subtract mean before softmax to prevent overflow
• Alternative: Simple argmax of pairwise scores, but loses granularity

References

• AI4Code Pairwise BertSmall inference