wenmin-wu/nlp-per-class-span-filtering

Per-Class Span Filtering

Overview

NER models produce many short, low-confidence span predictions that are false positives. Different entity types have different natural lengths — an "Evidence" span should be at least 14 words, while a "Claim" can be as short as 3. Per-class filtering applies separate minimum length and confidence thresholds for each class, tuned on validation data. This simple post-processing step consistently improves F1 by 2-5 points.

Quick Start

import numpy as np

# Thresholds tuned on validation set
MIN_WORDS = {"Lead": 9, "Position": 5, "Evidence": 14,
             "Claim": 3, "Concluding Statement": 11,
             "Counterclaim": 6, "Rebuttal": 4}
MIN_PROBA = {"Lead": 0.70, "Position": 0.55, "Evidence": 0.65,
             "Claim": 0.55, "Concluding Statement": 0.70,
             "Counterclaim": 0.50, "Rebuttal": 0.55}

def filter_spans(predictions):
    """Filter spans by per-class length and probability thresholds.

    predictions: list of (doc_id, class, word_indices_str, mean_proba)
    """
    filtered = []
    for doc_id, cls, pred_str, proba in predictions:
        n_words = len(pred_str.split())
        if n_words >= MIN_WORDS.get(cls, 1) and proba >= MIN_PROBA.get(cls, 0.5):
            filtered.append((doc_id, cls, pred_str))
    return filtered

Workflow

1. Generate raw span predictions with confidence scores
2. For each span, compute word count and mean token probability
3. Filter out spans below per-class minimum length
4. Filter out spans below per-class probability threshold
5. Tune thresholds on validation set (grid search or Optuna)

Key Decisions

• Threshold tuning: Grid search over (length, proba) pairs per class on validation F1
• Length vs probability: Length filtering alone gives most of the gain; probability adds refinement
• Class-specific: One-size-fits-all thresholds lose 1-2 F1 points vs per-class
• Order: Apply length filter first (cheaper), then probability filter

References

• Two Longformers Are Better Than 1
• Infer Fast Ensemble Models