wenmin-wu/nlp-jaccard-dedup-prediction-filter

Jaccard Dedup Prediction Filter

Overview

NER and entity extraction pipelines often produce near-duplicate predictions ("National Health Survey" vs "National Health Survey Data"). Greedy Jaccard deduplication: sort candidates by length, accept each only if its word-level Jaccard similarity with all previously accepted labels is below a threshold. Removes redundant extractions while preserving distinct entities.

Quick Start

def jaccard_similarity(s1, s2):
    words1 = set(s1.lower().split())
    words2 = set(s2.lower().split())
    intersection = len(words1 & words2)
    union = len(words1 | words2)
    return intersection / union if union > 0 else 0.0

def dedup_entities(entities, threshold=0.75):
    filtered = []
    for entity in sorted(entities, key=len, reverse=True):
        if not filtered or all(
            jaccard_similarity(entity, kept) < threshold
            for kept in filtered
        ):
            filtered.append(entity)
    return filtered

# Usage:
raw_predictions = ["National Health Survey", "National Health Survey Data", "Census Bureau"]
clean = dedup_entities(raw_predictions, threshold=0.75)
# -> ["National Health Survey Data", "Census Bureau"]

Workflow

1. Collect all raw entity predictions for a document
2. Sort by length (longest first to prefer more complete mentions)
3. Greedily accept each candidate if Jaccard < threshold vs all accepted
4. Output filtered set as pipe-delimited string or list

Key Decisions

• Threshold: 0.5 = strict (keeps more variants); 0.75 = moderate; 1.0 = exact dedup only
• Sort order: Longest-first keeps the most informative mention; shortest-first is more conservative
• Word-level vs char-level: Word Jaccard is standard for entity dedup; char-level for typo handling
• Case sensitivity: Lowercase before comparison to catch capitalization variants

References

• Coleridge: Matching + BERT NER
• External_Datasets_Matching + MLM