wenmin-wu/nlp-tfidf-weighted-word-match

TF-IDF Weighted Word Match

Overview

Simple word overlap (Jaccard) treats all words equally — "the" counts as much as "tensorflow". Weighting each shared word by inverse corpus frequency (IDF-style) makes rare shared terms contribute more to the similarity score. This produces a single float feature that strongly discriminates duplicate question pairs and similar text-matching tasks.

Quick Start

from collections import Counter
import numpy as np

# Build IDF-style weights from corpus
all_words = (" ".join(all_questions)).lower().split()
counts = Counter(all_words)
weights = {w: 1 / (c + 10000) for w, c in counts.items() if c >= 2}

stops = set(stopwords.words("english"))

def tfidf_word_match(row):
    q1 = {w for w in str(row["q1"]).lower().split() if w not in stops}
    q2 = {w for w in str(row["q2"]).lower().split() if w not in stops}
    if not q1 or not q2:
        return 0.0
    shared = [weights.get(w, 0) for w in q1 & q2]
    total = [weights.get(w, 0) for w in q1 | q2]
    return np.sum(shared) / (np.sum(total) + 1e-8)

df["tfidf_word_match"] = df.apply(tfidf_word_match, axis=1)

Workflow

1. Concatenate all text into a single corpus, tokenize, count word frequencies
2. Compute IDF-style weight: 1 / (count + smoothing) per word
3. For each text pair, find shared words (excluding stopwords)
4. Sum weights of shared words, divide by sum of all words' weights
5. Use as a feature for classification or ranking

Key Decisions

• Smoothing: eps=10000 prevents rare words from dominating; tune to corpus size
• Min count: Filter words appearing fewer than 2 times to avoid noise
• Stopwords: Remove standard stopwords before matching
• Complement with: Exact word match ratio (unweighted) as an additional feature

References

• Data Analysis & XGBoost Starter (0.35460 LB)