wenmin-wu/nlp-tfidf-translation-memory

TF-IDF Translation Memory

Overview

For repetitive or formulaic text, retrieve the closest source from a translation memory using TF-IDF similarity, then return its paired target. Combines character n-gram and word n-gram TF-IDF with SequenceMatcher reranking. Works as a standalone system or ensemble component alongside neural MT.

Quick Start

from sklearn.feature_extraction.text import TfidfVectorizer
from difflib import SequenceMatcher
import numpy as np

class TranslationMemory:
    def __init__(self, sources, targets):
        self.targets = targets
        self.char_vec = TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 6))
        self.word_vec = TfidfVectorizer(analyzer="word", ngram_range=(1, 2))
        self.Xc = self.char_vec.fit_transform(sources)
        self.Xw = self.word_vec.fit_transform(sources)
        self.sources = sources

    def retrieve(self, query, top_k=5, min_score=0.3):
        sc = (self.char_vec.transform([query]) @ self.Xc.T).toarray()[0]
        sw = (self.word_vec.transform([query]) @ self.Xw.T).toarray()[0]
        combined = 0.6 * sc + 0.4 * sw
        top_idx = np.argsort(-combined)[:top_k]
        # Rerank with SequenceMatcher
        best_i, best_s = top_idx[0], -1
        for idx in top_idx:
            s = SequenceMatcher(None, query, self.sources[idx]).ratio()
            if s > best_s:
                best_i, best_s = idx, s
        return self.targets[best_i] if combined[best_i] > min_score else None

Key Decisions

• Char + word TF-IDF: char n-grams handle typos and morphology, word n-grams capture semantics
• SequenceMatcher rerank: edit-distance-based reranking on top candidates improves precision
• min_score threshold: below threshold, fall back to neural MT — avoids bad retrievals

References

• Source: dpc-starter-infer-add-sentencealign
• Competition: Deep Past Challenge - Translate Akkadian to English