wenmin-wu/nlp-translation-regex-postprocessing

Translation Regex Post-Processing

Overview

Seq2seq models produce artifacts: repeated phrases, prompt leakage, trailing fragments, inconsistent punctuation. A cascaded regex pipeline fixes these systematically. Apply after decoding, before MBR or final submission.

Quick Start

import re

RULES = [
    # Remove leaked prompt prefix
    (re.compile(r'(?i)^translate \w+ to \w+:\s*'), ''),
    # Deduplicate repeated phrases (2-4 word spans)
    (re.compile(r'\b(\w+(?:\s+\w+){1,3})\s+\1\b'), r'\1'),
    # Collapse repeated single words
    (re.compile(r'\b(\w+)(\s+\1){2,}\b'), r'\1'),
    # Remove trailing short fragments
    (re.compile(r'\s+\w{1,3}$'), ''),
    # Normalize multiple spaces
    (re.compile(r'\s{2,}'), ' '),
    # Fix space before punctuation
    (re.compile(r'\s+([.,;:!?])'), r'\1'),
]

def postprocess_translation(text):
    text = text.strip()
    for pattern, repl in RULES:
        text = pattern.sub(repl, text)
    if text and text[-1] not in '.!?"':
        text += '.'
    return text.strip()

Key Decisions

• Order matters: prompt removal first, dedup second, punctuation last
• Sentence ending: force period if missing — most metrics penalize incomplete sentences
• Conservative dedup: only exact phrase repeats, not paraphrases

References

• Source: hybrid-best-akkadian
• Competition: Deep Past Challenge - Translate Akkadian to English