wenmin-wu/llm-sentence-truncation-fallback

Sentence Truncation with Fallback

Overview

LLM outputs for structured tasks (prompt recovery, classification rationale, single-line answers) often include unwanted preamble, repetition, or rambling. Truncate to the first N sentences, then check length — if the result is too short or empty, substitute a well-scoring baseline string. This two-step post-processing ensures consistent submission quality.

Quick Start

import re

BASELINE = "Improve the following text using a more formal and academic tone while maintaining the original meaning"

def truncate_sentences(text, max_sentences=1):
    sentences = re.split(r'(?<=[.!?])\s+', text.strip())
    result = ' '.join(sentences[:max_sentences])
    return result

def postprocess(raw_output, min_length=15):
    cleaned = truncate_sentences(raw_output, max_sentences=1)
    cleaned = cleaned.strip().strip('"').strip("'")
    if len(cleaned) < min_length:
        return BASELINE
    return cleaned

predictions = [postprocess(output) for output in raw_outputs]

Workflow

1. Split raw LLM output by sentence boundaries
2. Keep only the first N sentences (typically 1 for single-prompt tasks)
3. Strip quotes, whitespace, and formatting artifacts
4. Check minimum length — outputs under threshold are likely garbage
5. Replace short/empty outputs with a high-scoring baseline string

Key Decisions

• Baseline string: use a generic, broadly applicable prompt that scores well on average (~0.60+ similarity)
• min_length: 15-20 characters filters out single-word or empty outputs
• Sentence count: 1 for prompt recovery; 2-3 for summaries or explanations
• Combine with timeout: apply fallback also when inference exceeds time budget

References

• Mistral 7B Prompt Recovery Version 2