wenmin-wu/llm-test-time-train-pseudo-label-expansion

Overview

Some Kaggle test sets ship with positive_example_1, positive_example_2, negative_example_1, negative_example_2 columns alongside the row to score. These are gold labels — they let you do test-time training (TTT) without external data. The trick is the leakage trap: if you flatten naively and feed each example as both the target AND the in-prompt demo, the model just copies the demo. The fix is to always use the other example of the same polarity as the demo (3 - i index pairing). You get a clean labeled set the size of 4 * len(test) and a few-shot prompt with one positive + one negative demo that never overlaps the target row.

Quick Start

import pandas as pd

rows = []
for _, r in test.iterrows():
    for i in [1, 2]:
        rows.append({
            'body': r[f'positive_example_{i}'],
            'rule': r['rule'],
            'subreddit': r['subreddit'],
            'pos_demo': r[f'positive_example_{3 - i}'],   # the OTHER positive
            'neg_demo': r[f'negative_example_{3 - i}'],   # paired negative
            'label': 1,
        })
        rows.append({
            'body': r[f'negative_example_{i}'],
            'rule': r['rule'],
            'subreddit': r['subreddit'],
            'pos_demo': r[f'positive_example_{3 - i}'],
            'neg_demo': r[f'negative_example_{3 - i}'],
            'label': 0,
        })
ttt_df = pd.DataFrame(rows)   # 4 rows per test row, fully labeled

Workflow

1. For each test row with positive_example_{1,2} and negative_example_{1,2}, emit 4 labeled rows
2. For each emitted row, set the few-shot demo to the other index (3 - i) so the target is never in its own prompt
3. Train a classifier (LoRA on an instruct model, or just fit logistic regression on hidden states) on these flattened pseudo-labels
4. At inference on the original test row, build the prompt with positive_example_1 + negative_example_1 (either index works since the target row is unlabeled)
5. Optionally combine with public-comp training data — TTT pseudo-labels are domain-perfect but small

Key Decisions

• 3 - i pairing, not random: random demo selection mixes polarities and adds variance; deterministic pairing is reproducible and leak-free.
• Both polarities flattened together: keeps class balance per test row at exactly 50/50, no resampling needed.
• TTT, not just few-shot: the labeled rows are real targets — fine-tune (LoRA) on them, don't only use them as prompt context. Few-shot alone leaves capacity on the table.
• Don't deduplicate across test rows: the same (body, rule) may legitimately repeat with different surrounding context; dedup hurts.
• Keep subreddit and rule in the prompt: the labels are conditional on the rule; stripping it collapses distinct decision boundaries.

References

• Test on testdataset (Qwen embedding + Llama + LR)