wenmin-wu/llm-answer-accumulation-filter

Answer Accumulation Filter

Overview

In multi-turn question-answering games, each yes/no answer constrains the solution space. Maintain parallel inclusion and exclusion sets for each attribute dimension (category, region, letter). After each answer, update the appropriate set, then apply all filters as a compound boolean mask on the candidate dataframe. This turns sequential dialogue into progressively tighter database queries.

Quick Start

import pandas as pd

class AnswerAccumulator:
    def __init__(self, candidates_df):
        self.candidates = candidates_df
        self.include = {}  # attr -> set of included values
        self.exclude = {}  # attr -> set of excluded values

    def update(self, attr, value, answer_is_yes):
        if answer_is_yes:
            self.include.setdefault(attr, set()).add(value)
        else:
            self.exclude.setdefault(attr, set()).add(value)

    def filter(self):
        mask = pd.Series(True, index=self.candidates.index)
        for attr, vals in self.include.items():
            mask &= self.candidates[attr].isin(vals)
        for attr, vals in self.exclude.items():
            mask &= ~self.candidates[attr].isin(vals)
        return self.candidates[mask]

acc = AnswerAccumulator(keywords_df)
acc.update('category', 'place', answer_is_yes=True)
acc.update('continent', 'Europe', answer_is_yes=False)
remaining = acc.filter()  # all places not in Europe

Workflow

1. Initialize with full candidate dataframe and empty include/exclude dicts
2. After each yes/no answer, call update(attribute, value, is_yes)
3. Call filter() to get remaining candidates matching all constraints
4. Use len(remaining) to decide whether to ask more questions or guess

Key Decisions

• Attribute dimensions: category, region, first_letter are typical; add more for richer datasets
• Include vs exclude: "yes" → include (whitelist), "no" → exclude (blacklist)
• Multiple includes: if multiple values included for same attribute, use OR (any match)
• Empty sets: no constraint on that dimension — don't filter

References

• Starter Code for Llama 8B LLM - [LB 0.750+]