wenmin-wu/llm-turn-based-prompt-accumulation

Turn-Based Prompt Accumulation

Overview

Causal LLMs without native chat APIs need conversation history manually injected into the prompt. Each turn, rebuild the full prompt by interleaving all previous user and assistant messages with the model's special tokens (e.g., <|start_header_id|>, <|eot_id|> for Llama). This gives the model full context of the conversation without maintaining KV cache across calls.

Quick Start

def build_chat_prompt(system_prompt, questions, answers, model_format="llama3"):
    prompt = (f"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n"
              f"{system_prompt}<|eot_id|>")

    for i, (q, a) in enumerate(zip(questions, answers)):
        prompt += (f"<|start_header_id|>assistant<|end_header_id|>\n\n"
                   f"{q}<|eot_id|>")
        prompt += (f"<|start_header_id|>user<|end_header_id|>\n\n"
                   f"{a}<|eot_id|>")

    # Open the next assistant turn
    prompt += "<|start_header_id|>assistant<|end_header_id|>\n\n"
    return prompt

prompt = build_chat_prompt(sys_prompt, obs.questions, obs.answers)
output = model.generate(tokenizer(prompt, return_tensors="pt").to("cuda"),
                        max_new_tokens=50)

Workflow

1. Start with system prompt wrapped in model's special tokens
2. For each previous turn, append assistant message then user response with proper headers
3. End with an open assistant header to prompt the next generation
4. Tokenize the full prompt and generate
5. Truncate output at EOT token to extract clean response

Key Decisions

• Token format: Llama3 uses <|start_header_id|>role<|end_header_id|>; Gemma uses <start_of_turn>model
• Rebuild vs append: rebuilding from scratch each turn is simpler and avoids KV cache issues
• Context length: monitor total token count — truncate oldest turns if approaching limit
• EOT truncation: find the EOT token in generated output and slice before it

References

• llama3-8b- LLM20 Questions [LB 750]
• LLM 20 Questions Starter Notebook