ADu2021/docdancer-document-agent

When to Use This Skill

• Document question-answering systems with limited training data
• Applications requiring tool-driven exploration (highlighting, extracting, reasoning)
• Long-document understanding where sequential processing is necessary
• Scenarios where synthetic data generation can reduce annotation costs
• Building open-source DocQA agents without proprietary models

When NOT to Use This Skill

• Single-turn simple fact lookup (standard retrieval sufficient)
• Applications with abundant labeled DocQA training data
• Systems with hard latency constraints (multi-step reasoning is slower)
• Short-document scenarios (tool exploration adds overhead)

Problem Summary

Existing document question-answering (DocQA) agents suffer from two critical limitations: (1) they lack effective tool utilization, relying on implicit understanding instead of explicit document exploration, and (2) they depend heavily on closed-source models, limiting accessibility and adaptability. The fundamental barrier is scarcity of high-quality training data for DocQA agents—annotation is expensive and difficult at scale.

Solution: Tool-Driven DocQA with Synthetic Data

Model DocQA as information-seeking with explicit tool integration, then generate synthetic training data through an exploration-then-synthesis pipeline.

class DocDancerAgent:
    def __init__(self, base_llm, document):
        self.llm = base_llm
        self.document = document
        self.interaction_history = []

    def answer_document_question(self, question):
        """Tool-driven exploration followed by answer synthesis"""

        # Phase 1: Exploration
        exploration_steps = self.explore_document(question)
        # exploration_steps = [
        #     {"action": "highlight", "text": "...", "rationale": "..."},
        #     {"action": "extract", "content": "...", "rationale": "..."},
        #     {"action": "reason", "inference": "...", "rationale": "..."}
        # ]

        # Phase 2: Synthesis
        answer = self.synthesize_answer(question, exploration_steps)

        self.interaction_history.append({
            "question": question,
            "exploration": exploration_steps,
            "answer": answer
        })

        return answer

    def explore_document(self, question):
        """Sequential tool invocation for information gathering"""
        steps = []
        context = f"Question: {question}\nDocument: {self.document[:2000]}..."

        for exploration_turn in range(max_exploration_steps):
            # Decide which tool to use next
            tool_decision = self.llm.generate(f"""
            Current exploration state:
            {format_exploration_history(steps)}

            Question: {question}

            What's the next exploration action?
            Options:
            - highlight: Mark important text regions
            - extract: Pull out specific information
            - reason: Make inference from gathered info
            - stop: Sufficient information gathered
            """)

            action = parse_tool_action(tool_decision)

            if action == "stop":
                break

            # Execute chosen tool
            if action == "highlight":
                highlighted_text = self.identify_relevant_sections(question, context)
                steps.append({
                    "action": "highlight",
                    "text": highlighted_text,
                    "rationale": tool_decision
                })
            elif action == "extract":
                extracted_content = self.extract_key_information(question, context)
                steps.append({
                    "action": "extract",
                    "content": extracted_content,
                    "rationale": tool_decision
                })
            elif action == "reason":
                inference = self.llm.generate(f"""
                Based on gathered evidence:
                {format_exploration_steps(steps)}

                Make an inference relevant to: {question}
                """)
                steps.append({
                    "action": "reason",
                    "inference": inference,
                    "rationale": tool_decision
                })

        return steps

    def synthesize_answer(self, question, exploration_steps):
        """Combine exploration traces into final answer"""
        synthesis_prompt = f"""
        Question: {question}

        Exploration process:
        {format_exploration_steps(exploration_steps)}

        Based on this exploration, provide the final answer.
        """
        return self.llm.generate(synthesis_prompt)

Key Implementation Details

Tool-Driven Architecture:

• Explicit tools: highlight, extract, reason, summarize
• Sequential tool invocation with history tracking
• Each tool provides interpretable signals for debugging

Exploration-Then-Synthesis Pipeline: Generates high-quality synthetic training data:

def generate_synthetic_training_data(document, gold_answer, num_samples=100):
    """Generate diverse question-exploration-answer triplets"""
    synthetic_data = []

    for sample_idx in range(num_samples):
        # Generate question variants that require document exploration
        question = generate_question_from_answer(gold_answer, document)

        # Simulate diverse exploration strategies
        exploration_trajectories = []
        for strategy in ["sequential", "selective", "comprehensive"]:
            trajectory = simulate_exploration(
                question, document, gold_answer, strategy
            )
            exploration_trajectories.append(trajectory)

        # Create training examples from best exploration
        best_trajectory = select_best_trajectory(
            exploration_trajectories, gold_answer
        )

        synthetic_data.append({
            "question": question,
            "document": document,
            "exploration": best_trajectory,
            "answer": gold_answer
        })

    return synthetic_data

Training Data Methodology

Data Scarcity Problem:

• MMLongBench-Doc: Limited labeled examples
• Manual annotation is expensive
• Diversity of document types and questions is limited

Solution: Synthetic Generation

• Start with reference answers
• Generate plausible questions backward
• Simulate exploration trajectories
• Create diverse exploration styles
• Output: 100x more training examples than labeled data

Performance Evaluation

Benchmarks:

• MMLongBench-Doc: Multimodal document QA
• DocBench: Document understanding across domains

Comparison:

• Outperforms closed-source baselines
• Open-source implementation enables reproducibility
• Tool-explicit approach provides interpretability

Advantages Over Baselines

• vs. LLM-Only: Tool-driven exploration prevents hallucination
• vs. Closed-Source: Open-source enables customization
• vs. Retrieval-Only: Reasoning capability over multi-hop questions
• vs. Unaided Agents: Synthetic data provides training signal

Deployment Strategy

1. Document Processing: Load and preprocess target documents
2. Tool Integration: Implement highlight, extract, reason tools
3. Data Generation: Create synthetic training examples
4. Agent Training: Fine-tune on exploration-answer pairs
5. Evaluation: Test on held-out document QA tasks
6. Iteration: Analyze errors to improve exploration strategies

Open-Source Implementation

Full codebase released with:

• Synthetic data generation scripts
• Tool implementations
• Training utilities for benchmark datasets
• Evaluation metrics