ADu2021/enterprise-tool-calling-finetuning

DiaFORGE: Disambiguation-Centric Fine-Tuning for Reliable Tool-Calling

In enterprise environments, a single business query often maps to multiple near-duplicate APIs, and most real calls arrive missing required parameters. Current LLMs trained on clean, fully-specified tool-calling benchmarks fail dramatically when deployed against production APIs. DiaFORGE solves this mismatch by generating realistic, ambiguous multi-turn dialogues where the agent must disambiguate between competing tools and ask clarifying questions—turning a brittle single-turn problem into a robust interactive one.

The gap between benchmark performance and production reality is massive: static benchmarks use fully-specified queries with one obvious tool, while real APIs have overlapping functionality, required parameters, and user confusion. DiaFORGE bridges this gap through synthetic dialogue generation that mimics real deployment dynamics.

Core Concept

The framework operates on the insight that realistic tool-calling is fundamentally interactive. Rather than expecting agents to infer all details from a single query, train them to:

1. Recognize ambiguity: Detect when multiple tools could satisfy the request
2. Ask clarifying questions: Engage users to disambiguate and collect missing parameters
3. Validate completeness: Ensure all required parameters are present before calling the tool
4. Recover from errors: Gracefully handle ambiguous or impossible requests

This interactive approach reduces both false positives (wrong tool calls) and false negatives (refusing to act).

Architecture Overview

• UTC-Gen engine: Multi-agent system generating realistic multi-turn dialogues with ground-truth tools and semantic distractors
• Dialogue structure: Simulated user gradually reveals information while assistant asks clarifying questions
• Validation layer: Rule-based and LLM-based checks ensure generated data is correct
• Dynamic evaluation: Interactive rollout where trained agent engages with user simulator (not just static metrics)
• Parameter handling: Explicit tracking of required vs. provided parameters across turns

Implementation

Generate synthetic dialogues using the UTC-Gen multi-agent system. The engine seeds conversations with a ground-truth tool and includes semantic "distractor" tools:

from diaforge.generator import UTCGen

gen = UTCGen(api_catalog="enterprise_apis.json")

# Seed dialogue with ground-truth tool and distractors
dialogue = gen.generate_dialogue(
    ground_truth_tool="CRM.CreateLead",
    # Similar tools to create disambiguation need
    distractors=[
        "CRM.UpdateLead",
        "CRM.CreateContact",
        "Sales.CreateOpportunity"
    ],
    # Define what information is revealed per turn
    information_stages=[
        {"turn": 1, "revealed": ["customer_name"]},
        {"turn": 2, "revealed": ["email", "phone"]},
        {"turn": 3, "revealed": ["industry", "company_size"]}
    ]
)

# Output: multi-turn conversation where assistant must ask
# clarifying questions to determine correct tool
print(dialogue)
# User: "Add John to our system"
# Assistant: "Is John a prospect or an existing customer?"
# User: "He's a new prospect we want to track"
# Assistant: "I'll create him as a new lead. What's his email?"

Fine-tune an open-source model (like Llama) on the generated dialogues. Use supervised learning with loss masking to focus on assistant responses:

from diaforge.training import ToolCallingTrainer

trainer = ToolCallingTrainer(model="meta-llama/Llama-2-7b-chat")

# Load generated dialogues
dialogues = load_synthetic_data("generated_dialogues.jsonl")

# Fine-tune focusing only on assistant turns
trainer.train(
    dialogues=dialogues,
    loss_mask="assistant_only",  # Only optimize assistant responses
    epochs=3,
    batch_size=32,
    learning_rate=2e-5
)

Evaluate models using dynamic evaluation where they interact with a user simulator:

from diaforge.evaluation import DynamicEvaluator

evaluator = DynamicEvaluator(
    user_simulator="interactive",  # Simulated user responds to agent questions
    api_catalog="enterprise_apis.json"
)

# Full interactive rollout: agent picks tool, gets feedback
results = evaluator.evaluate(
    model=trained_model,
    test_scenarios=test_dialogues,
    metrics={
        "tool_accuracy": "Did agent pick the right tool?",
        "false_positives": "Any incorrect tool calls?",
        "abstention_rate": "When does agent refuse to act?",
        "success_rate": "Did tool call complete the user's goal?"
    }
)

print(f"Tool accuracy: {results['tool_accuracy']:.1%}")
print(f"Success rate: {results['success_rate']:.1%}")

Practical Guidance

When to Use DiaFORGE

Use this approach when:

• Deploying agents against enterprise API catalogs with overlapping functionality
• Real-world tool calls arrive incomplete or ambiguous
• False positives (wrong tools) are dangerous or costly
• Users expect the agent to ask clarifying questions
• You need to reduce training data by generating realistic scenarios

When NOT to Use

Avoid this for:

• Single well-defined tool domains (calculator, weather API)
• Scenarios where user interaction is not possible
• Tasks with perfectly unambiguous specifications
• Environments where asking clarifying questions creates friction

Training Data Strategy

| Scenario | Distractors | Difficulty | |----------|------------|-----------| | Simple disambiguation | 2-3 similar tools | Baseline | | Semantic overlap | 4-5 tools with overlapping capabilities | Medium | | Parameter interdependence | Tools where required params depend on previous choices | Hard | | Multi-step resolution | Requires 3+ clarifying questions to fully specify | Expert |

Generate ~5,000 dialogues mixing all difficulty levels for robust coverage.

Critical Metrics

1. Tool accuracy: Correct tool selected at least 90% of the time
2. False positive rate: Wrong tools called <5% of time
3. Abstention rate: Agent refuses to act when truly ambiguous
4. Success rate: Final tool call achieves user's goal
5. Interaction cost: Average turns to complete vs. human baseline

The paper reports DiaFORGE-trained models achieve 27 percentage points improvement over GPT-4o and 49 points over Claude-3.5-Sonnet on dynamic evaluation.

Common Pitfalls

1. Generating perfect dialogues: Real users are messy. Add typos, false starts, and unclear requests.
2. Over-using static metrics: Agent performance on isolated queries ≠ performance in interactive settings.
3. Missing parameter validation: Track required vs. optional parameters explicitly; don't assume models infer this.
4. Ignoring failure modes: Test what happens when the agent genuinely cannot disambiguate.
5. Forgetting context length: Enterprise APIs are verbose; ensure model can track state across 5+ turns.

Reference

"Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Risky" - arXiv:2507.03336