ADu2021/cove-tool-use-training

Technique: Constraint-Guided Verification for Tool-Use Agent Training

Training agents to use tools (APIs, functions, domain-specific commands) is notoriously difficult. The challenge: creating diverse, realistic interaction trajectories where agents navigate complex, ambiguous user requests through deterministic actions is expensive and error-prone. Manual curation scales poorly, and unconstrained data synthesis produces trajectories that violate domain logic or are unrealistic.

CoVe solves this by embedding explicit task constraints (business rules, domain requirements) directly into the data synthesis process. Constraints serve dual purposes: (1) guiding generation of sophisticated, realistic trajectories, and (2) providing deterministic verification that outputs are correct. This eliminates manual annotation while ensuring data quality.

Core Concept

The core insight: constraints are semantic specifications that can guide both generation and validation. Rather than generating trajectories unconstrained and hoping they're correct, define constraints that the agent must satisfy (e.g., "booking a flight must include selecting date, passengers, and payment"). Use these constraints to:

1. Guide generation: LLM generates trajectories that respect constraints
2. Verify correctness: Automatically check that trajectories satisfy all constraints
3. Train agents: High-quality data with verified correctness enables effective SFT and RL

This creates a virtuous cycle: better constraints yield better data, which trains better agents.

Architecture Overview

• Constraint Definition: Specify task requirements (preconditions, actions, postconditions)
• Trajectory Generation: LLM generates interaction sequences respecting constraints
• Constraint Verification: Deterministic validator confirms trajectories satisfy all constraints
• Training Data: Filtered, verified trajectories for SFT and RL
• Agent Training: Standard supervised fine-tuning on verified trajectories

Implementation Steps

CoVe involves defining constraints, generating trajectories, and training agents. Here's how to implement it:

Define explicit constraints that specify valid task trajectories. Constraints encode domain logic:

from dataclasses import dataclass
from typing import List, Dict, Any

@dataclass
class Constraint:
    """Base class for task constraints."""
    name: str
    description: str

    def verify(self, trajectory: List[Dict]) -> bool:
        """Check if trajectory satisfies this constraint."""
        raise NotImplementedError

class AirlineBookingConstraints:
    """Constraints for airline booking tasks."""

    class HasFlight(Constraint):
        def __init__(self):
            super().__init__(
                "has_flight",
                "Trajectory must include flight selection action"
            )

        def verify(self, trajectory: List[Dict]) -> bool:
            actions = [step.get('action') for step in trajectory]
            return 'select_flight' in actions or 'confirm_flight' in actions

    class HasPassengers(Constraint):
        def __init__(self):
            super().__init__(
                "has_passengers",
                "Must specify passenger details"
            )

        def verify(self, trajectory: List[Dict]) -> bool:
            for step in trajectory:
                if step.get('action') == 'enter_passenger_info':
                    return 'passengers' in step and len(step['passengers']) > 0
            return False

    class ValidPayment(Constraint):
        def __init__(self):
            super().__init__(
                "valid_payment",
                "Payment must be processed"
            )

        def verify(self, trajectory: List[Dict]) -> bool:
            for step in trajectory:
                if step.get('action') == 'process_payment':
                    payment = step.get('payment_method')
                    return payment in ['credit_card', 'debit_card', 'wallet']
            return False

    def __init__(self):
        self.constraints = [
            self.HasFlight(),
            self.HasPassengers(),
            self.ValidPayment(),
        ]

    def verify_trajectory(self, trajectory: List[Dict]) -> bool:
        """Check if trajectory satisfies all constraints."""
        return all(c.verify(trajectory) for c in self.constraints)

    def get_constraint_prompt(self) -> str:
        """Generate prompt guidance for trajectory generation."""
        prompt = "Generate a valid airline booking trajectory that:\n"
        for i, constraint in enumerate(self.constraints, 1):
            prompt += f"{i}. {constraint.description}\n"
        return prompt

Generate trajectories using an LLM, guided by constraints:

def generate_tool_trajectories(
    model,
    constraints,
    num_trajectories=100,
    temperature=0.8,
):
    """
    Generate synthetic tool-use trajectories respecting constraints.
    """
    constraint_guidance = constraints.get_constraint_prompt()

    prompt_template = f"""
    {constraint_guidance}

    Generate a realistic user request and corresponding agent interaction trajectory.
    The trajectory should show the agent using tools to complete the task.

    Format:
    USER_REQUEST: [User's initial request]
    TRAJECTORY:
    [Step 1]: action=..., parameters={{...}}
    [Step 2]: action=..., parameters={{...}}
    ...
    CONFIRMATION: [Final booking/result]
    """

    trajectories = []
    verified_count = 0

    for _ in range(num_trajectories):
        # Generate trajectory
        generation = model.generate(
            prompt_template,
            max_length=500,
            temperature=temperature,
            num_return_sequences=1,
        )[0]

        # Parse trajectory
        parsed = parse_trajectory_output(generation)

        # Verify constraints
        if constraints.verify_trajectory(parsed['steps']):
            trajectories.append(parsed)
            verified_count += 1

    print(f"Generated {verified_count}/{num_trajectories} valid trajectories")
    return trajectories

def parse_trajectory_output(text: str) -> Dict:
    """Parse model output into structured trajectory."""
    lines = text.split('\n')
    trajectory = {
        'request': '',
        'steps': [],
        'confirmation': ''
    }

    current_section = None
    for line in lines:
        if 'USER_REQUEST:' in line:
            trajectory['request'] = line.split('USER_REQUEST:')[1].strip()
            current_section = 'request'
        elif 'TRAJECTORY:' in line:
            current_section = 'steps'
        elif 'CONFIRMATION:' in line:
            trajectory['confirmation'] = line.split('CONFIRMATION:')[1].strip()
            current_section = 'confirmation'
        elif current_section == 'steps' and line.strip():
            # Parse step: [Step N]: action=..., parameters={...}
            step_data = parse_step_line(line)
            trajectory['steps'].append(step_data)

    return trajectory

def parse_step_line(line: str) -> Dict:
    """Parse individual trajectory step."""
    import re
    # Example: [Step 1]: action=select_flight, parameters={'flight_id': 'AA123'}
    match = re.search(r'action=(\w+),\s*parameters=(\{.*\})', line)
    if match:
        return {
            'action': match.group(1),
            'parameters': eval(match.group(2))  # In practice, use safer parsing
        }
    return {}

Create training data from verified trajectories:

def create_training_data(trajectories, constraints):
    """
    Convert verified trajectories into supervised fine-tuning examples.
    """
    training_examples = []

    for trajectory in trajectories:
        # Verify one more time before adding to training
        if not constraints.verify_trajectory(trajectory['steps']):
            continue

        # Create prompt-response pairs
        user_request = trajectory['request']

        # Multi-turn conversation: user request + agent actions
        conversation = [
            {'role': 'user', 'content': user_request}
        ]

        for step in trajectory['steps']:
            action_str = f"Action: {step['action']}\n"
            params_str = f"Parameters: {step['parameters']}"
            conversation.append({
                'role': 'assistant',
                'content': f"{action_str}{params_str}"
            })

        training_examples.append({
            'conversation': conversation,
            'trajectory': trajectory,
            'verified': True
        })

    return training_examples

# Usage
constraints = AirlineBookingConstraints()
trajectories = generate_tool_trajectories(
    model,
    constraints,
    num_trajectories=1000
)
training_data = create_training_data(trajectories, constraints)

Train agent on verified trajectories:

def train_tool_use_agent(
    model,
    training_data,
    num_epochs=3,
    learning_rate=1e-5,
):
    """
    Fine-tune model on constraint-verified tool-use data.
    """
    optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)

    for epoch in range(num_epochs):
        total_loss = 0.0

        for example in training_data:
            # Format conversation for training
            conversation_text = format_conversation(example['conversation'])

            # Standard language modeling loss
            outputs = model(conversation_text)
            loss = outputs.loss

            # Backward pass
            optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
            optimizer.step()

            total_loss += loss.item()

        avg_loss = total_loss / len(training_data)
        print(f"Epoch {epoch+1}: Loss = {avg_loss:.4f}")

    return model

def format_conversation(conversation):
    """Format multi-turn conversation for LLM fine-tuning."""
    formatted = ""
    for turn in conversation:
        role = turn['role'].upper()
        content = turn['content']
        formatted += f"{role}: {content}\n"
    return formatted

Practical Guidance

When to Use:

• Tool-use agents (booking, shopping, information retrieval)
• Domain-specific tasks with clear business rules
• When manual trajectory curation is expensive
• For production agent deployment

When NOT to Use:

• Open-ended generation tasks without clear constraints
• When defining comprehensive constraints is infeasible
• Real-time systems needing immediate deployment (generation takes time)

Constraint Design:

• Start with 3–5 core constraints covering essential requirements
• Make constraints verifiable (deterministic checks, no subjective judgment)
• Include preconditions (what must be true before task) and postconditions (what must be true after)
• Test constraints on manually curated examples first

Generation and Filtering:

• Generate 5–10x more trajectories than you need to account for verification failures
• Monitor verification success rate; if <50%, constraints may be too strict
• Increase temperature slightly (0.7–0.9) to encourage diverse trajectories

Training:

• Use standard SFT first, then optional RL for refinement
• 4B parameter models can achieve competitive results with ~4000 verified examples
• Test on target tasks to ensure agent learns tool-use patterns

Results:

• Airline booking: 43% success (4B model vs. larger baselines)
• Retail tasks: 59.4% success
• Competitive with models 17x larger when using constraint-verified data

---

Reference: CoVe: Training Interactive Tool-Use Agents via Constraint-Guided Verification