ADu2021/eaglet-planner

EAGLET: Efficient Planner Training via Synthetic Plan Generation

Long-horizon agent planning requires expensive manual annotation or extensive RL training. EAGLET generates synthetic high-quality plans from advanced LLMs without manual labeling, then refines them with specialized RL using executor feedback signals.

Core insight: strong planning comes from both good initial plans and learning from execution signals. By bootstrapping from LLM-generated plans and using executor capability as reward signal, you achieve sample-efficient training at 8x lower cost than traditional RL.

Core Concept

Homologous Consensus Filtering: Generate multiple candidate plans from a strong LLM and keep only those with high consensus across samples. This automatically filters out hallucinated or low-quality plans without human annotation.

Executor Capability Gain Reward: Reward signal based on whether executing the plan reveals new agent capabilities, not just whether tasks succeed. Encourages plans that push agent boundaries.

Architecture Overview

• Plan Synthesizer: Advanced LLM (GPT-4/Claude) generates diverse candidate plans
• Consensus Filter: Scores plans by agreement across multiple samples and baseline quality
• Executor Simulator: Simulates plan execution to estimate feasibility
• RL Trainer: Refines planner with capability-gain rewards

Implementation Steps

Stage 1: Synthetic Plan Generation with Consensus Filtering

Generate plans from strong LLM and filter by consensus:

def generate_consensus_plans(task, num_candidates=10, strong_llm='gpt-4'):
    """
    Generate diverse plans and keep only consensus-agreed ones.

    Args:
        task: description of agent task
        num_candidates: how many plan samples to generate
        strong_llm: which model to use for generation

    Returns:
        filtered_plans: high-confidence plans
    """

    # Generate diverse candidates
    candidates = []
    for i in range(num_candidates):
        temperature = 0.8  # Encourage diversity

        prompt = f"""
        For the task: {task}

        Generate a detailed step-by-step plan.
        Consider:
        - Necessary preconditions
        - High-level milestones
        - Dependency ordering
        - Executor constraints
        """

        plan = strong_llm.generate(
            prompt,
            temperature=temperature,
            max_tokens=1024
        )
        candidates.append(plan)

    # Compute consensus scores
    consensus_scores = compute_plan_similarity(candidates)

    # Filter high-consensus plans
    threshold = np.percentile(consensus_scores, 40)  # Keep top 60%
    filtered_plans = [
        candidates[i] for i in range(len(candidates))
        if consensus_scores[i] >= threshold
    ]

    return filtered_plans

def compute_plan_similarity(plans):
    """
    Score plans by how similar they are to other plans.
    High similarity = high confidence.
    """
    scores = []
    for i, plan_i in enumerate(plans):
        # Extract key steps from plan
        steps_i = extract_plan_steps(plan_i)

        similarities = []
        for j, plan_j in enumerate(plans):
            if i != j:
                steps_j = extract_plan_steps(plan_j)
                sim = compute_step_similarity(steps_i, steps_j)
                similarities.append(sim)

        # Score = average similarity to other plans
        scores.append(np.mean(similarities))

    return scores

Stage 2: Cold Start Fine-tuning

Use filtered plans to bootstrap the agent planner:

def cold_start_finetuning(
    planner_model,
    filtered_plans,
    task_descriptions,
    num_epochs=3
):
    """
    Initial fine-tuning on high-quality synthetic plans.
    Acts as warm start before RL refinement.
    """

    optimizer = torch.optim.AdamW(
        planner_model.parameters(),
        lr=5e-5
    )

    for epoch in range(num_epochs):
        for task, plan in zip(task_descriptions, filtered_plans):
            # Tokenize task and plan
            task_tokens = tokenize(task)
            plan_tokens = tokenize(plan)

            # Teacher forcing: model learns to generate plan given task
            logits = planner_model(task_tokens)

            # Compute cross-entropy loss
            loss = F.cross_entropy(
                logits.view(-1, vocab_size),
                plan_tokens.view(-1)
            )

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

    return planner_model

Stage 3: Executor Capability Reward Training

Refine planner using execution feedback:

def rl_training_with_capability_reward(
    planner_model,
    executor_agent,
    task_distribution,
    num_steps=5000
):
    """
    Train planner with rewards based on executor learning progress.
    Capability gain = new skills discovered through plan execution.
    """

    optimizer = torch.optim.AdamW(
        planner_model.parameters(),
        lr=1e-5
    )

    # Track executor capabilities
    initial_skills = executor_agent.get_skill_set()

    for step in range(num_steps):
        # Sample task
        task = sample_from(task_distribution)

        # Generate plan
        plan = planner_model.generate(
            task,
            max_length=512,
            temperature=0.7
        )

        # Execute plan
        execution_result = executor_agent.execute_plan(plan, task)

        # Compute rewards
        success_reward = 1.0 if execution_result['success'] else 0.0

        # Capability gain: what new skills were used?
        skills_used = execution_result['skills_used']
        new_skills = skills_used - initial_skills
        capability_reward = len(new_skills) * 0.5

        # Total reward: success + exploration bonus
        total_reward = success_reward + capability_reward

        # Policy gradient update
        log_prob = compute_log_probability(planner_model, plan)
        loss = -(log_prob * total_reward)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # Update skill tracking
        if new_skills:
            initial_skills.update(new_skills)

Practical Guidance

When to Use EAGLET:

• Long-horizon tasks (10+ steps) with limited labeled plan data
• When you have access to strong LLM for synthetic plan generation
• Tasks where executor capability is measurable/observable

When NOT to Use:

• Short tasks (1-3 steps) where manual annotation is simple
• Domain without clear skill/capability progression
• Executors where capability feedback is hard to define

Plan Generation Tips:

| Strategy | Best For | Tradeoff | |----------|----------|----------| | High temperature (0.9-1.0) | Diverse candidates | More filtering needed | | Low temperature (0.5-0.6) | Focused candidates | Less coverage of solution space | | Consensus threshold 40% | Inclusive filtering | More noisy plans included | | Consensus threshold 60% | Strict filtering | Fewer plans, higher quality |

Common Pitfalls:

• Generating too few candidates (reduce diversity benefits)
• Consensus score overly permissive (include bad plans)
• Executor capability too strict (hard to satisfy)
• Not tracking skill progression (lose learning signal)

Reference

Based on the research at: https://arxiv.org/abs/2510.05608