curiositech/wang-2023-voyager

VOYAGER: Open-Ended Learning Through Autonomous Exploration

Decision Points

Task Difficulty Assessment → Curriculum Action

IF proposed task requires skills/items agent doesn't have:

• AND prerequisites are 1-2 steps away → Accept task, decompose into subtasks
• AND prerequisites are 3+ steps away → Reject, request easier task
• AND prerequisites are unclear → Request task with more specific requirements

IF proposed task is similar to recently completed tasks:

• AND success rate on similar tasks >80% → Request harder variation
• AND success rate 50-80% → Accept task (skill consolidation zone)
• AND success rate <50% → Request easier task

IF task involves completely new domain (new biome/tool/mechanic):

• AND agent has foundational skills → Accept, expect high iteration count
• AND agent lacks foundations → Reject, build prerequisites first

Skill Retrieval → Context Selection

IF current task has clear semantic match in library (similarity >0.85):

• Use top 3 matches as primary context
• Add 1-2 compositional skills that were used together previously

IF current task has partial matches (similarity 0.6-0.85):

• Use top 1 exact match + 2-3 related skills
• Include error patterns from previous failures on similar tasks

IF current task has no good matches (similarity <0.6):

• Use foundational skills (movement, inventory management, basic crafting)
• Include domain-specific APIs relevant to task context

Code Generation Failure → Refinement Strategy

IF code has syntax/runtime errors:

• Use error message + stack trace as refinement prompt (1-2 iterations usually sufficient)

IF code runs but fails verification:

• Compare expected vs actual state changes
• Use environment feedback in next iteration prompt
• IF same failure repeats 2+ times → Add explicit debugging prints

IF code times out or loops infinitely:

• Add timeout guards and progress checks
• Break complex operations into smaller steps
• IF still failing → Decompose task into subtasks

Failure Modes

Anti-Pattern 1: "Curriculum Treadmill"

Symptoms: Agent gets stuck proposing/failing same difficulty tasks repeatedly
Detection: Success rate flat for 10+ tasks, no new skills added to library
Fix: Force curriculum to propose easier tasks to rebuild confidence, or harder tasks to break through plateau

Anti-Pattern 2: "Context Overflow"

Symptoms: Code generation degrades as library grows, LLM context filled with irrelevant skills
Detection: Recent success rate declining despite library growth, retrieval returning low-similarity matches
Fix: Improve semantic indexing, add recency weighting to retrieval, compress old skills into documentation

Anti-Pattern 3: "Infinite Refinement"

Symptoms: Agent spends 4+ iterations on tasks that should succeed in 1-2 attempts
Detection: High iteration count with repeated similar errors, no progress between attempts
Fix: Better error categorization, early termination for unsolvable tasks, task decomposition

Anti-Pattern 4: "Skill Hoarding"

Symptoms: Library fills with hyper-specific skills that never get reused
Detection: Low skill reuse rate, many skills with usage_count=1
Fix: Encourage more general skill patterns, merge similar skills, add skill cleanup process

Anti-Pattern 5: "Verification Drift"

Symptoms: Agent claims success for tasks that obviously failed
Detection: Self-verification approval rate >90% but manual inspection shows failures
Fix: Add objective success criteria, cross-check with environment state, improve verification prompts

Worked Examples

Example: "Smelt Iron Ore" Task Walkthrough

Initial State: Agent has iron ore in inventory, knows location of coal

Step 1: Task Proposal

• Curriculum agent checks inventory, sees iron ore
• Proposes "smelt 8 iron ore into iron ingots" (appropriately challenging)
• Difficulty assessment: ACCEPT (has materials, understands smelting concept)

Step 2: Skill Retrieval

• Query library for "smelt" → finds smeltCoal() (similarity 0.72)
• Query for "furnace" → finds lightFurnace(), collectFromFurnace() (similarity 0.68, 0.64)
• Context: Include these 3 skills + basic inventory management

Step 3: Code Generation (Iteration 1)

async function smeltIronOre(bot) {
    // Generated code tries to use furnace without coal
    await bot.pathfinder.goto(...furnaceLocation);
    await bot.clickWindow(bot.currentWindow.slots[0]); // Place iron ore
    // Missing: check for coal, light furnace
}

Step 4: Execution + Feedback

• Environment: "Furnace not lit, smelting failed"
• Self-verification: "I placed iron ore but furnace didn't start smelting"

Step 5: Code Generation (Iteration 2)

async function smeltIronOre(bot) {
    await collectCoal(bot); // Reused skill
    await bot.pathfinder.goto(...furnaceLocation);
    await lightFurnace(bot); // Reused skill
    // Place iron ore and coal in correct slots
    await waitForSmelting(bot);
}

Step 6: Success + Library Addition

• Task succeeds, code added to library with embedding
• Curriculum notes success, proposes next challenge: "craft iron sword"

Expert vs Novice: Novice would retry without coal indefinitely; expert recognizes furnace lighting as prerequisite and reuses existing skills.

Example: "Build Castle" - Task Decomposition

Initial State: Agent has basic building blocks, intermediate construction skills

Step 1: Curriculum Reasoning

• Task too complex for single program
• Difficulty assessment: REJECT, request decomposition
• Counter-proposal: "build castle foundation 10x10"

Step 2: Skill Composition Pattern

• Foundation task uses digArea(), placeBlocks(), checkInventory()
• Success enables next subtask: "build castle walls"
• Each subtask builds on previous: foundation → walls → roof → details

Progressive Difficulty: Each castle component increases architectural complexity while reusing spatial reasoning skills.

Quality Gates

Task Completion Criteria:

• [ ] Code executes without runtime errors
• [ ] Environment state matches task requirements (verified objectively)
• [ ] Self-verification check passes with specific reasoning
• [ ] Resource usage is reasonable (no infinite loops/resource waste)
• [ ] Code reuses existing library skills where appropriate (>50% of functionality from library)

Skill Library Health:

• [ ] New skills added only after successful task completion
• [ ] Skill embeddings have semantic coherence (similar tasks cluster together)
• [ ] Library growth rate matches learning rate (new skills enable harder tasks)
• [ ] Skill reuse rate >30% (skills get used in multiple contexts)
• [ ] No dead code (skills unused for >50 tasks get flagged for cleanup)

System Progress Indicators:

• [ ] Success rate maintains 60-80% (difficulty calibrated correctly)
• [ ] Task diversity increasing over time (not stuck in local optima)
• [ ] Agent can complete tasks it couldn't attempt 20 tasks ago
• [ ] Curriculum proposals align with agent capability (not too easy/hard)
• [ ] Error recovery time decreasing (faster debugging through experience)

NOT-FOR Boundaries

Do NOT use VOYAGER for:

• Real-time control systems requiring <100ms response (use reactive policies instead)
• Safety-critical applications where code bugs cause physical harm (use formal verification instead)
• Environments without rich state feedback where verification is impossible (use reward-based RL instead)
• Single-task optimization where you want maximum performance on fixed objective (use specialized algorithms instead)
• Domains where LLMs lack knowledge like cutting-edge scientific equipment (use human experts instead)

Delegate to other approaches:

• For supervised learning with abundant data → use deep-learning-training.md
• For multi-agent coordination → use swarm-intelligence.md
• For real-time strategy → use mcts-planning.md
• For mathematical reasoning → use formal-verification.md
• For human-AI collaboration → use interactive-learning.md

VOYAGER excels at open-ended single-agent learning in environments with rich feedback, executable actions, and compositional task structure. Stay within these boundaries for best results.