ADu2021/agent-long-context-benchmark

Problem

Existing agent benchmarks typically use short, isolated tasks that don't reflect real-world requirements where agents must maintain context and reason over long interaction histories. Long-context agents need benchmarks that can systematically vary task complexity, history length, and environmental conditions to properly evaluate their capabilities.

Solution

Create AgentLongBench: a controllable benchmark framework that generates synthetic long-context agent tasks through environment rollouts. This approach allows you to:

1. Control Task Complexity: Vary the number of steps, branching decisions, and tool interactions
2. Simulate Extended Histories: Generate long chains of observations and actions that agents must reason over
3. Test Context Window Usage: Evaluate how agents handle increasing history lengths
4. Generate Diverse Scenarios: Create varied environmental states and task requirements programmatically

When to Use

• Evaluating long-context language models on agent tasks
• Testing agent memory and reasoning capabilities with extended histories
• Benchmarking context window limitations and efficiency
• Developing agents for complex multi-step workflows (code generation, research, planning)
• Analyzing how agent performance degrades with longer interaction histories

When NOT to Use

• Short, single-turn interaction tasks (standard benchmarks suffice)
• Real-world task evaluation (use human-curated benchmarks)
• Tasks requiring specialized domain knowledge (this is synthetic evaluation)

Implementation

Step 1: Define the Environment Simulator

Create a controllable environment that generates long-context agent interactions.

class LongContextEnvironment:
    """Simulation environment for generating long interaction sequences"""

    def __init__(self, complexity_level=1, max_steps=50):
        self.complexity = complexity_level  # 1-5 scale
        self.max_steps = max_steps
        self.state = self.initialize_state()
        self.step_count = 0
        self.history = []

    def initialize_state(self):
        """Set up initial task and environment state"""
        return {
            "goal": self.generate_goal(self.complexity),
            "environment_objects": self.generate_objects(self.complexity),
            "available_actions": self.available_actions(),
            "current_progress": 0.0
        }

    def step(self, agent_action):
        """
        Execute one agent action and return observation + reward
        Returns: (observation, reward, done, info)
        """
        self.step_count += 1

        # Validate and execute action
        is_valid = self.validate_action(agent_action)
        if not is_valid:
            observation = self.get_observation()
            return observation, -1.0, False, {"error": "invalid_action"}

        # Apply action to environment
        result = self.apply_action(agent_action)
        observation = self.get_observation()

        # Track in history
        self.history.append({
            "step": self.step_count,
            "action": agent_action,
            "result": result,
            "state_snapshot": self.state.copy()
        })

        # Compute reward
        progress_delta = self.get_progress() - self.state["current_progress"]
        reward = progress_delta * 10  # Scale reward signal
        self.state["current_progress"] += progress_delta

        done = (self.step_count >= self.max_steps) or self.goal_reached()

        return observation, reward, done, {"history_length": len(self.history)}

    def get_observation(self):
        """Return current observation including history summary"""
        return {
            "step": self.step_count,
            "goal": self.state["goal"],
            "visible_objects": self.get_visible_objects(),
            "recent_actions": self.history[-5:],  # Last 5 actions
            "progress_toward_goal": self.state["current_progress"],
            "available_actions": self.state["available_actions"]
        }

Step 2: Implement Task Generation

Create diverse tasks at varying complexity levels.

def generate_goal(complexity_level):
    """Generate task goal based on complexity"""
    if complexity_level == 1:
        return "Find and retrieve item X"
    elif complexity_level == 2:
        return "Complete sequence: find X, transform with Y, store in Z"
    elif complexity_level == 3:
        return "Multi-constraint planning: achieve A while maintaining B, avoid C"
    elif complexity_level == 4:
        return "Complex reasoning: infer hidden rules from observations, then achieve goal"
    else:
        return "Open-ended exploration and adaptation to changing goals"

def generate_environment_rollout(num_steps, complexity):
    """Generate a full trajectory of environment interactions"""
    env = LongContextEnvironment(complexity_level=complexity, max_steps=num_steps)
    trajectory = []

    observation = env.get_observation()
    done = False

    while not done and env.step_count < num_steps:
        # Simulate a sequence of reasonable actions
        action = select_trajectory_action(observation, env.state["goal"])
        observation, reward, done, info = env.step(action)

        trajectory.append({
            "observation": observation,
            "action": action,
            "reward": reward,
            "done": done
        })

    return {
        "trajectory": trajectory,
        "final_success": env.goal_reached(),
        "steps_taken": env.step_count,
        "context_length": len(env.history)
    }

Step 3: Build the Benchmark Dataset

Generate a collection of tasks at different complexity and length levels.

class AgentLongBench:
    """Full benchmark for long-context agent evaluation"""

    def __init__(self, num_tasks_per_level=10, max_context_length=1000):
        self.num_tasks = num_tasks_per_level
        self.max_context_length = max_context_length
        self.tasks = self.generate_benchmark()

    def generate_benchmark(self):
        """Create diverse tasks across complexity and context length dimensions"""
        benchmark = {
            "easy": [],
            "medium": [],
            "hard": [],
            "very_hard": [],
            "expert": []
        }

        # Easy: short context, simple goals
        for i in range(self.num_tasks):
            rollout = generate_environment_rollout(num_steps=10, complexity=1)
            benchmark["easy"].append(rollout)

        # Medium: longer context, multi-step goals
        for i in range(self.num_tasks):
            rollout = generate_environment_rollout(num_steps=30, complexity=2)
            benchmark["medium"].append(rollout)

        # Hard: long context, complex reasoning
        for i in range(self.num_tasks):
            rollout = generate_environment_rollout(num_steps=50, complexity=3)
            benchmark["hard"].append(rollout)

        # Very Hard: very long context with constraints
        for i in range(self.num_tasks):
            rollout = generate_environment_rollout(num_steps=100, complexity=4)
            benchmark["very_hard"].append(rollout)

        # Expert: longest context, open-ended adaptation
        for i in range(self.num_tasks):
            rollout = generate_environment_rollout(
                num_steps=min(150, self.max_context_length),
                complexity=5
            )
            benchmark["expert"].append(rollout)

        return benchmark

    def evaluate_agent(self, agent, difficulty="easy"):
        """
        Run agent on benchmark tasks and compute metrics
        """
        tasks = self.tasks[difficulty]
        results = {
            "success_rate": 0.0,
            "avg_steps": 0.0,
            "avg_history_length": 0.0,
            "context_efficiency": 0.0
        }

        successes = 0
        total_steps = 0
        total_history_length = 0

        for task in tasks:
            # Reset agent for task
            agent.reset()

            # Run agent through task
            trajectory = task["trajectory"]
            task_success = False

            for step_data in trajectory:
                observation = step_data["observation"]
                action = agent.act(observation)
                # (evaluate action against expected behavior)
                if action == step_data["action"]:
                    task_success = True

            if task_success:
                successes += 1

            total_steps += task["steps_taken"]
            total_history_length += task["context_length"]

        results["success_rate"] = successes / len(tasks)
        results["avg_steps"] = total_steps / len(tasks)
        results["avg_history_length"] = total_history_length / len(tasks)
        results["context_efficiency"] = results["success_rate"] / (
            results["avg_history_length"] / self.max_context_length
        )

        return results

Step 4: Integrate with Agent Training Loop

Use the benchmark for iterative agent improvement.

def train_agent_with_benchmark(agent, benchmark, num_epochs=10):
    """
    Train agent using AgentLongBench with curriculum learning
    Start with easy tasks, progress to harder ones
    """
    difficulties = ["easy", "medium", "hard", "very_hard", "expert"]

    for epoch in range(num_epochs):
        for difficulty in difficulties:
            # Evaluate on current difficulty
            metrics = benchmark.evaluate_agent(agent, difficulty)

            # Stop if success rate drops too low
            if metrics["success_rate"] < 0.3:
                continue

            # Train agent on this difficulty level
            agent.train_on_tasks(benchmark.tasks[difficulty], epochs=1)

            print(f"Epoch {epoch}, Difficulty {difficulty}: "
                  f"Success={metrics['success_rate']:.2f}, "
                  f"ContextEff={metrics['context_efficiency']:.2f}")

Key Insights

• Controllability: Environment rollouts allow precise control over task difficulty and context length
• Curriculum Learning: Start with easy tasks, progress systematically to harder ones
• Context Efficiency: Measure not just success but how efficiently agents use available context window
• Scalability: Benchmark can generate unlimited synthetic tasks at any complexity level

Benchmark Dimensions

• Complexity: 1-5 scale from simple retrieval to open-ended adaptation
• Context Length: 10-1000+ steps, testing how agents handle extended histories
• Task Types: Retrieval, transformation, multi-step planning, constrained reasoning, open exploration

References

• arXiv:2601.20730: AgentLongBench framework and evaluation results
• Controllable generation of long-context agent evaluation tasks