ADu2021/deep-agent-reasoning

DeepAgent: Unified Autonomous Reasoning with Tool Learning

Existing reasoning agents struggle with two key limitations: they accumulate errors across long-horizon tasks through verbose interaction histories, and they require task-specific tool interfaces rather than learning generalizable tool use patterns.

DeepAgent solves this by integrating autonomous thinking, tool discovery, and action execution into a single end-to-end reasoning process. The system combines memory compression with learned tool invocation, enabling agents to handle complex multi-step tasks efficiently.

Core Concept

DeepAgent operates through three integrated mechanisms:

• Autonomous Memory Folding: Compresses past interactions into structured episodic, working, and tool memories, reducing error propagation
• ToolPO (Tool Policy Optimization): An end-to-end RL strategy using simulated APIs and fine-grained tool-call advantage attribution
• Tool Retrieval: Handles both labeled-tool and open-set discovery scenarios

Architecture Overview

• Memory compression captures essential interaction patterns without verbose history
• Tool-call advantage attribution isolates credit signals to tool invocation tokens
• Memory types (episodic, working, tool) serve different reasoning stages
• End-to-end training enables discovery of effective tool combinations

Implementation Steps

The memory folding mechanism selectively summarizes interactions at each step. Rather than maintaining full conversation history, compress past state and actions into dense representations:

class MemoryFolder:
    def fold_interaction(self, history, current_state):
        # Compress episodic memory: factual outcomes from past steps
        episodic = self.compress_facts(history)
        # Working memory: intermediate reasoning state
        working = self.compress_reasoning(current_state)
        # Tool memory: effective tool patterns
        tools = self.extract_tool_patterns(history)
        return {episodic, working, tools}

    def compress_facts(self, history):
        # Extract key outcomes and state changes
        return [fact for fact in history if is_critical(fact)]

    def extract_tool_patterns(self, history):
        # Track which tools succeeded in which contexts
        return {(context, goal): tool for context, goal, tool in history}

ToolPO applies advantage attribution at the token level for tool calls. Rather than assigning credit to entire generation steps, focus reward signals on the tokens that invoke tools:

class ToolPO:
    def compute_advantage(self, trajectory, reward):
        # Identify tool-call tokens in the generation
        tool_tokens = [idx for idx, token in enumerate(trajectory)
                      if is_tool_invocation(token)]

        # Assign advantage only to tool-invocation tokens
        advantage = {}
        for idx in tool_tokens:
            # Fine-grained credit based on outcome
            advantage[idx] = compute_token_advantage(trajectory, idx, reward)

        return advantage

Practical Guidance

| Aspect | Recommendation | |--------|-----------------| | Memory compression ratio | 4:1 to 8:1 (reduce interaction sequences by 75-87%) | | Tool-call token weighting | 2-5x higher than other tokens during RL training | | Episodic memory retention | Keep last N=10 critical facts per domain | | Simulated API complexity | Match target environment sophistication |

When to use DeepAgent:

• Multi-step reasoning tasks requiring tool invocation
• Long-horizon problems where error accumulation matters
• Scenarios with large, diverse tool libraries to explore

When NOT to use:

• Single-step tasks without tool requirements
• Domains with strictly defined tool interfaces (use API-specific agents)
• Real-time systems where memory compression adds latency

Common pitfalls:

• Over-compressing memory and losing critical context
• Under-weighting tool-specific advantage signals
• Insufficient diversity in simulated API trajectories during training

Reference: DeepAgent on arXiv