ADu2021/arm-thinker-reward-modeling

Summary

ARM-Thinker introduces an agentic reward modeling framework that enables multimodal reward models to actively invoke external tools—image cropping, document retrieval, instruction validators—to ground judgments in verifiable evidence. Rather than passively scoring responses, the framework follows a "think-act-observe" loop with multi-stage reinforcement learning for adaptive tool-calling optimization.

Core Technique

Think-Act-Observe Loop: Reward models actively reason about what evidence is needed:

1. Think: Plan reasoning steps and identify required verification
2. Act: Call tools (crop image region, retrieve relevant document section)
3. Observe: Process tool outputs and base scores on verified facts

Multi-Stage GRPO: Two-phase reinforcement learning:

• Phase 1: Optimize tool-calling decisions via GRPO
• Phase 2: Refine final accuracy scoring on improved trajectories

Adaptive Reward Shaping: Progressively weight tool-calling versus accuracy optimization:

reward = λ_tool * tool_call_quality + λ_acc * accuracy_bonus

Implementation

Tool invocation architecture:

class ToolCallAgent:
    def __init__(self):
        self.planner = llm  # Plans tool calls
        self.tools = {
            'crop_image': crop_image_tool,
            'retrieve_docs': doc_retrieval_tool,
            'validate_instruction': instruction_validator
        }

    def think_act_observe(self, response, evidence):
        # Plan what to verify
        plan = self.planner(f"What should I verify about {response}?")

        # Execute tool calls
        observations = []
        for tool_call in parse_tool_calls(plan):
            tool_name, args = tool_call
            result = self.tools[tool_name](**args)
            observations.append((tool_name, result))

        return observations

Reward computation with evidence:

def compute_evidence_reward(response, observations, ground_truth):
    # Score based on verified facts
    accuracy = 0.0

    for tool_name, observation in observations:
        if tool_name == 'crop_image':
            # Check if crop correctly identifies object
            matches_gt = compare_visual_features(observation, ground_truth_region)
            accuracy += 0.3 * matches_gt

        elif tool_name == 'retrieve_docs':
            # Check if retrieval is relevant
            relevance = compute_relevance(observation, response)
            accuracy += 0.4 * relevance

        elif tool_name == 'validate_instruction':
            # Check if instruction follows constraints
            valid = instruction_is_valid(observation)
            accuracy += 0.3 * valid

    return accuracy

Multi-stage GRPO:

# Stage 1: Optimize tool selection
for step in range(num_steps_phase1):
    trajectory = agent.rollout(problem)
    tool_quality = evaluate_tool_calls(trajectory)
    agent.update(trajectory, reward=tool_quality)

# Stage 2: Refine accuracy
for step in range(num_steps_phase2):
    trajectory = agent.rollout(problem)
    accuracy = evaluate_response_accuracy(trajectory)
    agent.update(trajectory, reward=accuracy)

When to Use

• Multimodal evaluation requiring fine-grained verification
• Scenarios where reward models need to justify decisions with evidence
• Applications combining visual, textual, and document analysis
• Tasks requiring systematic grounding in verifiable facts

When NOT to Use

• Simple single-modality scoring without verification needs
• Real-time evaluation where tool latency is prohibitive
• Scenarios without access to multiple verification tools
• Applications where passive reward signals suffice

Key References

• Reward modeling and preference learning
• Tool-use and agent planning
• Multimodal evidence grounding
• Multi-stage reinforcement learning