agentscope-ai/rl-reward

RL Reward Construction with OpenJudge

Build reward signals for reinforcement learning from human feedback (RLHF) and reinforcement learning from AI feedback (RLAIF) using the openjudge library.

When to Use This Skill

• Building scalar rewards for GRPO / REINFORCE rollout scoring
• Generating (chosen, rejected) preference pairs for DPO / IPO
• Best-of-N candidate selection
• Multi-dimensional reward shaping (correctness + safety + format)
• Replacing or bootstrapping a reward model with LLM-as-judge

Step 1 — Choose Your Reward Strategy

Use this decision tree before writing any code:

RL Algorithm + Task type?
│
├── GRPO / REINFORCE — Verifiable task (math, code, structured output)
│   └── → POINTWISE  ✅  (FunctionGrader, exact score, zero LLM cost)
│
├── GRPO / REINFORCE — Subjective task (instruction following, dialogue, summarization)
│   └── → PAIRWISE TOURNAMENT  ✅  (compare each rollout vs all others in group,
│                                    reward = net win rate within group)
│
├── DPO / IPO / SLiC — need (chosen, rejected) pairs
│   └── → PAIRWISE  ✅  (two-way comparison, return winner/loser)
│
└── Best-of-N / reranking — rank N candidates
    └── → LISTWISE  ✅  (single call ranks all N at once)

Cost constraint?
├── Low budget
│   └── FunctionGrader (free) → pointwise; or pairwise with small judge model
│
├── Medium budget
│   └── Pointwise: 2–3 LLM graders + WeightedSumAggregator
│   └── Pairwise tournament: 1 LLM judge, N*(N-1)/2 comparisons per group
│
└── High quality / no cost limit
    └── Pointwise voting (3–5 calls) or pairwise with strong judge + debiasing

Sub-documents — Read When Relevant

| Topic | File | Read when… | |-------|------|------------| | Pointwise multi-dim reward | pointwise.md | GRPO on verifiable tasks; multi-dimension scoring | | Pairwise reward | pairwise.md | GRPO on subjective tasks (tournament); DPO/RLAIF preference pairs |

Read the relevant sub-document before writing any code.

Install

pip install py-openjudge

Strategy Comparison

| Strategy | Output | Reward signal | Typical use | Cost | |----------|--------|---------------|-------------|------| | Pointwise | scalar per response | direct reward r(x, y) | GRPO on verifiable tasks, filtering | Low–Medium | | Pairwise Tournament | net win rate per response | relative reward within group | GRPO on subjective tasks | Medium (N²/2 calls) | | Pairwise | winner/loser pair | implicit preference y+ > y- | DPO, IPO, RLAIF preference data | Medium | | Listwise | rank over N responses | ordinal reward / reranking | Best-of-N, reranking | Medium–High |

Score Normalization

All graders return scores on different scales. Always normalize before feeding into RL:

def normalize(score: float, min_score: float, max_score: float) -> float:
    """Map [min_score, max_score] → [0.0, 1.0]."""
    if max_score == min_score:
        return 0.0
    return (score - min_score) / (max_score - min_score)

# LLM graders (common/*) return 1–5 → normalize to 0–1
reward = normalize(result.score, min_score=1, max_score=5)

# FunctionGrader / text graders already return 0–1 → no normalization needed

Evaluation Strategies

Evaluation strategies control how many times a grader is called and how results are aggregated. They are independent of the grader itself.

Choose Your Strategy

Grader type?
│
├── Deterministic (FunctionGrader, StringMatch, CodeExecution, etc.)
│   └── → Direct  (zero variance, no need for aggregation)
│
├── LLM grader — Pointwise scoring
│   │
│   ├── Budget limited / speed critical
│   │   └── → Direct  (accept variance, 1× cost)
│   │
│   ├── Discrete scores (1–5 integer, pass/fail, binary)
│   │   └── → Voting  (majority vote, robust to outliers, N× cost)
│   │
│   └── Continuous / fine-grained scores (need precise ranking)
│       └── → Average  (mean, preserves signal, N× cost)
│
└── LLM grader — Pairwise GRPO tournament
    └── → GRPOTournament  (all-pairs comparison, net win rate)

| Strategy | Aggregation | Best for | Cost | |----------|-------------|----------|------| | DirectEvaluationStrategy | None | Deterministic graders; low budget | 1× | | VotingEvaluationStrategy | Majority vote | Discrete / integer LLM scores | N× | | AverageEvaluationStrategy | Mean | Continuous LLM scores | N× | | GRPOTournamentEvaluationStrategy | Net win rate | Pairwise GRPO on subjective tasks | N²/2× |

All strategies are imported from openjudge.evaluation_strategy.

Pointwise — Noise Reduction with Voting / Average

For high-variance LLM judges, wrap any grader with VotingEvaluationStrategy to run N calls and take the majority vote:

from openjudge.evaluation_strategy import VotingEvaluationStrategy

grader = CorrectnessGrader(
    model=model,
    strategy=VotingEvaluationStrategy(num_votes=3, tie_breaker="closest_to_mean"),
)
# Now each call internally runs 3 LLM evaluations and returns the most common score

Use odd num_votes (3, 5) to avoid ties.

Pairwise — GRPO Tournament

For GRPO on subjective tasks, use GRPOTournamentEvaluationStrategy to run all-pairs comparison and compute net win rate per rollout:

from openjudge.evaluation_strategy import GRPOTournamentEvaluationStrategy

strategy = GRPOTournamentEvaluationStrategy(debiased=False)
results = await strategy.execute(
    pairwise_grader.aevaluate,
    query="Write a haiku about the ocean.",
    responses=["rollout_1", "rollout_2", "rollout_3", "rollout_4"],
)
rewards = [r.score for r in results]  # net win rates in [-1.0, 1.0]

Set debiased=True to run each pair in both orders and only count consistent results (doubles LLM calls but mitigates position bias).