wshobson/grpo-rlvr-training

GRPO & RLVR Training

This skill assumes finetuning-method-selection already routed here because the target behavior has a verifiable pass/fail signal — not demonstrations (lora-qlora-recipes) or preference pairs (preference-optimization). What follows is when RL is the right tool, the reference recipe, the mandatory reward-inspection gate, and how to pick a GRPO variant when the base recipe misbehaves.

Input: a routing decision (RLVR via GRPO) plus a verifier (code executor, test suite, schema checker, or grader) for the target task. Output format: a validated GRPO config — the kwarg values in references/grpo-memory.md and the reward functions in references/reward-functions.md, not free-form advice — that llm-finetuning-training-engineer consumes directly.

When RL Applies

GRPO+RLVR only pays off when task success is algorithmically checkable — a unit test passes, a parser accepts the output, a tool call matches an expected schema, a math answer matches a ground truth. If grading the output requires human judgment or a subjective rubric, that's an eval-harness and judge-calibration problem first — see eval-harness-first — not a reason to skip straight to RL.

Before opening a GRPO run, confirm the model can sometimes succeed on the target task already. RL sharpens an existing capability by reweighting toward the samples that already work; it does not install a capability from zero.

• The model never succeeds, even at low temperature across many samples: the gap is format or task understanding, not policy refinement. Route back to SFT first (lora-qlora-recipes) and only return to this skill once the base success rate is nonzero.
• The model succeeds sometimes, inconsistently: this is the GRPO sweet spot — proceed to The Recipe below.

The standing rule for the whole plugin: DPO for taste, GRPO for reasoning. If the signal is a preference between two acceptable outputs, that's preference-optimization, not this skill.

The Recipe

The reference recipe is TRL's GRPOTrainer with vLLM-backed generation:

from trl import GRPOConfig, GRPOTrainer

grpo_args = GRPOConfig(
    output_dir="./outputs-grpo",
    use_vllm=True,
    vllm_mode="colocate",       # single GPU; "server" for multi-GPU
    num_generations=8,          # floor — fewer starves the group-relative baseline
    learning_rate=5e-7,         # settled range for GRPO
    beta=0.01,                  # KL coefficient vs the reference policy
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,
    bf16=True,
    logging_steps=10,
    seed=3407,
)

trainer = GRPOTrainer(
    model=SFT_CHECKPOINT,
    args=grpo_args,
    reward_funcs=[format_reward, correctness_reward],   # references/reward-functions.md
    train_dataset=prompts,       # prompt-only — GRPO generates its own completions
    processing_class=tokenizer,
)

trainer.train()

• vllm_mode="colocate" runs generation and training on the same GPU — the default for a single-GPU box.
• vllm_mode="server" points at a separate vLLM server process and is the multi-GPU path — generation and training don't compete for the same device.
• num_generations ≥ 8 is a floor, not a suggestion: GRPO's advantage estimate is relative to the group mean, and fewer than 8 samples per prompt produces a noisy baseline.
• Reward is composite — a format reward (did the output parse / match the required structure) plus a correctness reward (did the answer verify). A well-formed-but-wrong answer and a malformed one should not score identically; correctness alone loses that signal.
• learning_rate=5e-7 and beta=0.01 are the settled starting point; deviate only after the base run is stable and reward-inspected (below).

Memory sizing for this recipe by target size class: references/grpo-memory.md.

The Inspection Rule

Run the reward function against 50–100 sampled outputs and manually read the results before starting the actual training run. This is a gate, not a one-time sanity check.

If the reward function's judgment disagrees with a human reading of that sample, fix the reward function first. Training against an uninspected reward, or tuning hyperparameters to compensate for one silently scoring the wrong thing, is how a run reward-hacks: the model optimizes cleanly toward the wrong target, and that doesn't surface as a training-loop bug.

This inspection is a Phase 1 gate input for /finetune — the same 50–100-sample read that catches a broken reward function here is what that command checks for before it lets a GRPO brief proceed.

Complete reward function implementations to inspect against — exact-match, schema-validation, unit-test-execution, a length-penalty wrapper, and a rubric-as-reward judge pattern: references/reward-functions.md.

Variant Selection

The base recipe above is the default. Reach for a variant only when a specific failure mode shows up, not preemptively:

| Failure mode | Variant | Why | |---|---|---| | Entropy collapse / degenerate long chain-of-thought | DAPO | Decouples clip bounds and relaxes the KL penalty that over-regularizes exploration on long reasoning traces | | Reward or output length trends up regardless of quality | Dr.GRPO | Removes GRPO's length-normalization bias so reward tracks correctness, not completion length | | Training a mixture-of-experts model | GSPO | Moves the importance-sampling ratio to the sequence level instead of per-token — per-token ratios are unstable on MoE routing, so GSPO is required here, not optional |

Start with plain GRPO. Watch for the specific symptom — collapsing entropy on long CoT, a length-reward correlation, or MoE instability — and only then swap in the matching variant above. Don't pre-select a variant before the base recipe has actually shown the failure mode.

VLM RL Is Reference-Only

Vision-language RL is not executed by this plugin in v1 — it's documented here for context, not as a runnable path. Tooling is fragmented across ms-swift and EasyR1-derived forks with no one-line TRL command yet, and naive text-only GRPO applied to a VLM tends to reward-hack by optimizing the text-reasoning trace while ignoring the image — the model learns to sound right without looking at the input. A VLM RL run is a research spike outside this skill's supported recipe, not a variant of The Recipe above.

References

• references/reward-functions.md — complete Python reward functions (exact-match correctness, schema validation, unit-test execution, a length-penalty wrapper, and a rubric-as-reward judge pattern) to inspect under The Inspection Rule before any training run.
• references/grpo-memory.md — memory sizing by target size class, vLLM sleep-mode and optimizer-state tactics, Unsloth's long-context RL chunking, and the DGX Spark bandwidth caveat for decode-heavy rollouts.

Related skills: finetuning-method-selection routes here once a verifiable pass/fail signal exists; preference-optimization is the sibling skill for preference pairs rather than verifiable rewards; eval-harness-first covers judge calibration for any reward that isn't purely code-checkable. On DGX Spark, defer to the dgx-spark-ops plugin's skills, when installed, for the memory/thermal remediation ladder this skill's memory table doesn't cover.