ADu2021/exploration-exploitation-rlvr
skills/skillxiv-v0.0.2-claude-opus-4.6/exploration-exploitation-rlvr/SKILL.md
Investigate exploration-exploitation trade-offs in reinforcement learning with verifiable rewards through theoretical analysis and empirical validation. Derive explicit clipping bias bounds, establish policy-entropy shift formulation, and introduce reward-misalignment framework. Show policy entropy and performance lack direct causal relationships.
$ install --global
skillsauthnpx skillsauth add ADu2021/skillXiv exploration-exploitation-rlvrInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
Security Scan Results
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
3
Scanners Passed
9
Scanners in report
Clean
TrivyContainer and dependency vulnerability scanner
95%
Clean
SemgrepStatic code analysis for vulnerabilities
95%
Clean
mcp-scan (Snyk)Model Context Protocol security validation
95%
Skipped
Snyk (dep)Open source security scanning
50%
Skipped
Socket.devSupply chain security analysis
50%
Skipped
VirusTotalMulti-engine malware detection
50%
Skipped
CrowdStrikeAdvanced threat intelligence
50%
Skipped
OSV-ScannerOpen Source Vulnerability database check
50%
Skipped
OWASP Dep-Check
50%
Last scanned: Apr 17, 2026, 5:34 AM6.2s1 file scanned
SKILL.md
- name:
- exploration-exploitation-rlvr
- title:
- Exploration vs Exploitation in RLVR: Rethinking Clipping, Entropy, and Spurious Reward Effects
- version:
- 0.0.2
- engine:
- skillxiv-v0.0.2-claude-opus-4.6
- license:
- MIT
- url:
- https://arxiv.org/abs/2512.16912
- keywords:
- [reinforcement-learning, exploration-exploitation, policy-entropy, reward-misalignment, RLVR]
- description:
- Investigate exploration-exploitation trade-offs in reinforcement learning with verifiable rewards through theoretical analysis and empirical validation. Derive explicit clipping bias bounds, establish policy-entropy shift formulation, and introduce reward-misalignment framework. Show policy entropy and performance lack direct causal relationships.
Skill Summary
This research investigates fundamental exploration-exploitation trade-offs in RLVR through theoretical analysis and empirical validation. The work derives explicit bounds on clipping bias and introduces a one-step policy-entropy shift formulation establishing that clipping systematically reduces entropy independent of reward signals. Paradoxically, clipping decreases performance under random rewards, contradicting prior assumptions. A probabilistic reward-misalignment framework explains when spurious rewards help: stronger models improve because they generate more correct rollouts, experiencing less damage from reward label mislabeling. Key finding: policy entropy and performance lack direct causal relationships—entropy reduction complements rather than replaces principled reward signals.
When To Use
- Training reasoning models with RLVR where entropy management is critical
- Projects exploring theoretical foundations of RLVR
- Scenarios questioning standard entropy assumptions
- Research on reward signal quality and model strength interaction
When NOT To Use
- Applications already achieving strong RLVR results with standard approaches
- Scenarios where empirical entropy tricks work well
- Domains not using reward-verifiable training
- Models where simple entropy minimization suffices
Core Technique
Three interconnected contributions clarify exploration-exploitation dynamics:
1. Theoretical Analysis of Clipping Derive explicit bounds on clipping bias showing that clipping "does not constitute a meaningful learning signal" under random rewards and instead functions as regularization controlling entropy dynamics. Clipping systematically reduces entropy independent of reward signals through policy divergence effects.
2. One-Step Policy-Entropy Shift Formulation Establish mathematical framework showing how clipping affects policy entropy over single training steps. This reveals mechanisms by which clipping operates—not through learning signal but through regularization dynamics.
3. Reward-Misalignment Framework Introduce probabilistic model explaining when spurious rewards benefit models. Key insight: stronger models improve under random rewards because they generate more correct rollouts (high baseline quality), experiencing less "damage" from false positive/negative reward mislabeling. Weaker models suffer more.
Empirical Findings
- Clipping paradoxically decreases performance under random rewards
- Contradicts prior assumptions about upper-clipping bias driving improvements
- Policy entropy and performance lack direct causal relationships
- Entropy minimization effectiveness depends on model strength and dataset difficulty
Implementation Notes
Understand that clipping primarily functions as entropy regularization, not learning signal. Consider reward signal quality—stronger models tolerate noisier rewards better. Use entropy reduction to complement (not replace) high-quality rewards. Validate entropy effects empirically on your specific task/model combination.
References
- Original paper: Exploration vs Exploitation (Dec 2025)
- Reinforcement learning with verifiable rewards
- Policy gradient theory and analysis
Related Skills
ADu2021/flow-map-trajectory-tilting
testing
VerifiedTrustedCommunity
Uses flow maps as look-ahead operators to enable principled reward-guided diffusion by predicting trajectory endpoints at any denoising step. Deploy when applying rewards or preferences to diffusion trajectories with meaningful gradients throughout generation.
2SKILL.mdUpdated Apr 17, 2026
ADu2021/flexible-data-mixture-of-experts
testing
VerifiedTrustedCommunity
Train language models where each expert learns independently on closed datasets, enabling flexible inference with selective data inclusion or exclusion. 41% performance improvement while allowing users to opt out of specific data sources without retraining.
2SKILL.mdUpdated Apr 17, 2026
ADu2021/flexibility-trap-diffusion-reasoning
data-ai
VerifiedTrustedCommunity
Understand how token generation flexibility in diffusion LMs paradoxically constrains reasoning, as models exploit ordering flexibility to avoid uncertain tokens, and apply simplified approaches that preserve parallel decoding benefits. Use when optimizing diffusion-based language models for reasoning tasks.
2SKILL.mdUpdated Apr 17, 2026
ADu2021/flex-continuous-agent-evolution
devops
VerifiedTrustedCommunity
Enable LLM agents to improve continuously during deployment by constructing structured experience libraries through self-reflection on successes and failures—achieving 23% improvement on reasoning without gradient-based parameter updates or external training.
2SKILL.mdUpdated Apr 17, 2026