ADu2021/art-of-scaling-test-time
skills/skillxiv-v0.0.2-claude-opus-4.6/art-of-scaling-test-time/SKILL.md
Comprehensive empirical study recommending model-specific test-time scaling strategies (majority voting, first-finish search) across eight LLMs based on architectural family, problem difficulty, and compute budget rather than universal approaches.
$ install --global
skillsauthnpx skillsauth add ADu2021/skillXiv art-of-scaling-test-timeInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
Security Scan Results
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
3
Scanners Passed
9
Scanners in report
Clean
TrivyContainer and dependency vulnerability scanner
95%
Clean
SemgrepStatic code analysis for vulnerabilities
95%
Clean
mcp-scan (Snyk)Model Context Protocol security validation
95%
Skipped
Snyk (dep)Open source security scanning
50%
Skipped
Socket.devSupply chain security analysis
50%
Skipped
VirusTotalMulti-engine malware detection
50%
Skipped
CrowdStrikeAdvanced threat intelligence
50%
Skipped
OSV-ScannerOpen Source Vulnerability database check
50%
Skipped
OWASP Dep-Check
50%
Last scanned: Apr 24, 2026, 8:55 PM1.8s1 file scanned
SKILL.md
- name:
- art-of-scaling-test-time
- title:
- The Art of Scaling Test-Time Compute for Large Language Models
- version:
- 0.0.2
- engine:
- skillxiv-v0.0.2-claude-opus-4.6
- license:
- MIT
- url:
- https://arxiv.org/abs/2512.02008
- keywords:
- [test-time-scaling, llm-inference, majority-voting, beam-search, model-aware-strategies]
- description:
- Comprehensive empirical study recommending model-specific test-time scaling strategies (majority voting, first-finish search) across eight LLMs based on architectural family, problem difficulty, and compute budget rather than universal approaches.
Summary
The Art of Scaling Test-Time Compute is an empirical study and decision framework comparing test-time scaling (TTS) strategies across eight LLMs and four reasoning datasets, generating over 30 billion tokens. The key finding is that no universal TTS strategy dominates—optimal performance depends on model family, problem difficulty, and compute budget. The authors provide a practical recipe table for strategy selection.
Core Technique
Two Model Categories: The study identifies distinct categories based on post-training algorithms:
- Short-horizon models: Trained with algorithms other than GRPO (e.g., GSPO), show decreasing quality with longer traces
- Long-horizon models: Trained with GRPO, maintain or improve quality with longer traces
This distinction is critical for strategy selection.
TTS Strategy Comparison:
- Beam Search: Keep top-k candidates at each step, select highest-reward final output
- Majority Voting: Generate multiple independent solutions, return most common answer
- First-Finish Search: Return first correct solution found, stop early
- Last-Finish Search: Exhaust budget finding all correct solutions, return last found
Implementation
Model categorization: Profile your model on a small validation set:
- Generate 10 solutions each of length L and 2L
- Compare quality at both lengths
- If quality improves at 2L: long-horizon; if decreases: short-horizon
Strategy selection: Use decision tree:
if problem_difficulty == "easy":
use majority_voting(n_samples=3)
elif model_family == "long-horizon":
use first_finish_search(budget=large)
else: # short-horizon, hard problem
use majority_voting(n_samples=5)
Budget allocation: Allocate compute proportionally:
if compute_budget < 2x:
majority_voting(n=2)
elif compute_budget < 4x:
first_finish_search(budget=4x)
else:
first_finish_search(budget=compute_budget)
When to Use
- Deploying LLMs where reasoning quality significantly impacts downstream tasks
- Scenarios with variable compute budgets where you need adaptive scaling
- Applications requiring understanding which strategy is optimal for your model
- Tasks where you can measure solution quality automatically
When NOT to Use
- Real-time inference where latency doesn't allow multiple samples
- Models where TTS overhead exceeds quality gains
- Scenarios without automated correctness evaluation
- Applications where single-sample inference is preferred for determinism
Key References
- Beam search and sampling strategies in LLMs
- Majority voting and ensemble methods for reasoning
- Model-specific optimization and strategy selection
- Test-time compute scaling literature
Related Skills
ADu2021/flow-map-trajectory-tilting
testing
VerifiedTrustedCommunity
Uses flow maps as look-ahead operators to enable principled reward-guided diffusion by predicting trajectory endpoints at any denoising step. Deploy when applying rewards or preferences to diffusion trajectories with meaningful gradients throughout generation.
2SKILL.mdUpdated Apr 17, 2026
ADu2021/flexible-data-mixture-of-experts
testing
VerifiedTrustedCommunity
Train language models where each expert learns independently on closed datasets, enabling flexible inference with selective data inclusion or exclusion. 41% performance improvement while allowing users to opt out of specific data sources without retraining.
2SKILL.mdUpdated Apr 17, 2026
ADu2021/flexibility-trap-diffusion-reasoning
data-ai
VerifiedTrustedCommunity
Understand how token generation flexibility in diffusion LMs paradoxically constrains reasoning, as models exploit ordering flexibility to avoid uncertain tokens, and apply simplified approaches that preserve parallel decoding benefits. Use when optimizing diffusion-based language models for reasoning tasks.
2SKILL.mdUpdated Apr 17, 2026
ADu2021/flex-continuous-agent-evolution
devops
VerifiedTrustedCommunity
Enable LLM agents to improve continuously during deployment by constructing structured experience libraries through self-reflection on successes and failures—achieving 23% improvement on reasoning without gradient-based parameter updates or external training.
2SKILL.mdUpdated Apr 17, 2026