akaszubski/scientific-validation

Scientific Validation Skill

Rigorous methodology for validating claims from any source - books, papers, theories, or intuition.

When This Skill Activates

• Testing claims from books, papers, or expert sources
• Validating rules, strategies, or hypotheses
• Running experiments or backtests
• Keywords: "validate", "test hypothesis", "experiment", "backtest", "prove", "evidence"

---

Core Principle

Data is the arbiter. Sources can be wrong.

• Expert books can be wrong
• Only empirical validation decides what works
• Document negative results - they're valuable

---

Phase Overview

| Phase | Name | Key Requirement | |-------|------|-----------------| | 0 | Claim Verification | Understand what source ACTUALLY claims | | 1 | Claims Extraction | Document with source citations | | 1.5 | Publication Bias Prevention | Document ALL claims before selecting | | 2 | Pre-Registration | Hypothesis BEFORE seeing results | | 2.3 | Power Analysis | Calculate required n (MANDATORY) | | 3 | Bias Prevention | Look-ahead, survivorship, selection | | 3.5 | Walk-Forward | Required for time series (MANDATORY) | | 4 | Statistical Requirements | p-values, effect sizes, corrections | | 4.7 | Bayesian Complement | Bayes Factors for ambiguous results | | 5 | Multi-Source Validation | Test across 3+ contexts | | 5.3 | Sensitivity Analysis | ±20% parameter stability (MANDATORY) | | 5.5 | Adversarial Review | Invoke experiment-critic agent | | 6 | Classification | VALIDATED / REJECTED / INSUFFICIENT | | 7 | Documentation | Complete audit trail | | 7.3 | Negative Results | Structured failure documentation |

See: workflow.md for detailed step-by-step instructions per phase.

---

Quick Reference

Claim Types

| Type | Testable? | Example | |------|-----------|---------| | PERFORMANCE | YES | "A beats B on metric X" | | METHODOLOGICAL | YES | "A enables capability X" | | PHILOSOPHICAL | MAYBE | "X is important because Y" | | BEHAVIORAL | HARD | "Humans do X in situation Y" |

Sample Size Requirements (80% Power)

| Effect Size | Cohen's d | Required n | |-------------|-----------|------------| | Small | 0.2 | 394 | | Medium | 0.5 | 64 | | Large | 0.8 | 26 |

See: code-examples.md#power-analysis for calculation code.

Classification Criteria

| Status | Criteria | |--------|----------| | VALIDATED | OOS meets all criteria + critic PROCEED | | CONDITIONAL | OOS meets relaxed criteria (p < 0.10) | | REJECTED | OOS fails OR negative effect | | INSUFFICIENT | n < 15 in OOS | | UNTESTABLE | Required data unavailable | | INVALID | Circular validation detected |

Domain Effect Thresholds (Trading)

| Metric | Minimum | Strong | Exceptional | |--------|---------|--------|-------------| | Sharpe Ratio | > 0.5 | > 1.0 | > 2.0 | | Win Rate | > 55% | > 60% | > 70% | | Profit Factor | > 1.2 | > 1.5 | > 2.0 |

See: code-examples.md#effect-thresholds for other domains.

Bayes Factor Interpretation

| BF | Evidence | |----|----------| | < 1 | Supports null | | 1-3 | Anecdotal | | 3-10 | Moderate | | 10-30 | Strong | | > 30 | Very strong |

---

Critical Rules

1. Pre-Registration

• Document hypothesis BEFORE seeing any results
• Define success criteria BEFORE testing
• No peeking at test data

2. Power Analysis (Phase 2.3)

from statsmodels.stats.power import TTestIndPower
n = TTestIndPower().solve_power(effect_size=0.5, power=0.80, alpha=0.05)

Rule: Underpowered studies cannot achieve VALIDATED status.

3. Walk-Forward for Time Series (Phase 3.5)

• Standard K-fold CV → INVALID (temporal leakage)
• Single train/test → CONDITIONAL at best
• Walk-forward → Can achieve VALIDATED

See: code-examples.md#walk-forward for implementation.

4. Multiple Comparison Correction

alpha_corrected = 0.05 / num_claims  # Bonferroni

For trading claims: require t-ratio > 3.0 (Harvey et al. standard).

5. Sensitivity Analysis (Phase 5.3)

Test ±20% parameter variation:

• All variations positive → Can achieve VALIDATED
• 1-2 sign flips → CONDITIONAL at best
• 3+ sign flips → REJECTED (fragile)

See: code-examples.md#sensitivity-analysis for implementation.

6. Adversarial Review (Phase 5.5)

Use Task tool:
  subagent_type: "experiment-critic"
  prompt: "Review experiment EXP-XXX"

MANDATORY before any classification.

---

Bias Prevention Checklist

| Bias | Prevention | |------|------------| | Look-ahead | Process data sequentially, compare batch vs streaming | | Survivorship | Track ALL attempts, not just completions | | Selection | Report ALL experiments including failures | | Data snooping | Strict train/test split, no tuning on test data | | Publication | Document ALL claims before selecting which to test |

---

Pre-Experiment Checklist

• [ ] Claim extracted with source citation
• [ ] ALL claims documented (not just tested ones)
• [ ] Hypothesis documented BEFORE results
• [ ] Power analysis: required n calculated
• [ ] Success criteria defined
• [ ] Walk-forward configured (time series)
• [ ] Costs/constraints specified

Post-Experiment Checklist

• [ ] Sample size adequate per power analysis
• [ ] p-value AND effect size reported
• [ ] Bayesian analysis if ambiguous
• [ ] Sensitivity analysis passed
• [ ] Adversarial review completed
• [ ] Negative results documented if REJECTED

---

Red Flags

• 100% success rate → Possible bias
• OOS better than training → Possible leakage
• Result flips with ±20% params → Fragile
• Only tested "interesting" claims → Selection bias

---

Key Principles

1. Hypothesis BEFORE data - No peeking
2. Power analysis BEFORE experiment - Know required n
3. Walk-forward for time series - Preserve temporal order
4. Sensitivity analysis - Results must survive ±20% changes
5. Adversarial self-critique - Challenge your methodology
6. Document negative results - Failures are valuable
7. Sources can be wrong - Even experts, even textbooks

---

Detailed Documentation

| Topic | File | |-------|------| | Step-by-step workflow | workflow.md | | Python code examples | code-examples.md | | Markdown templates | templates.md | | Adversarial review | ../../agents/experiment-critic.md |

---

Hard Rules

FORBIDDEN:

• Reporting results without confidence intervals or statistical significance
• Cherry-picking favorable metrics while ignoring unfavorable ones
• Claiming causation from correlation without controlled experiments

REQUIRED:

• All experiments MUST have a documented hypothesis before execution
• All results MUST include sample size, variance, and statistical test used
• Negative results MUST be reported with the same rigor as positive results
• Baselines MUST be established and compared against for every metric