brycewang-stanford/replication-driven-research

Replication-Driven Research

Overview

This skill is the research-domain analogue of test-driven development. The core philosophy is the same as TDD: evidence before claims, automated verification, invalidation on input change. No result is valid until the pipeline runs end-to-end without error. No number enters the paper until a script regenerates it from raw data. This is the single most important discipline in the superpapers plugin, and every analysis skill defers to it.

When to Use

• Start of a new empirical project
• First time running analysis code in a project
• Before generating any table or figure for a paper
• After any change to raw data, sample selection, or model specification
• Before declaring a result "final" or "verified"
• When the user asks "is this reproducible?"
• When reviewer comments require re-running the analysis

Canonical Directory Structure

The canonical project layout is:

project-root/
├── data/
│   ├── raw/                    # raw downloads, never manually edited
│   ├── processed/              # cleaned data, output of scripts
│   └── manifest.md             # documents every dataset (source, URL, date, variables)
├── code/
│   ├── 01_collect.R            # or .py — fetches raw data
│   ├── 02_clean.R              # raw → processed
│   ├── 03_analyze.R            # processed → results
│   └── 04_figures.R            # processed → figures
├── output/
│   ├── tables/                 # .tex files generated by scripts
│   ├── figures/                # .pdf vector files generated by scripts
│   └── logs/                   # execution logs with timestamp + seed
├── paper/
│   ├── paper.tex               # main document
│   ├── references.bib          # bibliography
│   └── sections/               # split sections if needed
└── CLAUDE.superpapers.md       # project settings

On first invocation in a project, propose this structure to the user. The user can accept, adapt, or refuse. If the user refuses or has a different layout, the skill still works but flags deviations when encountered so they are not silent.

Mandatory Steps

1. Verify or scaffold structure. If data/raw, code/, and output/ do not exist, propose scaffolding. Wait for user confirmation before creating any directories or files at the project root.

2. Document every dataset in data/manifest.md. Required fields per dataset: name, source (URL or API endpoint), description, collection date, variables used, frequency, period covered, license or usage notes. See the Manifest Format section below.

3. Every result must have a generating script. No exceptions. Pasting numbers from a console or notebook into the paper is forbidden. Tables enter the paper via \input{output/tables/...}; figures enter via \includegraphics{output/figures/...}.

4. Fix the seed in every script that uses randomness. Document the seed in the script header. Use the project-level default seed from CLAUDE.superpapers.md — read from the current working directory, or walk up parent directories until found — unless the script explicitly overrides it and explains why. If the file is absent, ask the user for a project-level default on first need.

5. Run the pipeline end-to-end before declaring any result verified. Use a top-level run_all.sh (or Makefile) that executes scripts in the correct order. Verify exit code 0 and that every expected output file exists and is non-empty.

6. Log every run. Each end-to-end execution writes to output/logs/YYYY-MM-DD_HH-MM-SS.log with: timestamp, seed, relevant package versions, total runtime, input file hashes, exit status. See the Execution Log Format section below.

7. Invalidate on input change. If data/raw or any script in code/ changes, all downstream outputs are stale. Re-run the full pipeline. Do not trust cached tables or figures after an input change.

Manifest Format

Each dataset entry in data/manifest.md follows this format:

## unemployment_br

- **Source:** IBGE — PNADC Trimestral
- **URL:** https://sidra.ibge.gov.br/tabela/4099
- **Collected:** 2026-03-15
- **Variables:** unemployment_rate, quarter, state
- **Frequency:** Quarterly
- **Period:** 2012Q1 – 2025Q4
- **Collected by:** code/01_collect.R
- **License:** IBGE open data

The manifest is the single source of truth for what data is in the project. If a dataset exists in data/raw/ but not in the manifest, fix the manifest before proceeding.

Execution Log Format

Each end-to-end run appends a log entry with one field per line:

timestamp: 2026-04-10T14:23:45-03:00
seed: 20260410
R version: 4.4.1
renv lockfile hash: abc123...
inputs:
  data/raw/pnadc.csv: sha256:def456...
scripts:
  code/01_collect.R: OK (4.2s)
  code/02_clean.R: OK (12.8s)
  code/03_analyze.R: OK (45.1s)
outputs:
  output/tables/tab_descriptives.tex: created
  output/tables/tab_main.tex: created
  output/figures/fig_trend.pdf: created
exit: 0
total runtime: 62.1s

The log is append-only. Keep every log entry — they are the audit trail for every published result.

Anti-Patterns

• Copying a number from console output into the paper
• "It worked yesterday" without re-running after a change to data or code
• Results with no traceable script from raw data
• Undocumented manual steps in the pipeline (opening a GUI, editing a spreadsheet, clicking a button)
• Using set.seed(Sys.time()), random.seed(None), or no seed at all
• Ignoring warnings emitted by the pipeline run
• Running only one script and trusting cached outputs from other scripts
• Editing raw data files manually to "fix" anomalies
• Committing output/ files that are not produced by the current pipeline

Verification Before Completion

• [ ] data/raw/ contains only downloaded files, never hand-edited
• [ ] data/manifest.md documents every dataset in data/raw/ and data/processed/
• [ ] Every .tex table and .pdf figure in output/ has a script in code/ that generates it
• [ ] A top-level runner (run_all.sh or Makefile) exists and runs end-to-end with exit code 0
• [ ] Seeds are fixed and documented at the top of every stochastic script
• [ ] An execution log exists in output/logs/ for the latest run
• [ ] No numeric values hardcoded in paper.tex — all results pulled via \input{}
• [ ] No orphaned outputs in output/ that the current pipeline does not produce