brycewang-stanford/pre-registration-writing

Pre-Analysis Plan Writer

Standards anchor. This skill operationalizes the pre-data-collection component of DA-RT (Data Access and Research Transparency; see Druckman-Green 2021 ch. 18 §18.1.3 for the canonical political-science framing and the APSA Guide to Professional Ethics). DA-RT obligates researchers to facilitate evaluation of evidence-based claims through data access, production transparency, and analytic transparency; a PAP is how analytic transparency is established before the data arrive. Reporting downstream of the PAP is the domain of the methods-reporting skill (APSA Experimental Section guidelines via Gerber et al. 2014; JARS).

Instructions

1. Registry Selection

• OSF Registries (Open Science Framework): Use for maximum flexibility. Supports free-form documents, file attachments (analysis code, stimuli), version control, and optional embargo periods. Registration is timestamped and immutable once confirmed. Best for complex designs that require supplementary materials. Also the default destination for political-science PAPs since the EGAP registry closed: OSF offers an "EGAP Registration" form template that mirrors the old EGAP fields.
• EGAP (Evidence in Governance and Politics) -- CLOSED: EGAP stopped accepting new registrations on October 15, 2023. Existing EGAP registrations remain searchable through OSF. Researchers should now submit new registrations to OSF Registries (using the EGAP form template) or, for randomized experiments, to the AEA RCT Registry. Do not direct a user to "register at EGAP" -- the registry is closed.
• AEA RCT Registry (American Economic Association): Use for randomized controlled trials, particularly in economics, development, and governance. The form is tightly structured around RCT fields (intervention, outcomes, randomization unit, power) and is the EGAP-successor destination endorsed alongside OSF for field experiments.
• AsPredicted: Use for simple designs requiring fast registration. The structured 9-question format enforces brevity and is completable in under an hour. Registrations are private until the authors choose to make them public. Best for straightforward experiments with few analytical degrees of freedom. Requires an academic email for access.
• Registered Reports: A distinct format where the journal peer-reviews the introduction and methods before data collection. In-principle acceptance is contingent on design quality, not results; final acceptance is contingent on the authors following through on the registered methods. Pursue registered reports when the research question is important but the expected results are uncertain or likely null -- this eliminates publication bias by design. Registered reports require substantially more lead time than standard pre-registration.

2. PAP Document Structure

• Recommended Section Order: Organize the PAP into: (1) study information (title, authors, timeline, registry), (2) theoretical motivation and hypotheses, (3) research design (experimental conditions, randomization, sample), (4) sampling and recruitment, (5) variable definitions and measurement, (6) analysis plan (models, tests, decision rules), and (7) contingency plans. This order mirrors the research process and makes the document navigable for reviewers.
• Cross-Reference, Don't Repeat: For hypothesis specification, reference the three-level specification framework (conceptual, operationalized, statistical) from the hypothesis-building skill. For reporting elements, reference the JARS six elements from the methods-reporting skill. For experiment-type-specific design fields, reference the conjoint-design or survey-design skills (e.g., attribute architecture, wording protocols, attention checks). The PAP should implement these frameworks, not redefine them.
• Write for a Reader: The PAP is a communication document, not a private notebook. Every analytical decision must be (a) decidable from the PAP alone, (b) expressed in formal notation or code where possible, and (c) unambiguous to a reader without prior knowledge of the authors' local practices. Avoid shorthand, undefined acronyms, and references to "the usual approach."
• Version Control: Use the registry's built-in versioning. If amendments are needed after initial registration, create a new version rather than editing the original. Each version is timestamped, preserving the audit trail.

3. Specifying the Analytical Strategy

• Three-Tier Classification: Classify every planned analysis as locked (primary hypothesis tests -- cannot be changed), conditional (executed only if a pre-specified condition is met, e.g., "if the manipulation check passes, estimate the interaction model"), or exploratory (clearly labeled hypothesis-generating analyses with uncontrolled error rates). This mirrors the JARS primary/secondary/exploratory distinction (Lakens 2025 §13.4) and the confirmatory-exploratory continuum in Waldron & Allen (2022); it generalizes the conjoint-specific version from conjoint-design to all experimental designs.
• Decision Rules: For each confirmatory hypothesis, state in advance what constitutes support, falsification, or an inconclusive result. Specify: the test statistic, the alpha level, the SESOI, and the decision mapping (e.g., "If p < 0.05 and the coefficient exceeds 3 percentage points in the predicted direction, the hypothesis is supported; if the equivalence test rejects effects larger than 3 percentage points, the hypothesis is falsified; otherwise, the result is inconclusive"). Illustrative thresholds (e.g., the 3-percentage-point SESOI used above) must be justified from the user's own design, prior literature, and decision context -- they are not defaults.
• Exact Model Specifications: Write out every primary model in formal notation or code. For regression models, specify: the dependent variable, all independent variables, interaction terms, fixed effects, clustering structure, and the estimator. Ambiguous prose descriptions ("we will control for demographics") are insufficient -- name every variable. This is the remedy for the "garden of forking paths" problem (Gelman & Loken 2014): implicit analytical choices, even when made in good faith and before seeing results, inflate the false-positive rate unless pinned down in advance.
• Multiple Testing Corrections: Pre-specify the correction procedure and define which tests belong to the same family. For families of related tests (e.g., AMCEs across attributes within a single hypothesis), specify Benjamini-Hochberg (FDR control) or Bonferroni. Document the family groupings and the rationale for each.

4. Analysis Code Pre-Registration

• Simulated Data Approach: Generate a mock dataset that matches the expected data structure (variable names, types, distributions, sample size, missingness patterns). Write all analysis code -- data cleaning, primary models, robustness checks, planned figures -- to run on this simulated dataset. Register the code alongside the PAP. Lakens (2025 §13.6) calls this "the gold standard for a preregistration."
• Tooling for Simulated Data. For formal declare-design-diagnose workflows, use DeclareDesign (Blair, Cooper, Coppock, & Humphreys 2019, cited in Druckman-Green 2021 Table 18.1), which lets researchers specify the data-generating model, the inquiry, the data strategy, and the answer strategy and then diagnose the design before running it. Simpler simulations can use faux or simstudy in R or numpy + pandas in Python. The point is to produce a runnable pipeline, not a polished simulation.
• Benefits: Code pre-registration eliminates ambiguity about analytical decisions that prose alone cannot resolve (e.g., how exactly are covariates centered? What happens to observations with missing values on one covariate but not others?). It also catches specification errors before data collection -- if the code does not run on simulated data, it will not run on real data.
• What to Include: The registered code should cover: (1) data import and cleaning pipeline, (2) variable construction (indices, recodes, exclusion criteria), (3) primary confirmatory models, (4) conditional models with the triggering conditions, (5) planned figures with axis labels and titles, and (6) robustness checks. Exploratory code need not be pre-registered but should be documented post hoc.

5. Contingency Planning

Contingencies enumerate data-generating-process (DGP) failure modes -- manipulation failure, attrition, imbalance, under-recruitment -- and bind the response to each before the data are seen. All illustrative thresholds below (15% differential attrition, 0.10 standardized difference, etc.) are placeholders: justify thresholds from the user's own design, prior literature, and decision context.

• Manipulation Check Failure: If the manipulation check indicates that the treatment did not shift the intended construct, pre-specify the response: (a) report the ITT estimate regardless (it answers the policy-relevant question of what happens when the treatment is delivered), (b) investigate moderators of treatment uptake, and (c) do not selectively exclude non-compliers without a pre-specified rule.
• Excessive Attrition: Define an attrition threshold (e.g., >15% differential attrition across conditions) and the response if exceeded: (a) conduct attrition analysis (are attriters different from completers on observables?), (b) report Lee bounds on the treatment effect, (c) consider sensitivity analysis under alternative missing-data assumptions (MNAR models).
• Covariate Imbalance: Despite randomization, baseline imbalance may occur. Pre-specify: (a) which baseline variables will be checked, (b) the threshold for concern (e.g., a standardized difference > 0.10), and (c) the response (include the imbalanced covariate in the primary model as a robustness check, report both adjusted and unadjusted estimates).
• Sample Size Shortfall: If recruitment falls short of the target N, pre-specify: (a) the minimum N below which the study will not be analyzed (the threshold at which power drops below a defensible level), (b) a sensitivity analysis showing the MDE for the achieved N, and (c) whether the study will be reframed as exploratory if underpowered.
• Standard Operating Procedures (SOPs). Where the user or their lab has a written SOP (e.g., default attrition-handling or noncompliance rules in the Lin-Green tradition, discussed in Druckman-Green 2021 Table 18.1), cite the SOP version by filename and date and let it carry the default practice rather than restating rules inside every PAP. This lowers per-PAP cost and produces consistency across studies in the same program.
• Decision Trees: For complex contingencies, use explicit if-then decision trees: "If condition X holds, analyze using Model A; if condition X does not hold, analyze using Model B and report the deviation." This prevents post hoc rationalization of analytical choices.

6. Deviation Documentation

• Four-Part Deviation Record: Document every deviation from the registered plan with: (a) what changed (the specific analysis, variable, or procedure that differs from the PAP), (b) why it changed (the substantive or methodological reason), (c) the impact on severity -- a property of the test, does the deviation make the test more or less capable of falsifying the hypothesis? -- and (d) the impact on validity -- a property of the inference, does the deviation improve or degrade construct, internal, or external validity? The severity/validity distinction traces to the error-statistical (severe-testing) tradition; the concrete four-part protocol is Lakens (2025 §13.5).
• Principled vs. Convenience Deviations: Distinguish between principled deviations (fixing a validity problem discovered after registration, e.g., a measure that does not function as intended) and convenience deviations (changing the analysis because the pre-registered approach yields unfavorable results). Principled deviations can strengthen a study; convenience deviations undermine credibility.
• Side-by-Side Reporting: When a deviation occurs, report both the pre-registered analysis and the deviated analysis. Readers can then assess whether the deviation affects the conclusions. Never replace the pre-registered analysis with the deviated one -- always present both.

7. Timeline and Logistics

• When to Register: Register after the design is finalized and the analysis code runs on simulated data, but before any data collection begins. For registered reports, submit for journal review before data collection. For standard pre-registration, the timestamp must precede the first survey response.
• Embargo Options: OSF Registries permits registrations to be embargoed for up to 4 years, after which they become public (Lakens 2025 §13.8); this is the only major platform without an indefinite-embargo option. Use embargoes for competitive research where premature disclosure could enable scooping. AEA RCT Registry entries are public on submission. AsPredicted registrations are private until authors choose to share. (EGAP's independent registry is closed; pre-2024 EGAP registrations remain searchable via OSF, and new political-science registrations inherit OSF embargo rules.)
• Updating Registrations: If the design changes after registration (e.g., after pilot results), create a new version or a new registration that references the original. Document what changed and why. Do not delete or overwrite the original registration.
• IRB Coordination: In many institutions, IRB approval is required before data collection but not before pre-registration. Register early, then amend the PAP if IRB review requires design changes. Include the IRB protocol number in the PAP once approved. Budget for the possibility that IRB review will take several weeks (4--8 weeks is an institutional rule-of-thumb, not a methodological derivation).
• PAP-to-Paper Mapping: After data collection and analysis, include a PAP concordance table in the paper's appendix -- a table mapping each PAP element to the corresponding section of the paper, with deviations flagged. This makes compliance auditable and signals transparency. The concordance is the author-side complement to the audit checklists in the methods-reporting skill (Wicherts et al. 2016; Gerber et al. 2014).

Quality Checks

• [ ] DA-RT Framing: Does the PAP explicitly situate itself within DA-RT / APSA transparency obligations (data access, production transparency, analytic transparency)?
• [ ] Registry Selected: Is the registry platform chosen and justified (OSF Registries, AEA RCT Registry, AsPredicted, or registered report)? Note that EGAP's independent registry is closed.
• [ ] Readable Structure: Does the PAP follow a logical section order that an independent researcher could navigate without contacting the authors?
• [ ] Three-Tier Classification: Is every planned analysis classified as locked, conditional, or exploratory (Lakens 2025 §13.4; Waldron & Allen 2022)?
• [ ] Decision Rules: Are support, falsification, and inconclusive criteria specified for each confirmatory hypothesis, with each threshold (alpha, SESOI, effect-size floor) justified rather than defaulted?
• [ ] Exact Models: Are all primary models written in formal notation or code, with every variable named -- closing off the forking-paths problem (Gelman & Loken 2014)?
• [ ] Multiple Testing: Is the correction procedure and family grouping pre-specified?
• [ ] Code Registered: Does analysis code run on a simulated dataset that matches the expected data structure (DeclareDesign or equivalent)?
• [ ] Contingencies Specified: Are contingency plans written for manipulation check failure, attrition, imbalance, and sample size shortfall, with reference to any governing SOPs?
• [ ] Decision Trees: Are complex contingencies expressed as explicit if-then decision trees?
• [ ] Deviation Protocol: Is the four-part deviation documentation framework (what, why, severity, validity) committed to?
• [ ] Side-by-Side Commitment: Does the PAP commit to reporting both pre-registered and deviated analyses when deviations occur?
• [ ] Timeline: Is the registration timestamped before data collection, with IRB coordination planned?
• [ ] PAP Concordance: Does the plan include a commitment to a PAP-to-paper concordance table?
• [ ] Researcher-DF Audit: Has the PAP been checked against the Wicherts et al. (2016) 34-item researcher-degrees-of-freedom checklist (cross-referenced in the methods-reporting skill)?