brycewang-stanford/aer-consistency

AER Consistency

Overview

Referees and editors run cheap integrity checks before engaging with ideas: does the abstract's number appear in the tables? Do the Ns add up? Does "Table 4" exist? Does every citation resolve? A single mismatch reframes the entire report from "is this right?" to "what else is wrong?" — and for AI-assisted manuscripts these mismatches are the modal failure, because text and tables are often generated in separate passes.

This skill is the full-manuscript integrity audit. It is mechanical by design: every check below has a yes/no answer obtained by comparing two artifacts, not by judgment. Run it after every revision round, not only before first submission.

When to Use

• The body sections and exhibits exist and the manuscript is being assembled
• After an R&R revision, when numbers and exhibit ordering changed
• Before aer-referee-sim (so the simulated referees attack substance, not typos) and before aer-submission
• Any time results were re-run — even "tiny" re-runs desynchronize text

Audit 1 — The Headline-Number Register

Build a register of every number that appears more than once in the manuscript, then verify each row against its single source of truth (the table or the replication output):

NUMBER            SOURCE            ABSTRACT  INTRO  RESULTS  CONCL  MATCH
4.2 log points    Tab 3 col 4       yes       yes    yes      yes    OK
s.e. 1.1          Tab 3 col 4       yes       no     yes      no     OK
$84 billion       cited source      no        yes    no       yes    OK
N = 37,824        Tab 1             no        no     yes      no     OK

Rules:

• Every quoted estimate matches its table to the digit, including the standard error. No re-rounding in prose: if the table says 0.042 (0.011), the text says 4.2, not 4 or 4.20.
• The abstract, introduction, and conclusion quote the same headline specification. Quoting column 3 in the abstract and column 4 in the intro is a real and common failure.
• Externally sourced numbers (the hook's "$84 billion") carry a citation at first use, and the same value everywhere.

Audit 2 — Sample-Size Integrity

• The Data section's sample funnel arithmetic is exact: raw N minus each documented drop equals the analysis N.
• The analysis N in Table 1 equals the N in the main results table for the matching specification; every deviation (balanced panel, IV subsample) is explained in the table notes and the text.
• Observation counts are consistent with the unit of analysis (12,400 county-years from 620 counties × 20 years — check the multiplication).
• Heterogeneity subsample Ns sum to the full-sample N (minus documented exclusions).

Audit 3 — Units and Conversions

The conversion table for prose claims about coefficients:

| Outcome form | Coefficient β means | Exact percent effect | |---|---|---| | log(Y), binary D | 100·β log points | 100·(e^β − 1) | | log(Y), log(X) | elasticity | β% per 1% of X | | Y in levels, binary D | β units of Y | 100·β / mean(Y) | | Y is a rate (share) | β·100 percentage points | 100·β / baseline rate percent |

Checks:

• Every "percent" vs. "percentage point" usage verified against the outcome's units. A coefficient of 0.02 on an employment rate is 2 percentage points; on log employment it is 2.0 percent (exactly 2.02).
• The exact exponential conversion used whenever |β| > 0.10; 0.31 log points is 36 percent, not 31.
• Standardized effects state which SD (cross-sectional? within? whose sample?) and the SD's value.
• Currency years and deflators consistent across all sections; one base year, named.
• Signs: a negative coefficient on an inverted scale narrated correctly — the most embarrassing class of referee catch.

Audit 4 — Stars, Standard Errors, and Stated Significance

• For each starred coefficient, |β/se| is consistent with the stars under the declared convention (1.65 / 1.96 / 2.58 thresholds approximately).
• Every prose claim of "significant at the X percent level" matches the table's stars and the CI.
• Claims of "no effect" are backed by a CI the text reports, not by absence of stars (see aer-robustness on null-result discipline).
• One star convention across all tables (aer-tables-figures sets it; this audit verifies it held).

Audit 5 — Cross-Reference Integrity

• Every \ref / \autoref resolves; no "Table ??" anywhere in the PDF.
• Every table and figure is referenced in the text at least once, in order of first reference; exhibits nobody cites get cut or moved to the appendix.
• Section references survive renumbering ("see Section V" after V became IV is the classic R&R injury).
• Appendix references point to existing appendix objects, including the supplemental file if separate.
• Equation references match the renumbered equations.

Audit 6 — Citation Two-Way Match

• Every in-text citation has a bibliography entry; every bibliography entry is cited at least once. LaTeX builds with zero unresolved citation warnings.
• Author spellings and years in text match the entries (Cattaneo 2020 in text, 2019 in the bib — fail).
• The deeper verification — that entries are real and claims accurate — is aer-literature's integrity protocol; this audit confirms it was run and the ledger has no open rows.

Audit 7 — Claim-Evidence Map

For each empirical claim in the abstract, introduction, and conclusion, record where the evidence lives:

CLAIM                                          EVIDENCE          STATUS
"raises 90/10 ratio by 4.2 log points"         Tab 3 c4          OK
"driven by gains at the top"                   Fig 3 / Tab 4     OK
"absent in retail and construction"            Tab 5 c2-c3       OK
"consistent with skill-biased adoption"        Sec V battery     OK (consistency claim)

• Causal claims trace to design-based exhibits; mechanism claims are worded as consistency ("consistent with"), matching aer-paper-body rules.
• Any claim with no exhibit or citation is rewritten or deleted — "we find" with nothing to point at is how overclaiming enters a manuscript.
• When a replication package skeleton is available, maintain docs/claim-evidence-ledger.csv: use label:<tex-label> for manuscript exhibits, cite:<bib-key> for externally sourced claims, and file:<relative-output> for generated output files. Rows must be OK or PASS before handoff.

Mechanical Procedure

1. Run the bundled script for the deterministic LaTeX checks (citations two-way, ref/label two-way, duplicate labels, abstract word count):

   python3 skills/aer-consistency/scripts/audit_manuscript.py paper.tex references.bib
   python3 skills/aer-consistency/scripts/audit_manuscript.py paper_dir references.bib \
     --claim-ledger paper_dir/docs/claim-evidence-ledger.csv
   

2. Extract every number from the abstract and introduction (grep for digits); locate each in a table or a cited source; build the register.

3. Recompute the sample funnel and the unit conversions by hand — these are arithmetic, not judgment.

4. Diff the table files in the manuscript against the replication package's output/tables/ — they must be the same files, not lookalikes (aer-replication requires this anyway).

5. Produce the consistency report (below) and fix every FAIL before handing off.

The Consistency Report

AUDIT                       RESULT     DETAIL
1 headline numbers          PASS       12 numbers, 12 matched
2 sample sizes              FAIL       Tab 4 N=37,824 vs Tab 1 N=37,284
3 units and conversions     PASS       2 exact conversions applied
4 stars vs SEs              PASS
5 cross-references          PASS       31 refs, 0 dangling
6 citations two-way         FAIL       2 bib entries uncited
7 claim-evidence map        PASS       9 claims mapped

Fix-and-rerun until all PASS. The report travels with the handoff so aer-referee-sim and aer-submission know the floor is solid.

Common Failure Modes

• Tables regenerated after the prose was written, desynchronizing every quoted estimate
• Abstract edited for word count, changing "4.2" to "about 4" while the tables stayed exact
• Percentage points and percent swapped exactly once, in the abstract
• Two tables both numbered 3 after an R&R reshuffle
• The conclusion claiming a mechanism the Results section only called "suggestive"

Repository Resources

Bundled with the installed skill, no repository checkout needed --- read it before the repo resources below:

• references/audit-checklist.md --- manual audit passes plus what the bundled script does and does not catch

When working from the AER-skills repository or plugin bundle, load only the relevant resource:

• Exhibit-to-script mapping that fixes each number's source of truth: examples/replication-package-skeleton/docs/exhibit-register.md
• Claim-to-evidence template checked by the bundled script: examples/replication-package-skeleton/docs/claim-evidence-ledger.csv
• Narration rules the claim-evidence map enforces: skills/aer-paper-body/SKILL.md
• Citation verification protocol behind Audit 6: skills/aer-literature/SKILL.md
• Prose-level conventions for units and significance language: docs/style-guide.md

Handoff

AUDITS PASSED: <n>/7
HEADLINE NUMBERS MATCHED: <n>/<n>
OPEN FAILURES: <list, or "none">
ABSTRACT WORD COUNT: <n>/100
CITATION LEDGER: <closed / open rows remain>
CLAIM-EVIDENCE LEDGER: <n> claims, <closed / open rows remain>
NEXT SKILL: <aer-referee-sim | aer-submission>

Anti-Patterns

• Running this audit once, before first submission only — every revision reopens it
• "The numbers are close enough" — referees diff digits, not vibes
• Fixing the prose to match a table without checking which one the replication package actually produces
• Treating a FAIL as a note for later instead of a blocker for handoff
• Auditing by re-reading instead of by register — unstructured re-reading finds style issues and misses arithmetic