brycewang-stanford/aer-referee-sim

AER Referee Simulation

Overview

Most papers submitted to AER are rejected; the realistic acceptance rate is 6-8 percent, and a large share never reach referees. The cheapest referee report is the one generated before submission — but only if it is as harsh as the real one. The failure mode of self-review (human or AI) is leniency: reviewing the paper one hopes was written instead of the one on the page.

This skill runs the AER editorial process against the draft: a ten-minute desk screen, then three referee reports written from distinct, adversarial priors, then an editor's synthesis with a calibrated verdict and a prioritized revise list. The simulation has one rule that overrides all others:

The simulated reviewers' job is to reject the paper. Every comment must survive the question "would this withstand the authors' best rebuttal?" — but praise requires the same evidence as criticism.

When to Use

• A complete draft exists (body, exhibits, bibliography) and aer-consistency reports all-pass
• Before every submission and resubmission
• After a real rejection, to test whether the revision would survive the same reports
• When coauthors disagree about whether the paper is ready

Do not use on a half-draft — the simulation will correctly report that the paper is incomplete, which wastes the run. And do not let it replace aer-consistency: typo-hunting referees are wasted referees.

Stage 1 — The Desk Screen

Simulate the editor's first pass: ten minutes, first three pages, then the main tables, then the bibliography. The editor is deciding only one thing — is this worth three referees' time?

Work through docs/desk-rejection-audit.md items 1-5 plus three scans:

• Contribution scan. Can the editor state the contribution in one sentence after page 3? Would an economist outside the subfield care?
• Design scan. Is the identification strategy named on page 1-2, and is it a modern design (aer-identification red flags apply on sight)?
• Craft scan. Tables in house style, abstract within 100 words, prose free of the failure patterns in docs/style-guide.md. Editors read craft as a proxy for care in the empirics.

Output a desk decision with the editor's two-paragraph letter:

DESK DECISION: <reject | send to referees>
LETTER: <the letter an AER editor would actually send>

Calibration: if any Stage 1-2 item in the desk-rejection audit fails, the decision is reject — write the letter and stop. Do not soften a desk reject into "borderline" to keep the simulation going; fix the draft and rerun.

Stage 2 — Three Referee Reports

Three referees, three priors, three reading orders. Each writes independently — draft all three before reconciling anything, and never let R2 inherit R1's findings.

Referee 1 — The identification specialist

Reads: Empirical Strategy first, then Data, then the robustness appendix. Prior: "the design is broken until proven otherwise."

Attacks: the identifying assumption's plausibility in this setting; missing diagnostics from the aer-identification battery; inference mismatched to the variation's level; estimand-population gaps (whose effect is this?); the alternative story the design cannot exclude. R1 re-derives at least one magnitude from the tables and checks it against the prose.

Referee 2 — The field expert

Reads: Introduction, then the antecedents, then Results against the literature. Prior: "we probably already knew this."

Attacks: novelty against the working-paper frontier (names the closest papers, including any the draft missed — aer-literature's map is the checklist); whether magnitudes are plausible next to the literature's; whether the mechanism evidence distinguishes the favored channel from the obvious rival; institutional errors a field insider would catch. R2 is the referee most likely to have written one of the antecedents.

Referee 3 — The generalist

Reads: linearly, as an editor-board member from another subfield. Prior: "why should I care, and can I follow it?"

Attacks: cross-subfield interest (the explicit AER bar); whether the first three pages are self-contained; under-interpreted results (coefficients never converted to economic meaning — aer-paper-body rules); exhibit overload or disorder; the conclusion overreaching the evidence; external validity left unaddressed.

Report format (each referee)

SUMMARY: <2-3 sentences — the paper as the referee understood it>
MAJOR COMMENTS: <numbered; each one: quote or cite the page/table,
  state the problem, state what evidence would resolve it>
MINOR COMMENTS: <numbered, brief>
RECOMMENDATION: <reject | major revision | minor revision | accept>

Rules of engagement:

• Every major comment is anchored — it quotes the manuscript or names the exact table/figure. Unanchored vibes ("the paper feels thin") are banned.
• Every major comment is resolvable — it states what analysis, evidence, or rewrite would satisfy the referee. Comments with no resolution path are editor material, not referee material.
• Each referee must produce at least three major comments or explicitly certify, against their own checklist, why fewer exist. An AI reviewer that finds nothing major has defaulted to agreeable — restart that report with the prior dialed up.
• No praise sandwiches. One sentence of genuine strengths in the summary is the cap.

Stage 3 — Editor Synthesis and Verdict

Score the paper on the rubric in docs/referee-report-rubric.md (contribution, identification, data, robustness, magnitudes, exposition, integrity — each 0-5 with anchored definitions), then issue the decision the reports support:

RUBRIC SCORES: <dimension: score, ...>
VERDICT: <desk reject | reject after review | major R&R | minor R&R>
DECISION LETTER: <editor's letter, naming the comments that drove it>
REVISE LIST: <every major comment, deduplicated, ordered by severity:
  blocking → major → minor, each tagged with the skill that fixes it>

Calibration anchors (do not inflate):

• Any rubric dimension at 0-1 → reject. Identification ≤ 2 → reject; no robustness round fixes a broken design.
• Major R&R requires: contribution ≥ 3, identification ≥ 3, no dimension below 2. This is already a top-decile outcome for real submissions.
• Minor R&R from a simulation should be rare — if the first run returns minor R&R, suspect leniency and rerun Stage 2 with the priors sharpened.

The Loop

aer-consistency (all PASS)
   → aer-referee-sim
        → verdict reject? → route fixes:
              identification comments → aer-identification / aer-robustness
              novelty comments        → aer-literature / aer-topic-selection
              interpretation comments → aer-paper-body
              framing comments        → aer-introduction
              exhibit comments        → aer-tables-figures
        → revise → aer-consistency → aer-referee-sim (fresh reports)
   → verdict ≥ major R&R on a fresh run → aer-submission

Rerun with fresh reports each time — re-grading old comments measures compliance, not quality. Two consecutive runs at major-R&R-or-better, with no blocking comments, is the exit condition.

Honesty Constraints for the Simulation

• The reviewers attack the manuscript, not a summary of it. If the draft is too long to hold at once, review it section by section against each referee's checklist — never from recall.
• Findings of fact (a wrong conversion, a missing diagnostic, a scooped contribution) must be verified before they enter a report; a simulated referee who hallucinates a flaw costs a revision round.
• Report the verdict to the user unsoftened. "The simulation desk- rejected the draft for X" is the deliverable, not a diplomatic summary.
• The simulation cannot certify acceptance — only that the draft survives the attacks this skill knows how to mount. Say so in the output.

Common Failure Modes

• Referees that paraphrase the paper's own framing back as praise
• Three reports that are one report with three names — the priors and reading orders exist to prevent this
• Major comments that are really minor (citation formatting promoted to "major" pads the count without testing the paper)
• Re-running the simulation until it happens to pass, without changing the draft — variance is not improvement
• Treating the simulated verdict as a prediction of the real one rather than a lower bound on preparedness

Repository Resources

Bundled with the installed skill, no repository checkout needed --- read it before the repo resources below:

• references/scoring-rubric.md --- anchored 0-5 rubric, verdict mapping, desk-screen and report templates

When working from the AER-skills repository or plugin bundle, load only the relevant resource:

• Scoring rubric with anchored 0-5 definitions and a calibrated sample report: docs/referee-report-rubric.md
• Complete worked simulation (two runs, three reports, routed revise list): examples/referee-report-example.md
• Desk-screen checklist the Stage 1 editor runs: docs/desk-rejection-audit.md
• Identification red flags Referee 1 hunts: skills/aer-identification/SKILL.md and docs/methods-reference.md
• Referee-anticipation battery Referee 1 checks for completeness: skills/aer-robustness/SKILL.md
• Prose failure patterns the desk screen scans: docs/style-guide.md
• Response-letter conventions for acting on the revise list: skills/aer-rebuttal/SKILL.md

Handoff

DESK DECISION: <reject | sent to referees>
REFEREE RECOMMENDATIONS: <R1 / R2 / R3>
RUBRIC SCORES: <list>
VERDICT: <desk reject | reject | major R&R | minor R&R>
BLOCKING COMMENTS: <n — list>
REVISE LIST: <comment → skill routing>
NEXT SKILL: <routed fix skill | aer-submission if exit condition met>

Anti-Patterns

• Running the simulation as a checklist instead of as three hostile readers with different stakes
• Letting the model grade its own prose generously because it recognizes it
• Skipping Stage 1 because "we know it won't desk-reject" — the desk screen catches different failures than referees do
• Accepting one lucky major-R&R run as the exit condition
• Using the simulation's praise in the cover letter — its only export is the revise list