brycewang-stanford/paper-critique-framework

Paper Critique and Peer Review Framework

Overview

Writing constructive peer reviews is a core academic skill. This framework provides a systematic approach to evaluating manuscripts — from initial read-through to the final referee report. It covers what reviewers should assess, how to structure feedback, and how to calibrate between different review outcomes (accept, revise, reject). Applicable to conference papers, journal articles, and internal lab reviews.

The Three-Pass Review Method

Pass 1: Orientation (15-20 minutes)

Read only these elements:

• Title, abstract, and keywords
• Introduction (first and last paragraphs)
• Section headings and figure captions
• Conclusion

After Pass 1, answer:

□ What is the main claim?
□ What type of contribution? (empirical, theoretical, system, survey)
□ Is it within the venue's scope?
□ Does the abstract accurately represent the content?
□ Initial impression: novel or incremental?

Pass 2: Detailed Read (60-90 minutes)

Read the full paper. Annotate as you go:

Annotation symbols:
  ? = I don't understand this
  ! = This is interesting / strong point
  X = I disagree / see a problem
  → = This needs more evidence or justification
  ≈ = This is similar to [existing work] — check novelty

Focus on:

• Claims vs. evidence: Is every major claim supported by data?
• Methodology: Are the methods appropriate for the research question?
• Experimental design: Are baselines fair? Are ablations sufficient?
• Figures and tables: Do they support the narrative? Are they readable?
• Writing quality: Is it clear, concise, and well-organized?

Pass 3: Verification (30-60 minutes)

For papers you're seriously evaluating:

• Check key references — do they say what the authors claim?
• Verify mathematical derivations (spot-check, not exhaustive)
• Examine statistical claims (p-values, confidence intervals, effect sizes)
• Check for cherry-picking in results (only best runs? selected metrics?)
• Look for missing baselines that should have been compared

Review Report Structure

## Summary (3-5 sentences)
[Describe what the paper does, the approach, and the main finding.
 Demonstrate that you understood the paper.]

## Strengths (bulleted list)
- S1: [Specific strength with evidence from the paper]
- S2: [Another strength]
- S3: [Another strength]

## Weaknesses (bulleted list, ordered by severity)
- W1 (Major): [Specific weakness + why it matters + suggestion to fix]
- W2 (Major): [Another major weakness]
- W3 (Minor): [A less critical issue]
- W4 (Minor): [Another minor issue]

## Questions for Authors
- Q1: [Something you'd like clarified]
- Q2: [A concern that the authors might be able to address]

## Detailed Comments
[Page/line-specific comments, typos, suggestions]

## Overall Assessment
Recommendation: [Strong Accept / Accept / Weak Accept / Borderline /
                  Weak Reject / Reject / Strong Reject]
Confidence: [High / Medium / Low]

Assessment Criteria by Dimension

| Dimension | Questions to Ask | Weight | |-----------|-----------------|--------| | Novelty | Is the idea new? Is the contribution beyond incremental? | High | | Significance | Would this matter to the community? Does it advance the field? | High | | Soundness | Are the methods correct? Are conclusions supported? | High | | Clarity | Is it well-written? Can it be understood and reproduced? | Medium | | Completeness | Are related works covered? Are experiments thorough? | Medium | | Reproducibility | Could someone replicate this? Code/data available? | Medium |

Calibration Guide

Strong Accept: Significant contribution, technically sound, well-written.
  Would be a highlight of the venue.

Accept: Solid contribution with minor issues. Advances the field.
  Worth publishing as-is or with minor revisions.

Weak Accept: Has merit but notable weaknesses. Contribution is real but modest.
  Borderline for this venue; would be accepted at a less selective venue.

Borderline: Equal arguments for and against. Significant weaknesses offset
  by some novelty. Depends on other reviews.

Weak Reject: Interesting direction but fundamental issues not addressed.
  Major revisions needed that likely require a new submission cycle.

Reject: Significant problems in novelty, soundness, or relevance.
  Not suitable for this venue even with revisions.

Strong Reject: Fundamental flaws. Clearly below threshold.

Common Review Pitfalls to Avoid

| Pitfall | Better Approach | |---------|----------------| | "The writing needs improvement" (vague) | Give 2-3 specific examples with suggested fixes | | Rejecting for not solving YOUR problem | Evaluate the paper on its own stated goals | | Demanding impossible experiments | Suggest feasible improvements within scope | | Ignoring supplementary material | Check appendix — authors may have addressed your concern | | Being harsh without being constructive | Every weakness should include a suggestion for improvement | | Reviewing too quickly | Block dedicated time; a rushed review harms both authors and science | | Citing only your own work as "missing" | Only cite if genuinely relevant, not self-promotion |

Reviewing Different Paper Types

Empirical Papers

• Are datasets described completely? (Size, source, splits, preprocessing)
• Are baselines appropriate and fairly tuned?
• Statistical significance: error bars, multiple runs, significance tests
• Ablation studies: which components contribute to the gain?

Systems Papers

• Is the system actually built and evaluated (not just designed)?
• Performance at scale: latency, throughput, resource usage
• Comparison with existing systems (not just algorithms)
• Deployment experience or case studies

Survey/Position Papers

• Coverage: are major relevant works included?
• Taxonomy: is the organizational framework useful and complete?
• Analysis: does the survey add value beyond listing papers?
• Recency: are the most recent advances covered?

Ethics Considerations

During review, check:
□ Are human subjects involved? Was IRB/ethics approval obtained?
□ Are there potential harms from the technology described?
□ Is the data collection ethical? (Consent, privacy, bias)
□ Are dual-use concerns addressed? (Misuse potential)
□ Are limitations and societal implications discussed?

References

• Smith, R. (2006). "Peer review: a flawed process at the heart of science and journals." Journal of the Royal Society of Medicine, 99(4), 178-182.
• NeurIPS Reviewer Guidelines
• ACL Reviewer Tutorial
• Cormode, G. (2009). "How NOT to review a paper." ACM SIGMOD Record, 37(4), 100-104.