mathews-tom/literature-review

Literature Review

Systematic discovery, extraction, and synthesis of academic research on a defined topic.

When to use this skill vs. others

| Need | Skill | | ----------------------------------------------------- | ---------------------------------- | | Survey a research area, synthesize multiple papers | literature-review (this skill) | | Critique a single paper's methodology and claims | research-critique | | Audit a manuscript's formatting, structure, citations | manuscript-review | | Verify a manuscript's numbers trace to code | manuscript-provenance | | Search arXiv for papers matching a query | arxiv-search (utility) |

Workflow

Phase 1: Scope Definition

Before searching, establish the review boundaries:

1. Research question — What specific question does the review answer? Vague topics produce vague reviews. "What techniques exist for X" is weaker than "How do methods for X compare on metric Y across domains Z?"
2. Inclusion criteria — Define what counts:
   • Date range (e.g., 2020–present)
   • Publication type (peer-reviewed, preprints, both)
   • Domains/categories (e.g., cs.CL, cs.AI)
   • Minimum relevance threshold
3. Exclusion criteria — Define what does not count:
   • Tangentially related work
   • Non-primary sources (blog posts, tutorials) unless explicitly included
   • Duplicate or superseded versions
4. Expected output — What form should the review take? Narrative synthesis, tabular comparison, gap analysis, annotated bibliography, or related-work section?

Present the scope to the user for confirmation before proceeding.

Phase 2: Search & Discovery

Execute searches across available sources. Use multiple queries with varying specificity to avoid single-query blind spots.

Primary source: arXiv (via arxiv-search utility)

uv run --with arxiv python scripts/arxiv_search.py "QUERY" --max-results 30 --sort-by relevance

Vary queries systematically:

• Broad topic query: "retrieval augmented generation"
• Field-scoped query: ti:retrieval AND abs:generation AND cat:cs.CL
• Author-anchored query: au:lewis AND abs:retrieval (when key authors are known)
• Recency query: same terms with --sort-by submitted

Secondary sources (via web search/fetch):

• Semantic Scholar API: https://api.semanticscholar.org/graph/v1/paper/search?query=QUERY&limit=20&fields=title,authors,abstract,year,citationCount,externalIds
• Google Scholar (via web search): site:scholar.google.com QUERY
• Connected Papers (for citation graph exploration): https://www.connectedpapers.com/search?q=QUERY

Snowball strategy:

• Forward snowball: find papers that cite a key paper (Semantic Scholar citations endpoint)
• Backward snowball: follow the references of key papers
• Use citation count as a signal for influence, not quality

Phase 3: Screening & Filtering

For each discovered paper, apply the inclusion/exclusion criteria from Phase 1.

Produce a screening table:

| # | ID | Title | Authors | Year | Relevant? | Reason | | --- | ---------- | ----------- | ------------- | ---- | --------- | ------------------------------- | | 1 | 2301.07041 | Paper Title | Author et al. | 2023 | Yes | Directly addresses RQ | | 2 | 2302.12345 | Other Paper | Author B | 2023 | No | Tangential — focuses on X not Y |

Rules:

• Screen on title + abstract first. Read full paper only for borderline cases.
• When uncertain, include. It is cheaper to drop a paper later than to miss it.
• Track exclusion reasons — they inform the review's limitations section.
• Flag papers that appear in multiple search queries as likely high-relevance.

Phase 4: Data Extraction

For each included paper, extract a structured record:

- id: "2301.07041"
  title: "Paper Title"
  authors: ["Author One", "Author Two"]
  year: 2023
  venue: "NeurIPS 2023"
  research_question: "How does X affect Y?"
  methodology: "Controlled experiment with N=1000"
  key_findings:
    - "Finding 1 with quantitative result"
    - "Finding 2 with effect size"
  limitations: "Single-domain evaluation"
  relevance_to_rq: "Directly compares methods A and B on metric Y"
  citation_count: 142

Extraction discipline:

• Record what the paper demonstrates, not what it claims to demonstrate.
• Distinguish empirical findings (data-backed) from interpretive claims (author's framing).
• Note methodology details that enable cross-paper comparison (datasets, metrics, baselines).
• If the paper is available via pdf_url, read it for extraction. Do not extract from abstracts alone for included papers.

Phase 5: Synthesis

Transform extracted records into structured analysis. The synthesis method depends on the output format requested in Phase 1.

Thematic synthesis — Group papers by theme, approach, or finding:

• Identify recurring themes across papers
• Note where papers agree, disagree, or address different aspects
• Highlight methodological trends (what approaches are gaining/losing traction)

Comparative synthesis — Build comparison tables:

| Method | Paper(s) | Dataset | Metric | Result | Limitations | | -------- | -------- | ------- | ------ | ------ | --------------- | | Method A | [1], [3] | D1 | F1 | 0.85 | Domain-specific | | Method B | [2], [4] | D1, D2 | F1 | 0.82 | Requires X |

Chronological synthesis — Map the evolution of the field:

• What was the state of knowledge at time T?
• What shifted and why?
• Where is the field heading?

Gap analysis — Identify what is missing:

• Questions raised but not answered by existing work
• Methodological gaps (no one has tried approach X on problem Y)
• Domain gaps (studied in domain A but not B)
• Contradictions between studies that remain unresolved

Phase 6: Output

Produce the review document in the format specified in Phase 1.

Standard structure for a narrative review:

1. Introduction — Research question, scope, and motivation for the review
2. Search methodology — Databases searched, queries used, inclusion/exclusion criteria, screening results (N found → N screened → N included)
3. Findings — Thematic or chronological synthesis of included papers
4. Discussion — Cross-cutting analysis, trends, contradictions, gaps
5. Limitations of this review — Search scope restrictions, potential biases, papers not accessible
6. References — Full citation list for all included papers

Standard structure for a tabular review:

1. Summary table (all included papers with key metadata)
2. Comparison matrix (methods × metrics × results)
3. Gap analysis table (questions × coverage)
4. Reference list

Citation format: Default to Author et al. (Year) in-text with full references at the end. Adapt to the user's specified format (APA, Chicago, IEEE) if requested.

Quality Checks

Before delivering the review, verify:

• [ ] Every included paper has a structured extraction record
• [ ] Every claim in the synthesis is traceable to at least one extracted finding
• [ ] The gap analysis identifies at least one concrete research opportunity
• [ ] The review acknowledges its own limitations (search scope, access, biases)
• [ ] Citation format is consistent throughout
• [ ] No paper is cited that was excluded during screening
• [ ] Contradictions between papers are noted, not silently resolved by picking a side

Edge Cases

| Situation | Adaptation | | ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Very few papers found (<5) | The field may be nascent. Note this explicitly. Broaden search terms or check if the topic goes by different terminology. Consider adjacent fields. | | Too many papers found (>100) | Tighten inclusion criteria. Consider limiting to top venues, recent years, or high-citation papers. Produce a scoping review rather than exhaustive review. | | User provides a paper list instead of a topic | Skip Phase 2 (Search). Start from Phase 3 (Screening) with the provided list. | | User wants a related-work section for their own paper | Tailor synthesis to position the user's contribution. Organize by approaches the user's work builds on, alternatives it competes with, and gaps it fills. | | No full-text access to key papers | Extract from abstracts and note the limitation. Do not fabricate methodology details. Flag which papers were abstract-only in the extraction records. | | Interdisciplinary topic | Search across multiple category prefixes. Note when different fields use different terminology for the same concept. | | User asks for a "quick" literature review | Reduce Phase 2 to a single search query, Phase 3 to title-only screening, Phase 4 to abstract-only extraction. Label the output as a preliminary survey, not a systematic review. |