brycewang-stanford/academic-paper-search

Academic Paper Search

A unified toolkit for discovering, retrieving metadata, and downloading academic papers across all major sources. The right API depends on the task — this skill routes to the best source.

Decision Tree: Which API to Use

What is needed?
│
├─ SEARCH for papers on a topic
│  ├─ arXiv preprints (CS/AI/Physics/Econ) ──→ arXiv API
│  ├─ NBER working papers (economics) ───────→ NBER API
│  ├─ SSRN working papers (social science) ──→ OpenAlex (primary) + CrossRef prefix:10.2139
│  ├─ Published journal articles ────────────→ CrossRef API or OpenAlex
│  └─ Broad multi-source search ─────────────→ OpenAlex API
│
├─ GET METADATA for a known paper
│  ├─ Have DOI ──→ CrossRef (structured) or OpenAlex (richer)
│  ├─ Have arXiv ID ──→ arXiv API
│  ├─ Have NBER number ──→ NBER API
│  └─ Have SSRN abstract ID ──→ OpenAlex DOI lookup (DOI = 10.2139/ssrn.{id})
│
├─ FIND/DOWNLOAD PDF
│  ├─ Have DOI ──→ Unpaywall API (check OA) → OpenAlex (alt PDF links)
│  ├─ arXiv paper ──→ Direct: https://arxiv.org/pdf/{id}
│  ├─ NBER paper ──→ Direct: https://www.nber.org/system/files/working_papers/w{n}/w{n}.pdf
│  └─ SSRN paper ──→ Unpaywall (~30%) or institutional repos (EconStor, IZA)
│
└─ IMPORT TO ZOTERO
   └─ Always use zotero_add_by_doi (best metadata + auto PDF via Unpaywall cascade)

---

1. OpenAlex API (Best All-Round)

Free, no API key needed. Covers 250M+ works including SSRN, NBER, arXiv, all journals. Add [email protected] for polite pool (faster responses).

Search by Source

# SSRN papers (source ID: S4210172589, NOT S4306400806 which is Europe PMC)
curl -s "https://api.openalex.org/works?filter=primary_location.source.id:S4210172589,default.search:social+preferences,publication_year:2024-2026&per_page=25&[email protected]"

# NBER papers
curl -s "https://api.openalex.org/works?filter=primary_location.source.id:S4210174556,default.search:behavioral+economics&per_page=25&[email protected]"

Search by Topic (All Sources)

curl -s "https://api.openalex.org/works?search=network+formation+game+theory&filter=publication_year:2024-2026&per_page=25&sort=cited_by_count:desc&[email protected]"

DOI Lookup

curl -s "https://api.openalex.org/works/doi:10.2139/[email protected]"

Python Pattern

import requests

def openalex_search(query, source_id=None, year_from=2024, year_to=2026, per_page=25):
    """Search OpenAlex. source_id: S4210172589 (SSRN), S4210174556 (NBER), etc."""
    filters = [f"default.search:{query}", f"publication_year:{year_from}-{year_to}"]
    if source_id:
        filters.insert(0, f"primary_location.source.id:{source_id}")
    params = {
        "filter": ",".join(filters),
        "per_page": per_page,
        "mailto": "[email protected]"
    }
    data = requests.get("https://api.openalex.org/works", params=params).json()
    results = []
    for w in data.get("results", []):
        pdf_urls = [loc["pdf_url"] for loc in w.get("locations", []) if loc.get("pdf_url")]
        results.append({
            "title": w["title"],
            "doi": w.get("doi", "").replace("https://doi.org/", ""),
            "year": w.get("publication_year"),
            "authors": [a["author"]["display_name"] for a in w.get("authorships", [])],
            "cited_by": w.get("cited_by_count", 0),
            "pdf_urls": pdf_urls,
            "landing_url": (w.get("primary_location") or {}).get("landing_page_url", ""),
        })
    return {"total": data.get("meta", {}).get("count", 0), "results": results}

Key Source IDs

Full reference: See references/journal_identifiers.md for a comprehensive table of 77 verified OpenAlex Source IDs and CrossRef ISSNs covering Top 5 economics, AEA journals, finance, management, marketing, accounting, IS, and 15+ field journal categories.

| Source | OpenAlex ID | CrossRef ISSN | |--------|-------------|---------------| | SSRN Electronic Journal | S4210172589 | 1556-5068 | | NBER Working Papers | S2809516038 | N/A | | American Economic Review | S23254222 | 0002-8282 | | Quarterly Journal of Economics | S203860005 | 0033-5533 | | Econometrica | S95464858 | 0012-9682 |

Warning: OpenAlex source IDs are opaque and not guessable. Always verify via https://api.openalex.org/sources?search=journal+name before using a new ID. For verified IDs of 77 journals, consult references/journal_identifiers.md.

PDF from OpenAlex

primary_location.pdf_url is null for SSRN papers (Elsevier blocks it). Check locations[].pdf_url for third-party repos — ~24% of SSRN papers have PDFs via EconStor, IZA, MPRA, etc. These are real, downloadable PDFs.

---

2. CrossRef API (Best for DOI Metadata)

Free, no key needed. Add mailto= for 50 req/sec (vs 1 req/sec without).

Search with SSRN DOI Prefix

# All SSRN papers matching a query
curl -s "https://api.crossref.org/works?query=%22social+preferences%22&filter=prefix:10.2139,from-pub-date:2024-01-01&rows=25&sort=relevance&[email protected]"

Search within a Journal

# See references/journal_identifiers.md for 77 verified ISSNs
# AER: 0002-8282, QJE: 0033-5533, JPE: 0022-3808, Econometrica: 0012-9682, REStud: 0034-6527
curl -s "https://api.crossref.org/journals/0002-8282/works?query=auction+mechanism&rows=10&[email protected]"

DOI Lookup

curl -s "https://api.crossref.org/works/10.1257/[email protected]"

DOI Construction Rules

| Source | DOI Format | Example | |--------|-----------|---------| | SSRN | 10.2139/ssrn.{abstract_id} | 10.2139/ssrn.4812743 | | NBER | 10.3386/w{number} | 10.3386/w34297 |

These predictable DOI formats allow constructing DOIs from paper IDs without making extra API calls.

Limitations

CrossRef returns title, authors, DOI, date, journal — but no abstracts for SSRN papers (Elsevier doesn't submit them to CrossRef). Use OpenAlex for richer metadata.

---

3. arXiv API (Preprints — CS/AI/Physics/Math/Econ)

Free, no key needed. Returns Atom XML. Rate limit: ~1 request per 3 seconds.

curl -s "https://export.arxiv.org/api/query?search_query=abs:social+preferences+AND+(cat:econ.TH+OR+cat:econ.GN)&max_results=20&sortBy=submittedDate&sortOrder=descending"

Search Prefixes

all: (full text), ti: (title), au: (author), abs: (abstract), cat: (category). Boolean: AND, OR, ANDNOT. Exact phrase: %22social+preferences%22.

Economics Categories

econ.TH (Theory), econ.GN (General), econ.EM (Econometrics).

Caveat: The cat: filter in search_query does not always reliably restrict results to specified categories — arXiv may still return papers from other fields. Verify category matches in results and filter client-side if needed.

Direct PDF

https://arxiv.org/pdf/{id}          # e.g. https://arxiv.org/pdf/2401.12345
https://arxiv.org/abs/{id}          # abstract page

---

4. NBER API (Economics Working Papers)

Free, no key needed. Returns JSON.

# Search
curl -s "https://www.nber.org/api/v1/working_page_listing/contentType/working_paper/_/_/search?page=1&perPage=20&q=social+preferences"

# Filter by Program (replace _/_ with programs/{name})
curl -s "https://www.nber.org/api/v1/working_page_listing/contentType/working_paper/programs/Labor+Studies/search?page=1&perPage=20&q=minimum+wage"

# New this week
curl -s "https://www.nber.org/api/v1/working_page_listing/contentType/working_paper/_/_/search?page=1&perPage=20&newThisWeek=true"

Programs

Labor Studies, Economic Fluctuations and Growth, Industrial Organization, Public Economics, Development Economics, International Finance and Macroeconomics, Corporate Finance, Asset Pricing, Health Economics, etc.

Response Processing

Authors field contains HTML <a> tags — strip with: re.sub(r'<[^>]+>', '', text)

No built-in date filter — API sorts by relevance. Filter client-side by displaydate.

Direct PDF

https://www.nber.org/system/files/working_papers/w{number}/w{number}.pdf

---

5. Unpaywall API (Open Access PDF Finder)

Free, no key needed. Email as query param. 100,000 requests/day.

curl -s "https://api.unpaywall.org/v2/10.2139/[email protected]" \
  | python3 -c "
import sys, json
d = json.load(sys.stdin)
print(f'OA: {d[\"is_oa\"]} ({d[\"oa_status\"]})')
pdf = (d.get('best_oa_location') or {}).get('url_for_pdf')
print(f'PDF: {pdf or \"none\"}')
# Check all locations for alternative PDFs
for loc in d.get('oa_locations', []):
    if loc.get('url_for_pdf'):
        print(f'Alt PDF: {loc[\"url_for_pdf\"]} ({loc[\"host_type\"]})')
"

OA Status Values

| Status | Meaning | |--------|---------| | gold | Published OA by publisher | | green | Free copy in a repository | | hybrid | OA article in subscription journal | | bronze | Free to read on publisher site | | closed | No free version found |

Platform-Specific PDF Availability

| Source | Unpaywall PDF Rate | Why | |--------|-------------------|-----| | arXiv | ~100% | Native OA, direct PDF links | | NBER | ~97% | Green OA via author/institution repos | | SSRN | ~30% | Cloudflare blocks Unpaywall; only finds institutional repo mirrors |

For SSRN papers where Unpaywall returns url_for_pdf=null, check oa_locations[] for institutional repositories (EconStor, university repos) — follow their landing pages to find PDF links.

---

6. Semantic Scholar API (Citations & Impact)

Free, no key needed. Rate limit: 100 requests per 5 minutes. Warning: As of 2026, Semantic Scholar may return 429 errors frequently. Consider registering for an API key at https://www.semanticscholar.org/product/api for higher limits. Add header: x-api-key: YOUR_KEY.

# Search
curl -s "https://api.semanticscholar.org/graph/v1/paper/search?query=social+preferences+experiment&fields=title,year,authors,citationCount,externalIds,venue&limit=20&year=2024-2026"

# DOI lookup
curl -s "https://api.semanticscholar.org/graph/v1/paper/DOI:10.1257/aer.20171330?fields=title,year,authors,citationCount,abstract"

Field of Study Filter

fieldsOfStudy=Economics,Sociology,Political+Science,Business,Psychology

Warning: The Economics filter is very strict and may return 0 results. Often better to search without it and filter manually.

When to Use

Best for: citation counts, finding related papers, author impact metrics. Not best for: SSRN-specific search (poor coverage), bulk downloading.

---

7. Tavily Search (Fallback for Any Source)

When APIs do not cover the need, Tavily can search any academic site. Particularly useful for SSRN (no public API) and as a fallback for any source.

# Via MCP tool
mcp__tavily__tavily-search({
    "query": 'site:ssrn.com "social preferences" economics 2024 2025',
    "search_depth": "advanced",
    "max_results": 20,
    "include_domains": ["ssrn.com", "papers.ssrn.com"]
})

# Also works for NBER, RePEc, etc.
mcp__tavily__tavily-search({
    "query": 'site:nber.org "network formation" working paper',
    "include_domains": ["nber.org"]
})

---

Recommended Workflows

Workflow A: Comprehensive Topic Search

For finding all recent papers on a topic across multiple sources:

1. OpenAlex: broad search (all sources, richest metadata)
2. CrossRef prefix:10.2139: catch SSRN papers OpenAlex might miss
3. NBER API: NBER-specific search with program filtering
4. arXiv API: preprints in CS/AI/Physics/Econ
5. Tavily: sweep for anything the APIs missed
6. Deduplicate by DOI

Workflow B: SSRN Paper Search

SSRN has no public API. Use this cascade:

1. OpenAlex (filter: source.id:S4210172589) — best: 1 API call, rich metadata
2. CrossRef (filter: prefix:10.2139) — backup: structured DOI search
3. Tavily (include_domains: ssrn.com) — catches papers not yet in OpenAlex/CrossRef

Workflow C: PDF Download Pipeline

Try sources in this order (most reliable first):

1. arXiv direct: https://arxiv.org/pdf/{id}
2. NBER direct: https://www.nber.org/system/files/working_papers/w{n}/w{n}.pdf
3. Unpaywall: check best_oa_location.url_for_pdf via DOI
4. OpenAlex: check locations[].pdf_url for third-party repo PDFs
5. Institutional repos: follow Unpaywall oa_locations landing pages
6. SSRN direct: construct Delivery.cfm URL (may be Cloudflare-blocked)

Workflow D: Zotero Import

Always prefer zotero_add_by_doi over zotero_add_by_url:

• Better metadata (correct entry types, complete fields)
• Higher PDF download success (Zotero uses Unpaywall → arXiv → S2 → PMC cascade)
• DOI can be constructed for SSRN (10.2139/ssrn.{id}) and NBER (10.3386/w{number})

---

Rate Limits & Best Practices

| API | Rate Limit | Best Practice | |-----|-----------|---------------| | OpenAlex | 10 req/sec (polite pool) | Add mailto= parameter | | CrossRef | 50 req/sec with mailto=, 1/sec without | Always add mailto= | | Unpaywall | 100K/day | Burst OK, no throttling needed | | arXiv | ~1 req/3sec | Add time.sleep(3) between requests | | NBER | No documented limit | Be polite, ~1 req/sec | | Semantic Scholar | 100 req/5min | Add time.sleep(3) or use API key for higher limit | | Tavily | Per plan | Use search_depth: "basic" when possible |