brycewang-stanford/arxiv-latex-source

arXiv LaTeX Source Access Guide

Overview

arXiv stores the original LaTeX source files for the vast majority of its 2.4 million+ preprints. Accessing LaTeX source provides major advantages over PDF parsing: exact mathematical notation as written by the author, structured sections and labels, machine-readable bibliography entries, and intact figure captions, table data, and cross-references.

For formula extraction, citation graph construction, section-level text analysis, or training data curation for scientific language models, LaTeX source is the gold standard. PDF parsing introduces OCR errors in equations, loses structural hierarchy, and mangles complex tables.

The e-print endpoint serves source bundles as gzip-compressed tarballs (.tar.gz) containing .tex files, figures, .bib/.bbl bibliography files, style files, and supplementary materials. No authentication is required.

Authentication

No authentication or API key is required. The e-print endpoint is publicly accessible. However, arXiv asks that automated tools set a descriptive User-Agent header and comply with rate limits.

Core Endpoints

Download LaTeX Source

• URL: GET https://arxiv.org/e-print/{arxiv_id}

• Response: application/gzip — a .tar.gz archive containing the source files

• Parameters: | Param | Type | Required | Description | |-------|------|----------|-------------| | arxiv_id | string | Yes | arXiv identifier, e.g. 2301.00001 or 2301.00001v2 for a specific version |

• Example:

  # Download source archive (response: 200, application/gzip, ~1.3 MB)
  curl -sL -o source.tar.gz "https://arxiv.org/e-print/2301.00001"
  
  # List archive contents
  tar tz -f source.tar.gz | head -10
  # ACM-Reference-Format.bbx
  # ACM-Reference-Format.bst
  # Image_1.jpg
  # README.txt
  # acmart.cls
  

• Content-Disposition header: attachment; filename="arXiv-2301.00001v1.tar.gz"

• ETag: SHA-256 hash provided for caching: sha256:f1ffe8ec...

Format Detection

The endpoint almost always returns a gzip-compressed tar archive. Rare cases (very old or single-file submissions) may return a single gzip-compressed .tex file without tar wrapper. Always verify format before extracting:

curl -sL "https://arxiv.org/e-print/{arxiv_id}" -o source.gz
file source.gz  # "gzip compressed data, was 'XXXX.tar', ..."

Metadata API (Companion)

Pair source downloads with the arXiv Atom API for structured metadata:

• URL: GET https://export.arxiv.org/api/query?id_list={arxiv_id}
• Response: Atom XML with <title>, <author>, <summary>, <category>, <published>
• Example: curl -s "https://export.arxiv.org/api/query?id_list=2301.00001"

LaTeX Source Parsing Guide

Locating the Main .tex File

A source archive typically contains multiple files. To find the main document:

1. Look for \documentclass in .tex files — this marks the root document
2. Check for a README.txt that may specify the main file
3. If multiple .tex files contain \documentclass, prefer the one with \begin{document}

import tarfile, re

def find_main_tex(tar_path):
    with tarfile.open(tar_path, 'r:gz') as tar:
        tex_files = [m for m in tar.getmembers() if m.name.endswith('.tex')]
        for member in tex_files:
            content = tar.extractfile(member).read().decode('utf-8', errors='ignore')
            if r'\documentclass' in content and r'\begin{document}' in content:
                return member.name, content
    return None, None

Extracting Sections

LaTeX sections follow a predictable hierarchy:

import re

def extract_sections(tex_content):
    pattern = r'\\(section|subsection|subsubsection)\{([^}]+)\}'
    sections = re.findall(pattern, tex_content)
    return [(level, title) for level, title in sections]
    # [('section', 'Introduction'), ('section', 'Related Work'), ...]

Extracting Equations

def extract_equations(tex_content):
    patterns = [
        r'\\\[(.+?)\\\]',
        r'\\begin\{equation\}(.+?)\\end\{equation\}',
        r'\\begin\{align\*?\}(.+?)\\end\{align\*?\}',
    ]
    equations = []
    for pat in patterns:
        equations.extend(re.findall(pat, tex_content, re.DOTALL))
    return equations

Extracting Bibliography

Parse .bib files (BibTeX entries) or .bbl files (compiled \bibitem commands):

def extract_bibliography(tar_path):
    refs = []
    with tarfile.open(tar_path, 'r:gz') as tar:
        for member in tar.getmembers():
            if member.name.endswith('.bib'):
                content = tar.extractfile(member).read().decode('utf-8', errors='ignore')
                refs.extend(re.findall(r'@\w+\{([^,]+),(.+?)\n\}', content, re.DOTALL))
            elif member.name.endswith('.bbl'):
                content = tar.extractfile(member).read().decode('utf-8', errors='ignore')
                refs.extend(re.findall(r'\\bibitem.*?\{(.+?)\}', content))
    return refs

Rate Limits

• Maximum: 4 requests per second for automated access
• Recommended: 1 request/second with delays between sequential downloads
• Bulk access: For 1000+ papers, use the arXiv S3 bulk data mirror instead
• HTTP 429: Rate limit exceeded; implement exponential backoff
• User-Agent: Required — set a descriptive string: MyTool/1.0 (mailto:[email protected])
• Persistent abuse may result in IP-level blocks

Academic Use Cases

• Formula extraction for ML training — Build equation datasets with ground-truth LaTeX notation, free of OCR noise from PDF parsing
• Citation network analysis — Parse .bib/.bbl files for exact reference keys to construct citation graphs
• Section-level text analysis — Extract specific sections (e.g., all "Related Work" across a subfield) for systematic reviews
• Reproducibility auditing — Examine algorithm environments, hyperparameter tables, and methodology sections
• Cross-paper notation alignment — Compare and normalize equation environments across papers in a subfield

Complete Python Example

import requests, tarfile, io, re, time, gzip

def download_arxiv_source(arxiv_id, delay=1.0):
    """Download and extract all .tex files from an arXiv paper's source."""
    url = f"https://arxiv.org/e-print/{arxiv_id}"
    headers = {"User-Agent": "ResearchTool/1.0 (mailto:[email protected])"}
    resp = requests.get(url, headers=headers)
    resp.raise_for_status()
    time.sleep(delay)

    buf = io.BytesIO(resp.content)
    try:
        with tarfile.open(fileobj=buf, mode='r:gz') as tar:
            return {m.name: tar.extractfile(m).read().decode('utf-8', errors='ignore')
                    for m in tar.getmembers() if m.name.endswith('.tex') and m.isfile()}
    except tarfile.ReadError:
        buf.seek(0)
        return {"main.tex": gzip.decompress(buf.read()).decode('utf-8', errors='ignore')}

# Usage
sources = download_arxiv_source("2301.00001")
for fname, content in sources.items():
    if r'\documentclass' in content:
        sections = re.findall(r'\\section\{([^}]+)\}', content)
        equations = re.findall(r'\\begin\{equation\}(.+?)\\end\{equation\}', content, re.DOTALL)
        print(f"{fname}: {len(sections)} sections, {len(equations)} equations")

References

• arXiv e-print access: https://info.arxiv.org/help/bulk_data_s3.html
• arXiv API documentation: https://info.arxiv.org/help/api/index.html
• arXiv terms of use: https://info.arxiv.org/help/api/tou.html
• arXiv S3 bulk data: https://info.arxiv.org/help/bulk_data_s3.html