jaechang-hits/ena-database

ENA Database — European Nucleotide Archive Programmatic Access

Overview

The European Nucleotide Archive (ENA) is EMBL-EBI's comprehensive nucleotide sequence database, encompassing raw sequencing reads, genome assemblies, annotated sequences, and associated metadata. It mirrors and extends INSDC data (GenBank, DDBJ). All access is via REST APIs with no authentication required.

When to Use

• Searching for sequencing studies, samples, or experiments by organism, project, or keyword
• Downloading raw FASTQ/BAM files for reanalysis of public sequencing datasets
• Retrieving genome assemblies with quality statistics (N50, contig count, genome size)
• Fetching nucleotide sequences in FASTA or EMBL flat-file format by accession
• Exploring taxonomic lineage and finding organisms by partial name
• Cross-referencing ENA records with external databases (ArrayExpress, UniProt, PDB)
• Building bulk download lists for large-scale sequencing projects
• For multi-database Python queries (ENA + UniProt + KEGG), prefer bioservices instead
• For NCBI-specific queries (PubMed literature, GenBank records), use pubmed-database or Biopython Entrez

Prerequisites

pip install requests

API constraints:

• Rate limit: 50 requests per second across all ENA APIs
• No authentication required
• Large result sets: use pagination (limit + offset) or streaming (limit=0 for TSV download)
• Portal API base: https://www.ebi.ac.uk/ena/portal/api
• Browser API base: https://www.ebi.ac.uk/ena/browser/api
• Taxonomy API base: https://www.ebi.ac.uk/ena/taxonomy/rest
• Cross-ref API base: https://www.ebi.ac.uk/ena/xref/rest

Quick Start

import requests
import time

BASE_PORTAL = "https://www.ebi.ac.uk/ena/portal/api"
BASE_BROWSER = "https://www.ebi.ac.uk/ena/browser/api"
BASE_TAXONOMY = "https://www.ebi.ac.uk/ena/taxonomy/rest"
BASE_XREF = "https://www.ebi.ac.uk/ena/xref/rest"

def ena_query(endpoint, params=None, base=BASE_PORTAL):
    """Reusable ENA API caller with rate-limit compliance."""
    resp = requests.get(f"{base}/{endpoint}", params=params)
    resp.raise_for_status()
    time.sleep(0.02)  # 50 req/sec limit
    return resp

# Search for human RNA-seq studies
resp = ena_query("search", params={
    "result": "study",
    "query": 'tax_tree(9606)',   # `library_strategy` is a `read_run`/`read_experiment` field, not a `study` field
    "fields": "study_accession,study_title",
    "format": "json",
    "limit": 3,
})
studies = resp.json()
for s in studies:
    print(f"{s['study_accession']}: {s['study_title'][:60]}")
# PRJEB12345: Transcriptome analysis of human liver tissue...

Core API

Module 1: Portal API Search

The Portal API provides advanced metadata search across all ENA data types with boolean query syntax, field selection, and pagination.

# Search read runs for a specific study
resp = ena_query("search", params={
    "result": "read_run",
    "query": 'study_accession="PRJEB1787"',
    "fields": "run_accession,sample_accession,instrument_model,read_count,base_count",
    "format": "json",
    "limit": 5,
})
runs = resp.json()
for r in runs:
    print(f"{r['run_accession']} — {r.get('instrument_model', 'N/A')}, "
          f"{int(r.get('read_count', 0)):,} reads")
# ERR123456 — Illumina HiSeq 2000, 45,231,890 reads

# Count total results without fetching data
count_resp = ena_query("count", params={
    "result": "read_run",
    "query": 'study_accession="PRJEB1787"',
})
print(f"Total runs: {count_resp.text.strip()}")
# Total runs: 142

Module 2: Browser API Retrieval

Fetch individual records by accession in multiple formats: XML, FASTA, EMBL flat-file, or plain text.

# Retrieve XML metadata for a study
resp = ena_query("xml/PRJEB1787", base=BASE_BROWSER)
print(resp.text[:300])
# <?xml version="1.0" encoding="UTF-8"?><PROJECT_SET>...

# Retrieve FASTA sequence for a coding sequence
resp = ena_query("fasta/M10051.1", base=BASE_BROWSER)
print(resp.text[:200])
# >ENA|M10051|M10051.1 Human insulin mRNA, complete cds.
# AGCCCTCCAGGACAGGCTGCAT...

# Retrieve EMBL flat-file format
resp = ena_query("embl/M10051.1", base=BASE_BROWSER)
print(resp.text[:300])
# ID   M10051; SV 1; linear; mRNA; STD; HUM; 786 BP.
# ...

Module 3: File Reports and Downloads

Get download URLs for FASTQ, submitted, and analysis files. File reports return FTP and Aspera paths.

# Get FASTQ file URLs for specific runs
resp = ena_query("filereport", params={
    "accession": "ERR000589",
    "result": "read_run",
    "fields": "run_accession,fastq_ftp,fastq_bytes,fastq_md5",
    "format": "json",
})
files = resp.json()
for f in files:
    ftp_urls = f.get("fastq_ftp", "").split(";")
    sizes = f.get("fastq_bytes", "").split(";")
    for url, size in zip(ftp_urls, sizes):
        if url:
            print(f"ftp://{url}  ({int(size)/1e6:.1f} MB)")
# ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR000/ERR000589/ERR000589_1.fastq.gz  (234.5 MB)
# ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ERR000/ERR000589/ERR000589_2.fastq.gz  (241.2 MB)

Module 4: Taxonomy Queries

Look up organisms by taxonomy ID, scientific name, or partial name match.

# Lookup by taxonomy ID
resp = ena_query("tax-id/9606", base=BASE_TAXONOMY)
tax = resp.json()
print(f"{tax['scientificName']} (taxId: {tax['taxId']}, rank: {tax['rank']})")
# Homo sapiens (taxId: 9606, rank: species)
print(f"Lineage: {tax['lineage'][:80]}...")

# Search by scientific name — endpoint returns a list (one entry per matching taxon)
resp = ena_query("scientific-name/Arabidopsis thaliana", base=BASE_TAXONOMY)
matches = resp.json()
result = matches[0] if isinstance(matches, list) else matches
print(f"Tax ID: {result['taxId']}, Common: {result.get('commonName', 'N/A')}")
# Tax ID: 3702, Common: thale cress

# Suggest organisms by partial name
resp = ena_query("suggest-for-search/salmo", base=BASE_TAXONOMY)
suggestions = resp.json()
for s in suggestions[:3]:
    print(f"  {s['scientificName']} (taxId: {s['taxId']})")
# Salmo salar (taxId: 8030)
# Salmo trutta (taxId: 8032)
# Salmonella enterica (taxId: 28901)

Module 5: Cross-Reference Service

Find links between ENA records and external databases (ArrayExpress, UniProt, PDB, etc.).

# Find cross-references for an ENA accession
resp = ena_query("json/search", base=BASE_XREF, params={
    "accession": "M10051",
})
xrefs = resp.json()
for x in xrefs[:5]:
    print(f"  {x['Source']} → {x['Source Primary Accession']} "
          f"({x.get('Source Description', '')[:50]})")
# UniProt → P01308 (Insulin precursor)
# PDB → 1A7F (Crystal structure of human insulin)

# Search cross-references by external database
resp = ena_query("json/search", base=BASE_XREF, params={
    "source": "UniProt",
    "accession": "P01308",
})
xrefs = resp.json()
for x in xrefs[:3]:
    print(f"  ENA: {x['Target Primary Accession']} — {x.get('Target Description', '')[:60]}")

Module 6: CRAM Reference Registry

Retrieve reference sequences used in CRAM files by MD5 or SHA1 checksum. Essential for CRAM decompression.

# Look up reference by MD5 checksum
md5 = "aef131c3b4b05d8e2b3f907faba5af9b"  # example
try:
    resp = ena_query(
        f"cram/md5/{md5}",
        base="https://www.ebi.ac.uk/ena/cram"
    )
    print(f"Reference found: {len(resp.content)} bytes")
except requests.HTTPError as e:
    if e.response.status_code == 404:
        print("Reference not found — check MD5 checksum")
    else:
        raise

Key Concepts

ENA Data Hierarchy

| Level | Accession Prefix | Description | Contains | |-------|-----------------|-------------|----------| | Study | PRJEB/ERP | Research project | Samples, Experiments | | Sample | ERS/SAMEA | Biological sample | Metadata, taxonomy | | Experiment | ERX | Library/sequencing setup | Runs | | Run | ERR | Sequencing run | Raw read files (FASTQ) | | Analysis | ERZ | Derived analysis | Assemblies, alignments | | Assembly | GCA | Genome assembly | Contigs, scaffolds | | Sequence | Accession.version | Annotated sequence | Features, coding seqs |

Query Syntax Operators

| Operator | Example | Description | |----------|---------|-------------| | Equality | instrument_model="Illumina NovaSeq 6000" | Exact match | | Wildcard | study_title="*melanoma*" | Partial match | | Range | base_count>=1000000 | Numeric comparison | | Taxonomy tree | tax_tree(9606) | Taxon and all descendants | | Exact taxon | tax_eq(9606) | Exact taxon only | | Date range | first_public>=2023-01-01 | Date filtering | | Boolean | AND, OR, NOT | Combine conditions | | Grouping | (A OR B) AND C | Parenthetical grouping |

Result Types

| Result Type | Description | Key Fields | |------------|-------------|------------| | study | Research projects | study_accession, study_title, center_name | | sample | Biological samples | sample_accession, tax_id, scientific_name | | read_run | Sequencing runs | run_accession, read_count, base_count, fastq_ftp | | read_experiment | Experiments | experiment_accession, library_strategy, instrument_model | | analysis | Derived analyses | analysis_accession, analysis_type | | assembly | Genome assemblies | assembly_accession, assembly_level, genome_representation | | sequence | Annotated sequences | accession, sequence_length, mol_type | | wgs_set | WGS scaffold sets | set_accession, set_size | | tsa_set | Transcriptome assemblies | set_accession, set_size | | coding | Coding sequences | accession, gene, product | | noncoding | Non-coding features | accession, description | | taxon | Taxonomy entries | tax_id, scientific_name, lineage |

Discoverable Fields

Use the returnFields endpoint to discover available fields for any result type:

resp = ena_query("returnFields", params={"result": "read_run"})
fields = resp.text.strip().split("\n")
print(f"Available fields for read_run: {len(fields)}")
print(fields[:10])
# ['accession', 'altitude', 'assembly_quality', 'assembly_software', ...]

Common Workflows

Workflow 1: Study Exploration Pipeline

Search for a study, list its samples, then retrieve run metadata.

import json

# Step 1: Find studies by organism — `study_title="*…*"` wildcards no longer match;
# use `tax_tree()` against the species tax ID (SARS-CoV-2 = 2697049).
resp = ena_query("search", params={
    "result": "study",
    "query": 'tax_tree(2697049) AND first_public>=2023-01-01',
    "fields": "study_accession,study_title,center_name",
    "format": "json",
    "limit": 3,
})
studies = resp.json()
study_acc = studies[0]["study_accession"]
print(f"Selected: {study_acc} — {studies[0]['study_title'][:60]}")

# Step 2: List samples in the study
resp = ena_query("search", params={
    "result": "sample",
    "query": f'study_accession="{study_acc}"',
    "fields": "sample_accession,scientific_name,collection_date",
    "format": "json",
    "limit": 5,
})
samples = resp.json()
print(f"Found {len(samples)} samples (showing first 5)")
for s in samples:
    print(f"  {s['sample_accession']} — {s.get('scientific_name', 'N/A')}")

# Step 3: Get run metadata for each sample
for s in samples[:2]:
    resp = ena_query("search", params={
        "result": "read_run",
        "query": f'sample_accession="{s["sample_accession"]}"',
        "fields": "run_accession,instrument_model,read_count,library_strategy",
        "format": "json",
    })
    runs = resp.json()
    for r in runs:
        print(f"  {r['run_accession']}: {r.get('library_strategy','N/A')}, "
              f"{int(r.get('read_count',0)):,} reads")
    time.sleep(0.02)

Workflow 2: Bulk FASTQ Download URL Collection

Search for runs matching criteria and collect download URLs.

# Step 1: Search for Illumina RNA-Seq runs from a specific organism
resp = ena_query("search", params={
    "result": "read_run",
    "query": ('tax_tree(10090) AND library_strategy="RNA-Seq" '
              'AND instrument_platform="ILLUMINA" AND read_count>=10000000'),
    "fields": "run_accession,study_accession,read_count",
    "format": "json",
    "limit": 10,
})
runs = resp.json()
print(f"Found {len(runs)} runs meeting criteria")

# Step 2: Get file reports with download URLs
download_list = []
for run in runs[:5]:
    acc = run["run_accession"]
    resp = ena_query("filereport", params={
        "accession": acc,
        "result": "read_run",
        "fields": "run_accession,fastq_ftp,fastq_bytes,fastq_md5",
        "format": "json",
    })
    for f in resp.json():
        urls = f.get("fastq_ftp", "").split(";")
        md5s = f.get("fastq_md5", "").split(";")
        for url, md5 in zip(urls, md5s):
            if url:
                download_list.append({"url": f"ftp://{url}", "md5": md5, "run": acc})
    time.sleep(0.02)

print(f"\nDownload list: {len(download_list)} files")
for d in download_list[:4]:
    print(f"  {d['run']}: {d['url'].split('/')[-1]}")

Workflow 3: Taxonomic Assembly Exploration

Find organisms, search their assemblies, and check quality statistics.

# Step 1: Resolve organism by exact scientific name (returns a list, take the first match).
# `suggest-for-search` only returns prefix matches and may not include the species you want.
resp = ena_query("scientific-name/Drosophila melanogaster", base=BASE_TAXONOMY)
matches = resp.json()
target_tax = matches[0]["taxId"]
print(f"Selected: {matches[0]['scientificName']} (taxId={target_tax})")

# Step 2: Search assemblies for this organism. `n50` is no longer a valid `assembly`
# field — use `base_count` and `program` for what's available.
resp = ena_query("search", params={
    "result": "assembly",
    "query": f'tax_eq({target_tax}) AND assembly_level="chromosome"',
    "fields": ("assembly_accession,assembly_name,assembly_level,"
               "genome_representation,base_count"),
    "format": "json",
    "limit": 5,
})
assemblies = resp.json()
for a in assemblies:
    size = int(a.get("base_count", 0))
    print(f"  {a['assembly_accession']}: {a.get('assembly_name','N/A')}, "
          f"Size={size/1e6:.1f} Mb")
# GCA_000001215.4: Release 6 plus ISO1 MT, Size=143.7 Mb

Key Parameters

| Endpoint | Parameter | Default | Description | |----------|-----------|---------|-------------| | search | result | (required) | Result type: study, sample, read_run, assembly, etc. | | search | query | (required) | Boolean query string with field operators | | search | fields | all | Comma-separated field names to return | | search | format | tsv | Output format: json, tsv, xml | | search | limit | 100000 | Max results (0 = unlimited streaming) | | search | offset | 0 | Skip first N results (pagination) | | search | sortFields | — | Field(s) to sort by | | filereport | accession | (required) | Study, sample, or run accession | | filereport | result | (required) | read_run, analysis, etc. | | xml/ | accession path | (required) | Any ENA accession (Browser API) | | fasta/ | accession path | (required) | Sequence accession (Browser API) |

Best Practices

1. Use tax_tree() over tax_eq() for organism queries — it includes subspecies and strains automatically
2. Request only needed fields — reduces response size and server load significantly
3. Prefer JSON format for programmatic access; TSV for large bulk exports (lower overhead)
4. Use limit=0 for streaming large result sets directly to file, avoiding memory issues
5. Check fastq_ftp and submitted_ftp — some runs have submitted files but no processed FASTQ
6. Verify downloads with MD5 — file reports include fastq_md5 for integrity checking
7. Anti-pattern: Do not fetch all fields then filter client-side — use query syntax server-side

Common Recipes

Recipe: Assembly Quality Filtering

# Find chromosome-level, full-representation assemblies.
# `n50` is no longer a valid `assembly` field — filter by base_count instead.
resp = ena_query("search", params={
    "result": "assembly",
    "query": ('tax_tree(7742) AND assembly_level="chromosome" '
              'AND genome_representation="full"'),
    "fields": "assembly_accession,scientific_name,base_count,assembly_level",
    "format": "json",
    "limit": 10,
})
for a in resp.json():
    size = int(a.get("base_count", 0))
    if size > 100_000_000:   # >100 Mb (proxy for "well-assembled")
        print(f"{a['assembly_accession']}: {a.get('scientific_name','?')}, Size={size/1e6:.0f} Mb")

Recipe: Cross-Database Linking

# Find UniProt/PDB cross-references for an ENA sequence
resp = ena_query("json/search", base=BASE_XREF, params={
    "accession": "M10051",
})
xrefs = resp.json()
by_source = {}
for x in xrefs:
    src = x.get("Source", "unknown")
    by_source.setdefault(src, []).append(x["Source Primary Accession"])
for src, accs in by_source.items():
    print(f"  {src}: {', '.join(accs[:5])}")
# UniProt: P01308
# PDB: 1A7F, 1AI0, 1BEN

Recipe: Retry Session with Exponential Backoff

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

def ena_session():
    """Create a requests session with retry logic for ENA APIs."""
    session = requests.Session()
    retry = Retry(
        total=3, backoff_factor=1.0,
        status_forcelist=[429, 500, 502, 503, 504],
    )
    session.mount("https://", HTTPAdapter(max_retries=retry))
    return session

session = ena_session()
resp = session.get(f"{BASE_PORTAL}/search", params={
    "result": "study",
    "query": 'tax_tree(9606)',
    "fields": "study_accession",
    "format": "json",
    "limit": 5,
})
print(f"Status: {resp.status_code}, results: {len(resp.json())}")

Troubleshooting

| Problem | Cause | Solution | |---------|-------|----------| | 400 Bad Request on search | Invalid query syntax or unknown field name | Use returnFields endpoint to verify field names; check query operator syntax | | 400 with result param | Invalid result type | Check result types table above; common: read_run not run | | Empty results for known data | Wrong taxonomy operator | Use tax_tree() (includes descendants) not tax_eq() (exact only) | | fastq_ftp field is empty | Submitted files not processed to FASTQ | Check submitted_ftp field instead; some datasets only have BAM/CRAM | | 429 Too Many Requests | Exceeded 50 req/sec rate limit | Add time.sleep(0.02) between requests; use retry session with backoff | | Timeout on large queries | Result set too large for single request | Use limit + offset pagination, or limit=0 with streaming to file | | XML parsing errors | Malformed XML for some records | Use JSON format instead (format=json) when available | | Wrong sequence version | Accession without version suffix | Always use versioned accessions (e.g., M10051.1 not M10051) for Browser API | | CRAM reference not found | MD5 checksum mismatch or non-INSDC reference | Verify MD5; check if reference is from a custom genome (not in registry) |

Bundled Resources

This skill is self-contained. The original entry had a separate references/api_reference.md (490 lines) covering all 6 API endpoints in detail. That content has been fully consolidated inline:

• Portal API (search, count, returnFields, filereport) — Core API Modules 1 and 3, Key Parameters table
• Browser API (XML, FASTA, EMBL retrieval) — Core API Module 2
• Taxonomy REST API (tax-id, scientific-name, suggest-for-search) — Core API Module 4
• Cross-Reference Service (json/search) — Core API Module 5
• CRAM Reference Registry (md5/sha1 lookup) — Core API Module 6
• Rate limiting and error handling — Prerequisites, Troubleshooting, Recipe 3 (retry session)
• Query syntax and result types — Key Concepts section (3 tables)
• Pagination and bulk download — Key Parameters, Best Practices, Workflow 2
• Omitted: detailed EMBL format field-by-field breakdown (rarely needed programmatically); Aspera download command examples (tool-specific, not requests-based)

Related Skills

• bioservices-multi-database — unified Python interface covering ENA via bioservices; prefer for multi-database workflows
• pubmed-database — PubMed literature search via NCBI E-utilities
• pysam-genomic-files — downstream processing of FASTQ/BAM/CRAM files retrieved from ENA
• biopython-molecular-biology — NCBI Entrez access and sequence parsing (GenBank/FASTA)
• ncbi-blast (planned) — BLAST sequence similarity search

References

• ENA Portal API docs: https://www.ebi.ac.uk/ena/portal/api/doc
• ENA Browser API docs: https://www.ebi.ac.uk/ena/browser/api/doc
• ENA Taxonomy REST API: https://www.ebi.ac.uk/ena/taxonomy/rest/
• ENA Cross-Reference Service: https://www.ebi.ac.uk/ena/xref/rest/
• ENA data model guide: https://ena-docs.readthedocs.io/en/latest/submit/general-guide/data-model.html
• INSDC standards: https://www.insdc.org/