ViggyV/ena-database

ENA Database

Overview

The European Nucleotide Archive (ENA) is a comprehensive public repository for nucleotide sequence data and associated metadata. Access and query DNA/RNA sequences, raw reads, genome assemblies, and functional annotations through REST APIs and FTP for genomics and bioinformatics pipelines.

When to Use This Skill

This skill should be used when:

• Retrieving nucleotide sequences or raw sequencing reads by accession
• Searching for samples, studies, or assemblies by metadata criteria
• Downloading FASTQ files or genome assemblies for analysis
• Querying taxonomic information for organisms
• Accessing sequence annotations and functional data
• Integrating ENA data into bioinformatics pipelines
• Performing cross-reference searches to related databases
• Bulk downloading datasets via FTP or Aspera

Core Capabilities

1. Data Types and Structure

ENA organizes data into hierarchical object types:

Studies/Projects - Group related data and control release dates. Studies are the primary unit for citing archived data.

Samples - Represent units of biomaterial from which sequencing libraries were produced. Samples must be registered before submitting most data types.

Raw Reads - Consist of:

• Experiments: Metadata about sequencing methods, library preparation, and instrument details
• Runs: References to data files containing raw sequencing reads from a single sequencing run

Assemblies - Genome, transcriptome, metagenome, or metatranscriptome assemblies at various completion levels.

Sequences - Assembled and annotated sequences stored in the EMBL Nucleotide Sequence Database, including coding/non-coding regions and functional annotations.

Analyses - Results from computational analyses of sequence data.

Taxonomy Records - Taxonomic information including lineage and rank.

2. Programmatic Access

ENA provides multiple REST APIs for data access. Consult references/api_reference.md for detailed endpoint documentation.

Key APIs:

ENA Portal API - Advanced search functionality across all ENA data types

• Documentation: https://www.ebi.ac.uk/ena/portal/api/doc
• Use for complex queries and metadata searches

ENA Browser API - Direct retrieval of records and metadata

• Documentation: https://www.ebi.ac.uk/ena/browser/api/doc
• Use for downloading specific records by accession
• Returns data in XML format

ENA Taxonomy REST API - Query taxonomic information

• Access lineage, rank, and related taxonomic data

ENA Cross Reference Service - Access related records from external databases

• Endpoint: https://www.ebi.ac.uk/ena/xref/rest/

CRAM Reference Registry - Retrieve reference sequences

• Endpoint: https://www.ebi.ac.uk/ena/cram/
• Query by MD5 or SHA1 checksums

Rate Limiting: All APIs have a rate limit of 50 requests per second. Exceeding this returns HTTP 429 (Too Many Requests).

3. Searching and Retrieving Data

Browser-Based Search:

• Free text search across all fields
• Sequence similarity search (BLAST integration)
• Cross-reference search to find related records
• Advanced search with Rulespace query builder

Programmatic Queries:

• Use Portal API for advanced searches at scale
• Filter by data type, date range, taxonomy, or metadata fields
• Download results as tabulated metadata summaries or XML records

Example API Query Pattern:

import requests

# Search for samples from a specific study
base_url = "https://www.ebi.ac.uk/ena/portal/api/search"
params = {
    "result": "sample",
    "query": "study_accession=PRJEB1234",
    "format": "json",
    "limit": 100
}

response = requests.get(base_url, params=params)
samples = response.json()

4. Data Retrieval Formats

Metadata Formats:

• XML (native ENA format)
• JSON (via Portal API)
• TSV/CSV (tabulated summaries)

Sequence Data:

• FASTQ (raw reads)
• BAM/CRAM (aligned reads)
• FASTA (assembled sequences)
• EMBL flat file format (annotated sequences)

Download Methods:

• Direct API download (small files)
• FTP for bulk data transfer
• Aspera for high-speed transfer of large datasets
• enaBrowserTools command-line utility for bulk downloads

5. Common Use Cases

Retrieve raw sequencing reads by accession:

# Download run files using Browser API
accession = "ERR123456"
url = f"https://www.ebi.ac.uk/ena/browser/api/xml/{accession}"

Search for all samples in a study:

# Use Portal API to list samples
study_id = "PRJNA123456"
url = f"https://www.ebi.ac.uk/ena/portal/api/search?result=sample&query=study_accession={study_id}&format=tsv"

Find assemblies for a specific organism:

# Search assemblies by taxonomy
organism = "Escherichia coli"
url = f"https://www.ebi.ac.uk/ena/portal/api/search?result=assembly&query=tax_tree({organism})&format=json"

Get taxonomic lineage:

# Query taxonomy API
taxon_id = "562"  # E. coli
url = f"https://www.ebi.ac.uk/ena/taxonomy/rest/tax-id/{taxon_id}"

6. Integration with Analysis Pipelines

Bulk Download Pattern:

1. Search for accessions matching criteria using Portal API
2. Extract file URLs from search results
3. Download files via FTP or using enaBrowserTools
4. Process downloaded data in pipeline

BLAST Integration: Integrate with EBI's NCBI BLAST service (REST/SOAP API) for sequence similarity searches against ENA sequences.

7. Best Practices

Rate Limiting:

• Implement exponential backoff when receiving HTTP 429 responses
• Batch requests when possible to stay within 50 req/sec limit
• Use bulk download tools for large datasets instead of iterating API calls

Data Citation:

• Always cite using Study/Project accessions when publishing
• Include accession numbers for specific samples, runs, or assemblies used

API Response Handling:

• Check HTTP status codes before processing responses
• Parse XML responses using proper XML libraries (not regex)
• Handle pagination for large result sets

Performance:

• Use FTP/Aspera for downloading large files (>100MB)
• Prefer TSV/JSON formats over XML when only metadata is needed
• Cache taxonomy lookups locally when processing many records

Resources

This skill includes detailed reference documentation for working with ENA:

references/

api_reference.md - Comprehensive API endpoint documentation including:

• Detailed parameters for Portal API and Browser API
• Response format specifications
• Advanced query syntax and operators
• Field names for filtering and searching
• Common API patterns and examples

Load this reference when constructing complex API queries, debugging API responses, or needing specific parameter details.