GPTomics/bio-entrez-fetch

Version Compatibility

Reference examples tested with: BioPython 1.83+, Entrez Direct 21.0+

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show biopython then help(Bio.Entrez.efetch) to check signatures
• CLI: efetch -version then efetch -help to confirm flags

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Entrez Fetch

"Download a record by accession from NCBI" -> EFetch returns the full record content in a chosen format (FASTA, GenBank, XML, MEDLINE, etc.). ESummary returns a lightweight "docsum" object — much faster when only metadata is needed.

The agent's first decision is always: does this workflow need the full record, or just metadata? ESummary is 5-10x cheaper than EFetch for the equivalent record set. For "tell me the organism, length, and definition line for 10,000 accessions", ESummary wins by an order of magnitude.

• Python: Entrez.efetch(db=..., id=..., rettype=..., retmode=...) (BioPython)
• CLI: efetch -db nucleotide -id NM_007294 -format gb (Entrez Direct, NBK179288)
• R: entrez_fetch(db=..., id=..., rettype=...) (rentrez)

Required Setup

from Bio import Entrez, SeqIO
Entrez.email = '[email protected]'
Entrez.api_key = 'optional_api_key'  # raises rate to 10 req/sec

Decision matrix: rettype + retmode per database

The combinations are not orthogonal — each (db, rettype, retmode) triple is enabled or disabled by NCBI server-side. Wrong combinations return either silent empty responses or HTTP 400. The triples below are the safe, current set.

nucleotide / protein

| rettype | retmode | Returns | Use when | |---|---|---|---| | fasta | text | FASTA | Just need sequence + defline | | gb (nuc) / gp (prot) | text | Full flat file | Need annotations, features, references | | gbwithparts | text | GB with CONTIG sequences inlined | Whole-genome shotgun assemblies; default gb returns CONTIG records requiring a chase to resolve | | fasta_cds_na | text | CDS-only nucleotide | Extract coding regions from annotated GB | | fasta_cds_aa | text | CDS-translated AA | Get translated proteins from GB record in one call | | xml (== gb XML) | xml | INSDSeq XML | Programmatic parsing; the schema is unversioned and shifts | | acc | text | Accession.version per line | Just resolve UID -> accession | | seqid | text | Internal seq-id | Rarely needed |

pubmed

| rettype | retmode | Returns | Use when | |---|---|---|---| | abstract | text | Title + authors + abstract | Reading abstracts | | medline | text | MEDLINE flat | Parsing with Bio.Medline | | xml | xml | Full PubMed XML | Programmatic — get MeSH, grants, PMC link | | (omitted) | (omitted) | Defaults to XML | EFetch default for pubmed is XML — pass retmode='xml' explicitly for clarity |

gene

| rettype | retmode | Returns | Use when | |---|---|---|---| | gene_table | text | Tabular per-transcript layout | Exon coordinates | | xml | xml | Full Entrez Gene XML | Everything else — name, synonyms, GeneRIFs, locus |

sra

| rettype | retmode | Returns | Use when | |---|---|---|---| | runinfo | text | CSV of run metadata | Convert SRA UID -> SRR accession + Run metrics | | xml | xml | Full SRA XML hierarchy | Need BioSample/BioProject linkage in one call |

taxonomy

| rettype | retmode | Returns | Use when | |---|---|---|---| | xml | xml (default) | TaxNode XML | Lineage, parent, common name |

gds (GEO)

| rettype | retmode | Returns | Use when | |---|---|---|---| | (default — no rettype) | text | Plaintext SOFT-style summary | Quick metadata; for full series matrix go to FTP |

EFetch for GDS records is intentionally minimal — full GEO downloads go via the FTP mirror or GEOparse. See geo-data skill.

GI deprecation (still bites in 2026)

NCBI stopped issuing new GI numbers for major nucleotide/protein submissions starting 2017. Records submitted after the cutoff have only accession.version identifiers. Many older scripts assume id=<numeric_gi>; passing a modern accession string also works, but mixing the two in one comma-separated id list is the bug.

Rules:

• For modern code, always pass accession.version strings.
• A bare accession without .version resolves to the latest version — fine for exploratory work, dangerous for reproducibility.
• Old id=12345 GI lookups still work for records issued before 2017, but a search returning a UID that looks like a GI may actually be the legacy GI for an old record — assume UID is an opaque identifier.
• EFetch accepts comma-separated IDs of mixed types but the URL has a ~2000 char practical limit; chunk large ID lists into batches.

ESummary vs EFetch triage

| Need | ESummary | EFetch (text) | EFetch (xml) | |---|---|---|---| | Title, organism, length | yes | overkill | overkill | | Authors of a PubMed article | yes | yes | yes | | Full abstract text | no | rettype=abstract | better — structured | | MeSH terms, grant info, PMC ID | no | no | yes | | Sequence | no | rettype=fasta | overkill | | Sequence features (CDS, exons) | no | rettype=gb | yes | | Cross-references (xref) | partial | yes (in GB) | yes | | Bulk metadata for 10K records | best (1 call per ~500) | slow | slow |

ESummary's documented hard limit is 10,000 docsums per call, but the practical sweet spot is ~500 (keeps the URL under length limits when IDs are comma-joined; for >500 use EPost to push IDs server-side first). Per-record payload is much smaller than EFetch. Use ESummary as the default for any metadata-only workflow.

XML schema brittleness

Entrez.read() parses INSDSeq XML, PubmedArticle XML, Gene XML, etc. The schemas are NOT versioned; NCBI adds and renames fields without notice. Real-world consequence: a parser that worked in 2022 may KeyError in 2026 because a nested field moved.

Defensive patterns:

• Use .get(key, default) not [key] for every nested field
• For sequence content, prefer SeqIO.read() over Entrez.read() — the SeqIO parsers are versioned with BioPython
• Pin BioPython version in production code; expect to update the parser when NCBI changes the XML
• For PubMed, Bio.Medline.parse(handle) (against rettype='medline') is more stable than the XML route

Code patterns

Single sequence by accession

Goal: Fetch one nucleotide record as a SeqRecord with features.

Approach: EFetch with rettype='gb', retmode='text'; parse with SeqIO.read().

Reference (BioPython 1.83+):

def fetch_genbank(accession):
    h = Entrez.efetch(db='nucleotide', id=accession, rettype='gb', retmode='text')
    record = SeqIO.read(h, 'genbank'); h.close()
    return record

gb = fetch_genbank('NM_007294.4')
for feat in gb.features:
    if feat.type == 'CDS':
        print(feat.location, feat.qualifiers.get('product', ['?'])[0])

Bulk metadata via ESummary

Goal: Get organism + length + title for 1,000 UIDs without downloading sequences.

Approach: ESummary on a comma-joined ID batch (max 500 per call by convention; supports 10K hard limit).

Reference (BioPython 1.83+):

def bulk_summaries(db, ids, chunk=500):
    out = []
    for i in range(0, len(ids), chunk):
        h = Entrez.esummary(db=db, id=','.join(ids[i:i+chunk]))
        out.extend(Entrez.read(h)); h.close()
        time.sleep(0.1 if Entrez.api_key else 0.34)
    return out

records = bulk_summaries('nucleotide', uid_list)

Extract CDS in one round-trip

Goal: Download the CDS-only translated protein sequences from a GenBank record without manually walking features.

Approach: Use rettype='fasta_cds_aa' — NCBI server-side extracts and translates every CDS in the record.

Reference (BioPython 1.83+):

def cds_proteins(accession):
    h = Entrez.efetch(db='nucleotide', id=accession, rettype='fasta_cds_aa', retmode='text')
    return list(SeqIO.parse(h, 'fasta'))

proteins = cds_proteins('NC_000913.3')  # E. coli K-12 genome
print(f'{len(proteins)} CDS-translated proteins')

Pull PubMed with structured MeSH

Goal: Get MeSH terms and grant information that aren't in the abstract format.

Approach: rettype='xml' and walk the PubmedArticle structure defensively.

Reference (BioPython 1.83+):

def pubmed_full(pmid):
    h = Entrez.efetch(db='pubmed', id=pmid, retmode='xml')
    records = Entrez.read(h); h.close()
    article = records['PubmedArticle'][0]
    citation = article['MedlineCitation']
    mesh = [m['DescriptorName'] for m in citation.get('MeshHeadingList', [])]
    title = citation['Article']['ArticleTitle']
    return {'pmid': pmid, 'title': title, 'mesh': mesh}

History-server fetch (post-ESearch)

Goal: Pull a 50,000-record result set without re-sending UIDs.

Approach: ESearch with usehistory='y'; iterate EFetch with webenv/query_key and retstart. See batch-downloads for the production pattern.

h = Entrez.esearch(db='nucleotide', term='Homo sapiens[ORGN] AND srcdb_refseq[PROP] AND biomol_mrna[PROP]',
                   usehistory='y', retmax=0)
r = Entrez.read(h); h.close()
total = int(r['Count'])

with open('out.fasta', 'w') as out:
    for start in range(0, total, 500):
        h = Entrez.efetch(db='nucleotide', rettype='fasta', retmode='text',
                          retstart=start, retmax=500,
                          webenv=r['WebEnv'], query_key=r['QueryKey'])
        out.write(h.read()); h.close()
        time.sleep(0.1 if Entrez.api_key else 0.34)

SRA UID -> SRR accession + run metrics

Goal: Convert an opaque SRA UID into the SRR run accession plus Bases/Spots metrics, in one EFetch.

Approach: rettype='runinfo' returns a CSV row per run.

def sra_runinfo(uids):
    h = Entrez.efetch(db='sra', id=','.join(uids), rettype='runinfo', retmode='text')
    text = h.read(); h.close()
    lines = text.strip().split('\n')
    header = lines[0].split(',')
    return [dict(zip(header, row.split(','))) for row in lines[1:]]

Taxonomy lineage by TXID

def lineage(txid):
    h = Entrez.efetch(db='taxonomy', id=str(txid), retmode='xml')
    record = Entrez.read(h)[0]; h.close()
    return record['Lineage'], record['ScientificName']

Failure modes

Mixed-format batch silently truncates

• Trigger: Mixing modern accessions and legacy GIs in one comma-separated id=.
• Mechanism: EFetch parses left-to-right; on type-mismatch it may return only the prefix that succeeded.
• Symptom: Batch of 100 returns 47 records with no error.
• Fix: Validate that all IDs in a batch are the same type before sending.

gb returns CONTIG instead of sequence

• Trigger: Fetching a whole-genome shotgun (WGS) assembly with rettype='gb'.
• Mechanism: Default GB output skips the contig sequence for assemblies, returning only the join() statement.
• Symptom: len(record.seq) == 0 despite the record showing a length in metadata.
• Fix: Use rettype='gbwithparts' for assemblies; or for FASTA use rettype='fasta' directly.

XML parse fails on schema drift

• Trigger: Code that was last touched in 2022 hits a new NCBI XML field layout.
• Mechanism: Entrez.read() uses cached DTDs that may not match current responses.
• Symptom: KeyError or ValidationError on a field that "always worked".
• Fix: Run Entrez.read._XMLParser._DTDs.clear() to force re-fetch of DTDs; upgrade BioPython; or switch to text format (rettype='medline' for pubmed) which is more stable.

Silent empty response on bad rettype

• Trigger: Asking for rettype='abstract' on the nucleotide db (only valid for pubmed).
• Mechanism: EFetch returns empty text — no HTTP error.
• Symptom: handle.read() returns '' or whitespace.
• Fix: Check the decision matrix above before sending unfamiliar combinations.

Accession without .version returns wrong record later

• Trigger: Storing 'NM_007294' (no version) for reproducibility years later.
• Mechanism: NCBI returns the current version, which may have changed annotation.
• Symptom: Re-run produces different CDS coordinates than the original analysis.
• Fix: Always pin accession.version (e.g. NM_007294.4); the version is in the GB LOCUS line.

EFetch returns HTML error page

• Trigger: Invalid UID, mid-maintenance window, or expired WebEnv.
• Mechanism: Failure surfaces in HTML body, HTTP status is 200.
• Symptom: SeqIO chokes parsing HTML as GenBank.
• Fix: Sniff the first line of the response — LOCUS for GB, > for FASTA — and raise on mismatch.

Common errors

| Error / symptom | Cause | Solution | |---|---|---| | HTTPError 400 | Invalid id/db/rettype combo | Verify against decision matrix; check accession exists | | HTTPError 429 | Rate limit exceeded | Add time.sleep(0.34) or use API key | | Empty SeqRecord.seq | WGS record with rettype='gb' | Use rettype='gbwithparts' | | ValueError: Sequence too short | Wrong format declared to SeqIO | Match rettype to SeqIO format string | | ExpatError | Got HTML where XML expected | Sniff response start; retry | | KeyError on nested XML field | Schema drift | Use .get() defensively; pin BioPython |

References

• Sayers EW et al. (2024) Database resources of the National Center for Biotechnology Information in 2024. Nucleic Acids Res 52:D33-D43.
• Kans J. (2024) Entrez Direct: E-utilities on the Unix Command Line. NCBI Bookshelf NBK179288.
• NCBI. EFetch help. NBK25499.
• Cock PJ et al. (2009) Biopython: freely available Python tools for computational molecular biology and bioinformatics. Bioinformatics 25:1422-1423.

Related Skills

• entrez-search - Find UIDs before fetching
• entrez-link - Cross-database navigation via ELink
• batch-downloads - History-server pipelines for large fetches
• ncbi-datasets-cli - Modern CLI for genome / gene metadata; often faster than EFetch
• sequence-io/read-sequences - Parse downloaded FASTA/GenBank with SeqIO