GPTomics/phasing-imputation/foundations

Version Compatibility

Reference examples tested with: bcftools 1.19+.

Before using code patterns, verify installed versions match. If versions differ:

• CLI: <tool> --version then <tool> --help to confirm flags

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

This skill is the conceptual spine for the category; it sequences the pipeline and routes the per-stage work to the sibling skills rather than running the engines, so the binding versions are the bcftools/Beagle/SHAPEIT5/Minimac4/GLIMPSE stack the siblings share. A reference panel is a dated release on a specific genome build (GRCh37 vs GRCh38); record the panel name, build, and release date with every result, because the same code against a different panel release imputes different sites.

Phasing and Imputation Foundations -- Sequencing the Pipeline and Carrying the Uncertainty

"Phase and impute my genotypes" -> Decide the genotyping strategy, order the stages, and carry a dosage (not a hard call) downstream - because phasing and imputation are one statistical machine whose output is a posterior, and every stage of the pipeline fails silently when misordered or misaligned.

• CLI: confirm the genome build and strand, normalize to biallelic, phase, then impute per chromosome against an ancestry-matched panel, then filter on the imputer's own quality field plus a MAF floor

Scope: the meta-decisions - which genotyping strategy (array vs low-coverage WGS), the stage ordering, why dosages carry the uncertainty, why the panel is the prior, and where each stage silently fails. Panel acquisition/selection -> reference-panels. Statistical phasing engines -> haplotype-phasing. Imputation engines and dosage fields -> genotype-imputation. Quality metrics and filtering -> imputation-qc. Read-backed / molecular single-sample phasing (long reads, Hi-C, 10x) -> long-read-sequencing/haplotype-phasing. The input VCF and its QC -> variant-calling/vcf-basics. The GWAS test on the dosages -> population-genetics/association-testing.

The Single Most Important Modern Insight -- Phasing and Imputation Are One Li-Stephens Copying HMM, and Its Honest Output Is a Dosage, Not a Genotype

There is no separate "phasing algorithm" and "imputation algorithm" at the core: both model a target haplotype as an imperfect mosaic of reference haplotypes, where switching which reference is copied is a recombination (an HMM transition whose rate is set by the genetic map and Ne) and a site where the copy disagrees is a mutation (an emission whose rate shrinks as the panel grows) (Li & Stephens 2003 Genetics 165:2213). Phasing asks which haplotype pair explains the diploid genotypes; imputation aligns those haplotypes onto the panel and reads the copied allele at untyped sites. Three facts drive every decision:

1. The honest output of imputation is a DOSAGE, not a hard call. The dosage is the posterior expected alternate-allele count E[#alt] in [0,2] (= P(het) + 2*P(hom-alt)), and the per-variant INFO/R2/DR2 is the model's self-estimate of how much real information that dosage carries (1 = perfect, 0 = nothing beyond allele frequency). Hard-thresholding the dosage into 0/1/2 and analyzing it as observed throws away the uncertainty imputation exists to quantify - it loses power at exactly the rare, low-R2 variants and, when imputation quality differs across batches, manufactures false associations that fail to replicate. Carry dosage (or the full genotype-probability triple) into a dosage-aware regression.
2. The pipeline is ORDERED and every stage fails silently. QC the array -> align genome build and strand to the panel -> phase -> impute per chromosome -> filter on INFO/R2 plus a MAF floor -> carry dosages to the GWAS. Skipping or misordering (imputing unphased input, a GRCh37/GRCh38 build mismatch, an unflipped A/T or C/G strand, no INFO filter) produces output that runs clean and is wrong. There is no exception or warning; the wrongness is only visible downstream as deflated accuracy or non-replicating hits.
3. The panel is the prior, and its self-graded quality cannot see ancestry mismatch. Imputation can only copy alleles the panel contains, so panel ancestry-match plus size sets the ceiling (Marchini & Howie 2010 Nat Rev Genet 11:499). When the target's ancestry is absent from the panel, the HMM still finds some template and reports a confident-looking R2 about a copy that is systematically wrong - high R2 in an under-represented ancestry is not reassurance. Pre-phasing (Howie 2012 Nat Genet 44:955) commits the study haplotypes once and propagates their switch errors into imputation, so phase well; the upside is that re-imputation against a newer panel needs no re-phasing.

The Copying Model in One Paragraph

A new haplotype is modeled as copying a panel of K reference haplotypes, switching templates along the chromosome. The hidden state at each marker is which template is being copied; the transition probability of switching rises with the genetic distance between markers (in centimorgans, NOT base pairs) and with Ne through the rho ~ 4*Ne recombination scaling; the emission allows a mismatch with a probability ~ theta/(theta+K) that shrinks as K grows (a bigger panel almost always has a near-perfect template, which is the quantitative root of "bigger panels impute better"). The forward-backward algorithm gives, at every site, the posterior over which template is copied; imputation sums that posterior to a dosage and reports its variance ratio as INFO/R2. IMPUTE/IMPUTE2, MaCH/minimac, Beagle, SHAPEIT, Eagle, and GLIMPSE are all this model with different approximations (which haplotypes to condition on, how PBWT makes the K-quadratic match-finding near-linear, EM vs MCMC, genotype-likelihood vs hard-call emission) - the differences are engineering, not different theories.

The Strategy Fork -- Array + Imputation vs Low-Coverage WGS + Imputation

The upstream design decision, made before any engine is chosen, is how to generate the genotypes that will be imputed. It is not primarily an accuracy question; it is an ascertainment question.

| | SNP array + pre-phase + impute | Low-coverage WGS (~0.5-4x) + impute from genotype likelihoods | |---|---|---| | Input to the HMM | hard genotype calls (array error is tiny) | genotype LIKELIHOODS (PL/GL); a hard call at 1x is mostly noise | | Ascertainment | FIXED - only the SNPs the array was designed for, biased to the design population | UNBIASED - whatever is in the genome is observed | | Common variants | excellent and cheap | excellent | | Rare variants | limited by the array scaffold and panel | matches or beats dense arrays (Rubinacci 2021 Nat Genet 53:120) | | Under-represented ancestry | poor (no good array, panel-mismatched) | the main route around array/panel bias | | Tools | Beagle / Minimac4 / IMPUTE5 -> genotype-imputation | GLIMPSE2 (panel) / STITCH (no panel) -> genotype-imputation |

The judgment: common-variant GWAS in a well-paneled ancestry -> array plus imputation is cheap and adequate; rare variants, under-represented ancestry, or a need for unbiased genome-wide ascertainment -> low-coverage WGS plus genotype-likelihood imputation, the direction the field is moving as sequencing costs fall. Low-coverage WGS is not magic: it is only as good as its panel and its likelihoods (bad mapping, contamination, or damage produce garbage GLs that impute garbage).

Tool Taxonomy

| Tool / panel | Citation | Role | Owns / route to | |--------------|----------|------|------------------| | Beagle | Browning 2018 Am J Hum Genet 103:338 | one tool that BOTH phases and imputes; bref3 panel format | haplotype-phasing, genotype-imputation | | SHAPEIT5 | Hofmeister 2023 Nat Genet 55:1243 | scalable phasing incl. rare/singleton; split into phase_common/phase_rare/ligate | haplotype-phasing | | Eagle2 | Loh 2016 Nat Genet 48:1443 | PBWT reference-based phasing; the Michigan-server phaser | haplotype-phasing | | Minimac4 | Das 2016 Nat Genet 48:1284 | imputation engine; msav/m3vcf; the Michigan/TOPMed server engine | genotype-imputation | | IMPUTE5 | Rubinacci 2020 PLoS Genet 16:e1009049 | PBWT-based imputation for large panels | genotype-imputation | | GLIMPSE2 | Rubinacci 2023 Nat Genet 55:1088 | low-coverage WGS imputation from genotype likelihoods | genotype-imputation | | STITCH | Davies 2016 Nat Genet 48:965 | low-coverage imputation with NO reference panel (latent panel via EM) | genotype-imputation | | PBWT | Durbin 2014 Bioinformatics 30:1266 | the data structure making biobank-scale haplotype matching near-linear | (underlies the phasers/imputers) | | 1000G / HRC / TOPMed | 1000G 2015 Nature 526:68; McCarthy 2016 Nat Genet 48:1279; Taliun 2021 Nature 590:290 | the reference panels (the prior) | reference-panels |

Decision Tree by Scenario

| Scenario | Recommended | Why | |----------|-------------|-----| | SNP array, common-variant GWAS, well-paneled ancestry | array -> pre-phase -> impute against a matched panel | mature, cheap, accurate for common variants | | Rare variants, under-represented ancestry, or unbiased ascertainment wanted | low-coverage WGS -> GLIMPSE2 (or STITCH if no panel) -> genotype-imputation | removes array ascertainment bias; wins at rare variants | | Moderate cohort, want one tool for phase and impute | Beagle -> haplotype-phasing, genotype-imputation | does both with the bref3 panel format | | Biobank-scale cohort (100k+), phasing | SHAPEIT5 -> haplotype-phasing | scales; phases rare variants on a common scaffold | | Choosing/preparing the panel for a target ancestry | -> reference-panels | panel ancestry-match is the first-order accuracy lever | | Filtering imputed output before analysis | -> imputation-qc | INFO/R2/DR2 + MAF floor on the imputer's own field | | Single individual, long reads / Hi-C / linked reads | -> long-read-sequencing/haplotype-phasing | read-backed phasing, a physically different signal | | The association test on the dosages | -> population-genetics/association-testing | the GWAS test is downstream | | Establishing target ancestry for panel choice | -> population-genetics/population-structure | PCA, not phasing |

Default when uncertain: for a common-variant GWAS in a well-served ancestry, array plus imputation against a large matched panel; otherwise consider low-coverage WGS plus GLIMPSE2.

The Three Physically Distinct Sources of Phase

Phase information comes from three non-interchangeable physical sources; they resolve different things and the modern answer is to combine, not rank, them. This skill and its phasing sibling own the first; the third is a mode of the same tools; the second is a different category routed out.

• Statistical (population LD). The Li-Stephens copying model. Genome-wide reach from one sample, but it cannot phase a truly private variant (no population shares it, so there is nothing to copy) -> haplotype-phasing.
• Read-backed (molecular). Two heterozygous sites observed on the same physical DNA molecule (long reads, Hi-C, linked reads). The only way to phase a private variant, but reach is the molecule length -> long-read-sequencing/haplotype-phasing.
• Pedigree / trio (Mendelian). Deterministic and genome-wide where parents are informative; the gold standard for validating the others, useless where parents are uninformative -> trio mode of SHAPEIT/Beagle.

Per-Method Failure Modes

Genome build or strand not aligned to the panel

Trigger: imputing GRCh37 array data against a GRCh38 panel, or leaving A/T and C/G (palindromic) SNPs unflipped. Mechanism: positions or alleles silently disagree with the panel, so the HMM copies the wrong templates. Symptom: imputation accuracy near zero over whole regions, with no error and a plausible-looking VCF. Fix: confirm the build, lift over if needed, and align strand/alleles to the panel (bcftools +fixref or the imputation-server check tool) before phasing -> reference-panels.

Hard-calling the dosage

Trigger: thresholding DS into 0/1/2 and analyzing those as observed genotypes. Mechanism: thresholding discards the posterior uncertainty, worst at low-R2 rare variants. Symptom: lost power read as a true null, or false hits from batch-differential imputation quality. Fix: regress on dosage or the genotype-probability triple -> imputation-qc, population-genetics/association-testing.

Trusting R2 in an under-represented ancestry

Trigger: reporting a confident INFO/R2 for a sample whose ancestry the panel barely contains. Mechanism: R2 is the model grading its own posterior; it cannot see that no panel haplotype resembled the target. Symptom: high self-graded quality on systematically wrong copies. Fix: match the panel ancestry, prefer low-coverage WGS for under-served ancestries, and validate with masked/held-out truth stratified by frequency -> reference-panels, imputation-qc.

Switch errors invisible to genotype QC

Trigger: reporting phasing by tool name with no switch-error rate. Mechanism: a switch error changes which haplotype an allele sits on without changing any genotype, so per-site QC is structurally blind to it. Symptom: genotype-concordant data that is phased garbage; a single switch inverts a compound-het (cis vs trans) conclusion. Fix: report a switch-error rate against a trio or read-backed truth set for any phase-dependent claim -> haplotype-phasing.

Expecting imputation to recover a private variant

Trigger: assuming a large panel "recovers" ultra-rare or sample-private variants. Mechanism: a private allele is carried by no panel haplotype, so there is no template to copy - it is un-imputable by construction. Symptom: the variant is absent or imputed at near-zero R2. Fix: sequence (do not impute) private variants; use larger panels only for the merely-rare.

Quantitative Thresholds

| Threshold | Source | Rationale | |-----------|--------|-----------| | Dosage range E[#alt] in [0,2], not {0,1,2} | Li & Stephens 2003 Genetics 165:2213; Marchini & Howie 2010 Nat Rev Genet 11:499 | the dosage is the posterior mean alt-count; analyze on dosage to propagate uncertainty | | INFO/R2 filter at 0.3 (common variants), 0.7-0.8 stricter | Marchini & Howie 2010 Nat Rev Genet 11:499 (framing) | R2<0.3 means the dosage carries little real information; the exact cutoff is a study choice, not a law -> imputation-qc | | Genetic distance in centimorgans (1 cM ~ 1% recombination/meiosis; 1 Morgan = 100 cM) | Li & Stephens 2003 Genetics 165:2213 | LS transitions use genetic, not physical, distance; a wrong-build or flat map silently degrades phasing | | Low-coverage WGS sweet spot ~0.5-4x (often ~1x) with a large panel | Rubinacci 2021 Nat Genet 53:120 | GLIMPSE operating range; below ~0.5x accuracy degrades, above ~4x hard-calling becomes viable | | HRC accurate to MAF ~0.1% | McCarthy 2016 Nat Genet 48:1279 | 64,976 haplotypes; larger panels (TOPMed) push the frequency floor lower | | Pre-phase the cohort once, re-impute against any panel | Howie 2012 Nat Genet 44:955 | imputation collapses from the diploid (K^2) to the haploid (K) problem; phasing is committed, so phase well |

Common Errors

| Error / symptom | Cause | Solution | |-----------------|-------|----------| | Imputation accuracy near zero across a region | genome build or strand mismatch to the panel | confirm build, lift over, align strand before phasing -> reference-panels | | Hits do not replicate across batches | hard-called dosages plus batch-differential imputation quality | analyze dosages; impute batches together -> imputation-qc | | All R2 low for a cohort | panel ancestry mismatch or wrong-build genetic map | use a matched panel; check the map build -> reference-panels | | Rare-variant burden differs across cohorts | a single global R2 cutoff removes more rare variants than common, differently per panel | use frequency-stratified accuracy; match panels -> imputation-qc | | Engine errors or mis-imputes at a site | multiallelic / non-biallelic input | bcftools norm -m -any to split and left-align first -> variant-calling/variant-normalization | | Expected variant missing after imputation | private / panel-absent allele | un-imputable by construction; sequence it |

References

• Li N, Stephens M. 2003. Modeling linkage disequilibrium and identifying recombination hotspots using single-nucleotide polymorphism data. Genetics 165:2213-2233.
• Marchini J, Howie B. 2010. Genotype imputation for genome-wide association studies. Nat Rev Genet 11:499-511.
• Howie B, Fuchsberger C, Stephens M, Marchini J, Abecasis GR. 2012. Fast and accurate genotype imputation in genome-wide association studies through pre-phasing. Nat Genet 44:955-959.
• Browning BL, Zhou Y, Browning SR. 2018. A one-penny imputed genome from next-generation reference panels. Am J Hum Genet 103:338-348.
• Hofmeister RJ, Ribeiro DM, Rubinacci S, Delaneau O. 2023. Accurate rare variant phasing of whole-genome and whole-exome sequencing data in the UK Biobank. Nat Genet 55:1243-1249.
• Loh PR, Danecek P, Palamara PF, et al. 2016. Reference-based phasing using the Haplotype Reference Consortium panel. Nat Genet 48:1443-1448.
• Das S, Forer L, Schonherr S, et al. 2016. Next-generation genotype imputation service and methods. Nat Genet 48:1284-1287.
• Rubinacci S, Ribeiro DM, Hofmeister RJ, Delaneau O. 2021. Efficient phasing and imputation of low-coverage sequencing data using large reference panels. Nat Genet 53:120-126.
• Rubinacci S, Hofmeister RJ, Sousa da Mota B, Delaneau O. 2023. Imputation of low-coverage sequencing data from 150,119 UK Biobank genomes. Nat Genet 55:1088-1090.
• Davies RW, Flint J, Myers S, Mott R. 2016. Rapid genotype imputation from sequence without reference panels. Nat Genet 48:965-969.
• Durbin R. 2014. Efficient haplotype matching and storage using the positional Burrows-Wheeler transform (PBWT). Bioinformatics 30:1266-1272.
• McCarthy S, Das S, Kretzschmar W, et al. 2016. A reference panel of 64,976 haplotypes for genotype imputation. Nat Genet 48:1279-1283.
• Taliun D, Harris DN, Kessler MD, et al. 2021. Sequencing of 53,831 diverse genomes from the NHLBI TOPMed Program. Nature 590:290-299.

Related Skills

• reference-panels - Select and prepare the panel (the prior); ancestry-match, build, strand alignment, server access
• haplotype-phasing - Statistical phasing engines and the switch-error metric
• genotype-imputation - Imputation engines, dosage fields, servers, and low-coverage WGS
• imputation-qc - INFO/R2/DR2 quality metrics, MAF-stratified accuracy, dosage filtering
• long-read-sequencing/haplotype-phasing - Read-backed / molecular single-sample phasing (a physically different signal)
• variant-calling/vcf-basics - Source and basic handling of the input VCF
• variant-calling/variant-normalization - Split multiallelics and left-align before phasing/imputation
• population-genetics/association-testing - GWAS test on the imputed dosages
• population-genetics/population-structure - PCA to establish target ancestry for panel selection
• workflows/gwas-pipeline - End-to-end QC -> phase -> impute -> associate -> visualize