GPTomics/bio-methylation-array-preprocessing

Version Compatibility

Reference examples tested with: sesame 1.20+, minfi 1.48+, ChAMP 2.32+, wateRmelon 2.8+.

Before using code patterns, verify installed versions match. If versions differ:

• R: packageVersion('<pkg>') then ?function_name to verify parameters

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

The ARRAY VERSION and GENOME BUILD are versions that matter as much as the package. EPICv2 REQUIRES sesame (mainstream minfi does not auto-detect it and returns "Unknown"); the manifest/annotation packages are array-version- and genome-build-specific (450K and EPICv1 are hg19; EPICv2 is hg38-native). sesame pulls platform/address data from its own hub, so sesameDataCache() must run once before processing. Record the array (450K/EPIC/EPICv2) and the genome build in any output, the way a sequencing run records its reference.

Array Preprocessing

"Give me a clean methylation matrix from my IDATs" -> Read the raw two-channel intensities, correct the Type I/II design mismatch, dye/background bias, and failed probes, then emit beta (for reporting) and M (for testing) - because an Infinium beta is a two-chemistry fluorescence ratio, not a methylation value, until those corrections are applied.

• R: openSesame(idat_dir, prep='QCDPB', func=getBetas) (sesame) or preprocessFunnorm(rgSet) (minfi)

Scope: IDAT -> corrected, masked, normalized beta/M matrix for one array version. Probe filtering (cross-reactive/SNP/sex), EPICv2 replicate collapse, and sample-identity QC -> array-qc-filtering. Per-CpG testing -> differential-cpg-testing. Region calling -> dmr-detection. Native long-read 5mC -> long-read-sequencing/nanopore-methylation. Bisulfite-sequencing (Bismark/WGBS/RRBS) is the other modality in this category, not this skill.

The Single Most Important Modern Insight -- An Infinium Beta Value Is a Two-Chemistry Fluorescence Ratio, Not a Methylation Measurement

An Infinium array does not measure methylation - it measures the relative fluorescence of a methylated vs unmethylated allele at a fixed, manufacturer-chosen set of CpGs, glued together from two incompatible chemistries. A raw beta becomes a comparable methylation estimate only after preprocessing; preprocessing IS the measurement, not optional cleanup. Three corollaries every misuse violates:

1. The manifest is the experiment. The array interrogates <3% of human CpGs, and a DIFFERENT <3% across 450K (~485K), EPIC (~865K), and EPICv2 (~935K). "Absent" almost always means "not on this array," and a 450K-trained clock or EWAS does not transfer to EPICv2 without intersecting probe sets. Do not start from a supplied beta matrix when IDATs exist - the raw two-channel intensities, control probes, and out-of-band signal that noob/funnorm/pOOBAH need are already gone.
2. Type I and Type II betas disagree by design. Type II probes (one bead, two dyes) have a narrower dynamic range and dye-incorporation bias, so raw Type II betas are compressed toward 0.5 relative to Type I (two beads, one channel). Mixing the two chemistries without a design correction (BMIQ/SWAN/sesame matchDesign) injects a probe-type artifact that can exceed the biological effect. The diagnostic is a per-type beta-density plot showing two mismatched peaks.
3. Raw beta is uninterpretable until detection-masked. Failed probes - low signal, germline/somatic deletions, cross-reactive, SNP-hit - return confident-looking betas that are pure noise. pOOBAH / detection-p masking is what separates a number from a measurement of nothing; pOOBAH additionally catches deletion-driven false-intermediate methylation that negative-control detection-p misses.

Organize the work around DELIVERING a defensible matrix (read -> correct -> mask -> normalize), not around listing minfi functions.

Three Modalities of the Same Biology

DNA methylation is measured three ways, each with different tradeoffs - state which one the data is before choosing tools:

| Modality | Readout | Coverage | Cohort-comparability | This skill | |----------|---------|----------|----------------------|------------| | Infinium array (450K/EPIC/EPICv2) | intensity ratio, no depth | fixed <3% of CpGs, regulatory-enriched | high (shared manifest, no alignment) | YES | | WGBS / RRBS bisulfite | count ratio, depth-gated | genome-wide (WGBS) or enriched (RRBS) | needs alignment + matched genome | -> bismark-alignment, methylkit-analysis | | Long-read native (ONT/PacBio) | per-molecule modification calls | genome-wide, phased | growing | -> long-read-sequencing/nanopore-methylation |

Arrays dominate human epigenetic epidemiology (essentially every published clock and large EWAS is array-based) because cost is a fraction of WGBS and the fixed manifest makes cohorts directly comparable.

Object Models (do not start from a beta matrix)

The raw output per sample is a pair of binary IDATs (_Grn.idat, _Red.idat); background, dye, and detection-p correction REQUIRE these plus the control probes.

• minfi: RGChannelSet (raw red/green) -> a preprocess* step -> MethylSet (M/U intensities) -> RatioSet (beta/M) -> GenomicRatioSet (genome-mapped). read.metharray.exp() reads IDATs; getBeta(), getM(), getCN() extract values.
• sesame: a SigDF (one signal data.frame per sample). readIDATpair() reads one sample; openSesame() drives the whole pipeline across a directory and returns a betas matrix directly.

Tool Taxonomy

| Tool | Citation | Mechanism / role | When | |------|----------|------------------|------| | sesame | Zhou 2018 Nucleic Acids Res 46:e123 | SigDF; openSesame QCDPB; pOOBAH OOB masking; EPICv2-native | EPICv2; best detection masking; the modern default | | minfi | Aryee 2014 Bioinformatics 30:1363 | RGChannelSet->GenomicRatioSet; noob/funnorm/quantile/SWAN | 450K/EPICv1; large downstream ecosystem (DMRcate, conumee) | | ChAMP | Tian 2017 Bioinformatics 33:3982 | end-to-end pipeline; BMIQ default | one-call newcomer pipeline on 450K/EPICv1 | | wateRmelon | Pidsley 2013 BMC Genomics 14:293 | dasen/nasen + metric-driven normalization eval | dasen default; normalization benchmarking |

Normalization Decision Tree by Scenario

Separate the two correction layers that get conflated: (a) background + dye bias (within-sample): noob, sesame dyeBias, dasen background step; (b) Type I/II design correction + between-array harmonization: SWAN, BMIQ, quantile, funnorm, dasen quantile step. A complete pipeline does both.

| Scenario | Recommended | Why | |----------|-------------|-----| | EPICv2 (any design) | sesame openSesame(prep='QCDPB') | EPICv2-native; pOOBAH; minfi mis-handles duplicate IDs | | Cancer / cross-tissue (global differences expected) | minfi preprocessFunnorm (noob + control-PCs) | preserves real global shifts; quantile would erase them | | Subtle blood EWAS (no global difference expected) | preprocessQuantile or wateRmelon dasen | marginal distributions assumed equal; safe to harmonize | | Strong Type I/II design correction wanted | BMIQ (Teschendorff 2013) or SWAN (Maksimovic 2012) | dilate Type II onto the Type I distribution; pair with a between-array step | | Single-sample / clinical / streaming | ssNoob or per-IDAT openSesame | reproducible without re-normalizing the cohort | | Probe/sample filtering, EPICv2 collapse, identity | -> array-qc-filtering | this skill stops at the corrected matrix | | Per-CpG testing on the matrix | -> differential-cpg-testing | test on M-values; report delta-beta |

There is no universally best normalization (Pidsley 2013 favored dasen; Fortin 2014 favored funnorm for global-difference studies; Welsh 2023 ranked a sesame/pOOBAH pipeline best and quantile worst on EPIC replicate-concordance). Key the choice on array version + whether global differences are expected + single-sample vs cohort, and verify against current benchmarks rather than hard-coding one method.

Signal -> Beta -> M

• Beta: beta = M / (M + U + alpha), M = methylated-allele intensity, U = unmethylated, alpha = 100 (minfi default) stabilizes the ratio when both intensities are near zero. beta in [0,1] is interpretable but HETEROSCEDASTIC (variance collapses near 0 and 1), violating the constant-variance assumption of linear models.
• M-value: M = log2((M_int + alpha) / (U_int + alpha)), the logit of beta. Approximately homoscedastic; the correct scale for limma/t-tests (Du 2010 BMC Bioinformatics 11:587). Rule: test on M-values, report delta-beta for effect size - the same rule as bisulfite sequencing.

Process IDATs with sesame (the EPICv2-safe default)

Goal: Produce a corrected, detection-masked betas matrix from a directory of IDAT pairs without manually juggling manifest packages.

Approach: Cache the sesame data hub once, then run openSesame with the default QCDPB prep (qualityMask, inferInfiniumIChannel, dyeBiasNL, pOOBAH, noob, in that order), which auto-detects the platform and returns betas; pOOBAH writes NA into failed probes in place.

library(sesame)
sesameDataCache()                          # once per machine; pulls platform/address data
betas <- openSesame('idat_dir', prep = 'QCDPB', func = getBetas)
# prep codes: Q qualityMask  C inferInfiniumIChannel  D dyeBiasNL  P pOOBAH  B noob
# pOOBAH masks (sets NA) probes whose out-of-band signal is indistinguishable from background,
# catching deletion-driven false-intermediate methylation that negative-control detection-p misses
mvals <- log2(betas / (1 - betas))         # M-values for statistical testing (logit of beta)

For EPICv2, openSesame detects the platform automatically; the replicate-probe collapse (betasCollapseToPfx) belongs to the next stage and is documented in array-qc-filtering.

Process IDATs with minfi (450K / EPICv1)

Goal: Build a normalized GenomicRatioSet and extract beta and M, choosing the normalization by whether global methylation differences are expected.

Approach: Read IDATs into an RGChannelSet, compute a detection-p mask before normalizing, then apply funnorm (global differences) or quantile (no global differences); extract beta and M with the offset-100 defaults.

library(minfi)
rgSet <- read.metharray.exp(base = 'idat_dir')
detP <- detectionP(rgSet)                   # neg-control-based; pre-normalization probe-failure map

grSet <- preprocessFunnorm(rgSet, nPCs = 2) # noob first, then 2 control-probe PCs; preserves global shifts
# preprocessQuantile(rgSet) instead when NO global difference is expected (subtle blood EWAS)

beta <- getBeta(grSet)                       # GenomicRatioSet holds precomputed betas (offset applied upstream)
mval <- getM(grSet)                          # log2(beta/(1-beta)) on the ratio set
beta[detP[rownames(beta), colnames(beta)] > 0.01] <- NA   # mask probes failing detection-p (0.01)

EPICv2 is NOT handled by mainstream minfi (it returns "Unknown" and duplicates probe IDs); use sesame for EPICv2.

Per-Method Failure Modes

Starting from a supplied beta matrix

Trigger: processing begins from a .csv/.RData beta matrix instead of IDATs. Mechanism: a beta matrix has discarded the raw two-channel intensities, control probes, and out-of-band signal. Symptom: noob/funnorm/pOOBAH/dye correction cannot run; detection-p cannot be recomputed. Fix: obtain the raw IDAT pairs; treat a beta matrix as a last resort and document that preprocessing could not be applied.

Type I/II mismatch left in the data

Trigger: testing on raw or only background-corrected betas. Mechanism: Type II betas are compressed toward 0.5 relative to Type I. Symptom: "differential" probes that are design artifacts; a two-peak per-type beta density. Fix: apply BMIQ/SWAN or sesame matchDesign (or use openSesame, which corrects channel/dye) before testing; confirm the two per-type peaks align.

minfi on EPICv2

Trigger: read.metharray.exp on EPICv2 IDATs with mainstream minfi. Mechanism: EPICv2 is not auto-detected; 5,483 loci carry duplicate IDs. Symptom: array reads as "Unknown"; getBeta() returns repeated rownames so match()-based merges silently misbehave. Fix: use sesame (EPICv2-native), or install a third-party EPICv2 manifest/anno, tag the annotation manually, and collapse replicates in array-qc-filtering.

Quantile-normalizing a global-difference contrast

Trigger: preprocessQuantile on cancer vs normal or cross-tissue data. Mechanism: between-array quantile assumes equal marginal beta distributions. Symptom: real global hypomethylation flattened away. Fix: use funnorm (control-probe PCs preserve global shifts); reserve quantile/dasen for subtle no-global-difference designs.

Testing on beta instead of M

Trigger: limma/t-tests run directly on beta. Mechanism: beta is heteroscedastic (variance collapses near 0 and 1). Symptom: miscalibrated variance; inflated or deflated p-values at extreme methylation. Fix: test on M-values, report delta-beta for effect size.

Quantitative Thresholds

| Threshold | Source | Rationale | |-----------|--------|-----------| | beta offset alpha = 100 | Aryee 2014; minfi default | stabilizes the ratio when M and U are both near zero | | detection-p > 0.01 = failed | minfi convention | signal indistinguishable from background; beta is noise | | pOOBAH default p ~ 0.05 | Zhou 2018 | OOB-based mask; also catches deletion-driven false intermediate methylation | | funnorm nPCs = 2 | Fortin 2014 | first 2 control-probe PCs absorb technical variation without erasing biology | | Test on M-values, report delta-beta | Du 2010 | M is homoscedastic for modeling; beta is interpretable for effect size | | sesame prep QCDPB (ordered) | Zhou 2018 | Q quality, C channel, D dye, P pOOBAH, B noob - the validated default order |

Common Errors

| Error / symptom | Cause | Solution | |-----------------|-------|----------| | Array reads as "Unknown" | EPICv2 in mainstream minfi | use sesame; or third-party manifest + manual annotation tag | | getBeta() has repeated rownames | EPICv2 duplicate probe IDs | collapse replicates (array-qc-filtering); do not merge by ID first | | Two-peak beta density per probe type | Type I/II design bias uncorrected | BMIQ/SWAN/openSesame before testing | | Global signal vanished after normalization | quantile applied to a global-difference study | use funnorm | | sesameDataCache / platform-not-found | hub not cached | run sesameDataCache() once before processing | | Coordinates misalign merging EPICv2 with 450K | EPICv2 is hg38, 450K/EPICv1 hg19 | track build per array; liftover before merging (array-qc-filtering) |

References

• Aryee MJ, Jaffe AE, Corrada-Bravo H, et al. 2014. Minfi: a flexible and comprehensive Bioconductor package for the analysis of Infinium DNA methylation microarrays. Bioinformatics 30:1363-1369.
• Zhou W, Triche TJ Jr, Laird PW, Shen H. 2018. SeSAMe: reducing artifactual detection of DNA methylation by Infinium BeadChips in genomic deletions. Nucleic Acids Res 46:e123.
• Triche TJ Jr, Weisenberger DJ, Van Den Berg D, Laird PW, Siegmund KD. 2013. Low-level processing of Illumina Infinium DNA methylation BeadArrays. Nucleic Acids Res 41:e90.
• Fortin JP, Labbe A, Lemire M, et al. 2014. Functional normalization of 450k methylation array data improves replication in large cancer studies. Genome Biol 15:503.
• Maksimovic J, Gordon L, Oshlack A. 2012. SWAN: subset-quantile within array normalization for Illumina Infinium HumanMethylation450 BeadChips. Genome Biol 13:R44.
• Teschendorff AE, Marabita F, Lechner M, et al. 2013. A beta-mixture quantile normalization method for correcting probe design bias in Illumina Infinium 450k DNA methylation data. Bioinformatics 29:189-196.
• Pidsley R, Wong CCY, Volta M, Lunnon K, Mill J, Schalkwyk LC. 2013. A data-driven approach to preprocessing Illumina 450K methylation array data. BMC Genomics 14:293.
• Du P, Zhang X, Huang CC, et al. 2010. Comparison of Beta-value and M-value methods for quantifying methylation levels by microarray analysis. BMC Bioinformatics 11:587.
• Tian Y, Morris TJ, Webster AP, et al. 2017. ChAMP: updated methylation analysis pipeline for Illumina BeadChips. Bioinformatics 33:3982-3984.
• Kaur D, Lee SM, Goldberg D, et al. 2023. Comprehensive evaluation of the Infinium human MethylationEPIC v2 BeadChip. Epigenetics Commun 3:6.

Related Skills

• array-qc-filtering - Probe and sample QC/filtering downstream of preprocessing
• differential-cpg-testing - Per-CpG testing on the resulting beta/M matrix
• dmr-detection - DMRcate array-mode region calling
• cell-type-deconvolution - Consumes the clean beta matrix
• epigenetic-clocks - Consumes the clean beta matrix
• ewas-design - Study design, batch, and inference layer
• long-read-sequencing/nanopore-methylation - Native long-read methylation (different platform)
• workflows/methylation-pipeline - End-to-end pipeline