provectus/building-genomics-pipelines

Genomics Pipeline Skill

Opinionated Defaults

Use Nextflow (nf-core ecosystem) for all new pipelines. For HPC-only environments without container support, use Snakemake instead.

Alignment

• DNA short reads: Use BWA-MEM2
• RNA-seq: Use STAR (splice-aware)
• Long reads (ONT/PacBio): Use Minimap2

Variant Calling

• Germline SNV/indel: Use DeepVariant (most accurate). GATK HaplotypeCaller if GATK ecosystem is required.
• Somatic variants: Use Mutect2 (tumor/normal pairs)
• Structural variants: Use Manta (Illumina). Add GRIDSS for complex SVs.

Annotation

• Use VEP (Ensembl) as the primary annotator. Use SnpEff as a lightweight alternative for quick checks.

File Formats

• FASTQ: Raw reads (always gzip compressed: .fastq.gz)
• BAM/CRAM: Aligned reads — prefer CRAM over BAM (30-50% smaller)
• VCF/BCF: Variant calls — always use bgzip + tabix for indexing
• BED: Genomic intervals
• GTF/GFF: Gene annotations

Always use indexed files (.bai, .crai, .tbi, .csi). Validate files before downstream analysis.

Pipeline Design Principles

1. Reproducibility: Pin exact tool versions, use containers (Docker/Singularity), track reference genome versions
2. Scalability: Design for scatter-gather parallelization, use chunking for large files, implement checkpointing/resume
3. Quality Control: QC at every stage (raw, aligned, called), generate MultiQC reports, set clear PASS/FAIL thresholds

Common Workflows

Determine which workflow type is needed and consult the corresponding reference:

1. RNA-seq Analysis — see references/rnaseq.md
2. Variant Annotation — see references/annotation.md
3. CNV & Structural Variants — see references/cnv.md

nf-core Pipelines

Use these production-ready pipelines instead of building from scratch:

| Pipeline | Use Case | |----------|----------| | nf-core/sarek | WGS/WES variant calling | | nf-core/rnaseq | RNA-seq analysis | | nf-core/chipseq | ChIP-seq analysis | | nf-core/atacseq | ATAC-seq analysis | | nf-core/viralrecon | Viral genome analysis | | nf-core/mag | Metagenome analysis | | nf-core/methylseq | Bisulfite sequencing |

Example nf-core usage:

nextflow run nf-core/sarek \
    -profile docker \
    --input samplesheet.csv \
    --genome GRCh38 \
    --tools haplotypecaller,snpeff