GPTomics/bio-workflow-management-snakemake-workflows

Version Compatibility

Reference examples tested with: Snakemake 8.0+, Python 3.11+

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures
• CLI: <tool> --version then <tool> --help to confirm flags

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Note: Snakemake 8 (Jan 2024) removed --cluster, --drmaa, and the *RemoteProvider classes from core and moved them to pip-installable EXECUTOR plugins (--executor slurm, package snakemake-executor-plugin-slurm) and STORAGE plugins (storage.s3(...), snakemake-storage-plugin-s3). --use-conda/--use-singularity became --software-deployment-method / --sdm conda apptainer. A Snakemake 7 command line does not run unchanged on 8/9. Run snakemake --version FIRST and branch all execution guidance on 7 vs 8/9.

Snakemake Workflows

"Build a reproducible bioinformatics pipeline with Snakemake" -> Declare each step as a rule that says "a file matching THIS output pattern is produced FROM those inputs", let the engine resolve the DAG backward from requested targets, fan out over samples with wildcards, and pin the software environment so the result reproduces next year.

• Python: Snakefile rule/checkpoint blocks with expand(), wildcards, config, resources, and conda:/container: (Snakemake)

The governing principle: Snakemake is pull/goal-oriented - it builds a STATIC DAG backward from requested target files

Snakemake is a pull, make-like engine. An author does NOT describe a forward flow of data. Each rule is a pattern-matched recipe ("a file that looks like THIS can be produced FROM that"), and the engine takes the requested target files and works BACKWARD, unifying wildcards by string-matching output filename patterns, until it reaches files already on disk. The whole plan - a static DAG - is computed at parse time, before a single job runs (Köster & Rahmann 2012 Bioinformatics 28:2520-2522). Almost every Snakemake bug a biologist hits is a downstream consequence of this one model:

• Rules are wired by OUTPUT-FILE PATTERN, not call order. A missing or typo'd output path silently drops a rule from the DAG - there is no error, the job just never runs. Debugging "why didn't it run" means tracing the backward dependency from the target, not reading top-to-bottom.
• Because the plan is fully known up front, snakemake -n (dry run), --dag, and --report are first-class. This is the payoff of the static model. Nextflow's reactive-dataflow model (processes connected by asynchronous channels, DAG emerges at runtime) has no true dry-run - hold both models in mind and most "why did/didn't it run" questions answer themselves.
• Data-dependent branching is impossible in the base model. If the NUMBER or identity of outputs is unknown until a step runs (split into one file per detected cluster, scatter over however many contigs an assembler emits), the static DAG cannot represent it -> that is exactly what CHECKPOINTS exist for. A biologist who thinks "the pipeline decides at runtime how many chunks" is fighting the paradigm and needs a checkpoint, not a clever run: block.
• A workflow manager buys reproducible LOGIC and nothing else automatically. The DAG being deterministic says nothing about tool versions. A rule with no conda:/container: runs against whatever is on $PATH; "reproducible" is unearned until the software environment is pinned (Grüning et al. 2018 Cell Syst 6:631-635). Pin containers by DIGEST and conda by LOCKFILE - see Software Deployment below.

Decision: Snakemake vs Nextflow (pick by team and infrastructure, not benchmarks)

| Dimension | Snakemake | Nextflow | Best when | |-----------|-----------|----------|-----------| | Model | pull/make, static DAG at parse time | push/reactive dataflow, dynamic DAG | Snakemake: the plan must be visible before committing an allocation | | Language | Python DSL (real Python + pandas in the Snakefile) | Groovy DSL | Snakemake: Python-native lab, file-pattern logic | | Dry run / DAG viz | first-class (-n, --dag, --report) | no true dry-run (-stub/-preview check wiring only) | Snakemake: HPC where a bad plan is expensive | | Data-dependent branching | needs checkpoints (escape hatch) | native (channels) | Nextflow: shape depends on runtime data | | Community pipelines | Workflow Catalog / wrappers (smaller) | nf-core (large, curated) | Nextflow: run a maintained pipeline as-is | | Sweet spot | single-lab reproducible research, HPC, tight Python integration | cloud/production, multi-institution, nf-core stacks | choose by the ecosystem to integrate with |

Reuse before authoring: for a mainstream analysis (RNA-seq, variant calling, ATAC-seq), a curated community pipeline already encodes years of QC and edge cases. Adopting one means RUNNING it (e.g. nf-core/rnaseq via workflow-management/nf-core-pipelines), not authoring Groovy - so a Python-shop preference for Snakemake only decides the authoring case, not whether to build at all. Author from scratch only for a novel method or an unsupported combination of steps.

Decision: rerun triggers - why a job reran, or did not

Since Snakemake 7.8 the default is NOT pure mtime. A rerun fires on a SET of triggers: {mtime, params, input, code, software-env} (Mölder et al. 2021 F1000Research 10:33). This surprises everyone upgrading from old Snakemake.

| Want | Use | |------|-----| | classic Make behavior, minimize surprise reruns | --rerun-triggers mtime | | max reproducibility (default) | all five triggers | | ignore a stable reference's timestamp | ancient("ref.fa") on that input | | mark results current without recompute | --touch | | force specific rules | --forcerun rule / -R | | see what WOULD rerun and why | snakemake -n -R / --list-changes code |

The code trigger catches shell/script/run body changes - reformatting whitespace or editing a comment counts as a code change and reruns the job. On very large DAGs or multi-TB inputs the provenance triggers add a hashing/stat storm; --rerun-triggers mtime skips it.

Decision: run vs script vs shell vs notebook vs wrapper

| Situation | Pick | Why | |-----------|------|-----| | call a CLI tool (samtools, bwa) | shell: | subprocess, conda/container-isolated | | reusable Python/R analysis needing isolation | script: | separate process, snakemake object injected | | standard tool, do not want to write shell | wrapper: (PINNED tag) | maintained, ships its own env | | exploratory, want a re-runnable notebook | notebook: | params injected, --edit-notebook | | trivial in-Snakefile glue only | run: | NEVER heavy work |

run: executes IN the main Snakemake process - it shares the interpreter and GIL, cannot be conda/container-isolated (the conda: directive is disallowed with run:), blocks the scheduler, and an OOM in it takes down the whole workflow. Move anything beyond trivial glue to script:.

Decision: execution backend (Snakemake 8/9)

| Target | Command | |--------|---------| | laptop/workstation | snakemake --cores N --sdm conda | | SLURM (native) | pip install snakemake-executor-plugin-slurm then --executor slurm --jobs N --default-resources | | SLURM (legacy sbatch string) | pip install snakemake-executor-plugin-cluster-generic then --executor cluster-generic --cluster-generic-submit-cmd "sbatch ..." | | S3/GCS I/O | pip install snakemake-storage-plugin-s3 then --default-storage-provider s3 --default-storage-prefix s3://.../ | | thousands of tiny jobs | add group: / --group-components to collapse scheduler overhead |

Porting a v7 --cluster "sbatch --account=X --partition=Y --mem=Z --time=T" command: for the native slurm executor, map those sbatch flags to resource keys (slurm_account, slurm_partition, mem_mb, runtime in minutes) set per-rule in resources: or globally via --default-resources; for a drop-in port keep the old string under cluster-generic (its own plugin, above). --cores = local cores; --jobs/-j = number of concurrent cluster/cloud jobs (in v8 these are separate). Profiles are versioned: the file is config/config.v8+.yaml, every long option becomes a YAML key.

Rules, wildcards, and expand

expand() returns a LIST of strings by combinatorial substitution - it does NOT touch the filesystem. Use it to enumerate targets in rule all.

configfile: 'config/config.yaml'
SAMPLES = config['samples']

rule all:                                              # the requested targets; the DAG is built backward from here
    input:
        expand('results/{sample}.bam', sample=SAMPLES)

rule align:
    input:
        r1 = 'data/{sample}_R1.fq.gz',
        r2 = 'data/{sample}_R2.fq.gz',
        index = 'ref/genome.fa'
    output:
        bam = 'aligned/{sample}.bam'                   # this OUTPUT PATTERN, matched against the target, wires the rule in
    threads: 8
    log:
        'logs/align/{sample}.log'
    shell:
        'bwa mem -t {threads} {input.index} {input.r1} {input.r2} | '
        'samtools sort -@ {threads} -o {output.bam} 2> {log}'

Wildcard constraints - stop silent mis-routing

Wildcards are greedy regex string-unification ({sample} compiles to .+), not typed parameters. An unconstrained wildcard swallows path separators and adjacent tokens: data/{sample}.txt matches data/a/b.txt as sample=a/b, and {a}.{b}.txt on 101.B.normal.txt has no unique parse. The failure is silent mis-routing, not an error. Constrain whenever a value can contain /, ., or _, or a filename has multiple variable tokens.

wildcard_constraints:
    sample = '[^/]+',                                  # no path separators
    chrom = r'\d+|X|Y|MT'                              # only real chromosome tokens

# two rules whose output patterns can both produce a requested file raise AmbiguousRuleException;
# prefer non-overlapping constraints to disambiguate, and fall back to `ruleorder: a > b` only if needed.

Checkpoints - the ONLY data-dependent-DAG mechanism

Goal: produce downstream jobs for a set of files whose number and identity are unknown until a step runs (split a FASTA into one file per detected cluster; scatter over an assembler's contigs).

Approach: declare the producing step a checkpoint with a directory() output; in an input function on the AGGREGATING rule, call checkpoints.<name>.get(**wildcards) FIRST - its exception is what forces the engine to run the checkpoint and RE-EVALUATE the DAG - then glob_wildcards the checkpoint's declared output dir and expand() the real targets.

checkpoint split_fasta:
    input:
        'data/all.fasta'
    output:
        directory('split/{sample}')                    # directory() because the file set is unknowable at parse time
    shell:
        'split_by_cluster.py {input} split/{wildcards.sample}'

def gather_clusters(wildcards):
    # .get() RAISES until the checkpoint has run; that exception drives DAG re-evaluation.
    # Omitting it globs at parse time (empty), so the aggregation silently gets zero inputs - the classic bug.
    ckpt_dir = checkpoints.split_fasta.get(**wildcards).output[0]
    ids = glob_wildcards(f'{ckpt_dir}/{{id}}.fasta').id
    return expand('processed/{sample}/{id}.done', sample=wildcards.sample, id=ids)

rule aggregate:                                         # the input function MUST be attached to the rule that consumes the set
    input:
        gather_clusters
    output:
        'results/{sample}_summary.txt'
    shell:
        'cat {input} > {output}'

Point glob_wildcards at the checkpoint's declared directory() output (a fresh dir) so stale files do not leak into the glob. Prefer one scatter->gather to chains of nested checkpoints.

Resources, escalating retries, and grouping

Resource callables differ by directive: resources is callable(wildcards [, input] [, threads] [, attempt]); threads is callable(wildcards [, input]) only; getting the signature wrong is a top error source. runtime is in MINUTES. attempt starts at 1 and increments per retry - the canonical fix for OOM-killed jobs.

rule call_variants:
    input:
        bam = 'aligned/{sample}.bam'
    output:
        'results/{sample}.vcf'
    threads: 4
    retries: 3                                          # or global --retries 3
    resources:
        mem_mb = lambda wildcards, attempt: 8000 * attempt,   # 8 GB, doubling to 16/24 on OOM restart
        runtime = 240                                   # MINUTES, not seconds; SLURM wall-time
    log:
        'logs/call/{sample}.log'
    shell:
        'variant_caller --threads {threads} {input.bam} > {output} 2> {log}'

For thousands of tiny jobs on HPC, per-job scheduler latency dominates: assign rules a group: (or --group-components rule=N) so they submit as one job. temp('x.bam') deletes an intermediate once all consumers are done (huge for disk); ancient('ref.fa') excludes an input from mtime-based rerun decisions.

Software deployment - the layer that makes it reproducible

A clean DAG over unpinned tools is not reproducible. The engine gives layer 1 (logic); the author must pin the software environment. Declare conda: (a pinnable file) or container: per rule, and activate deployment at run time.

rule fastqc:
    input:
        'data/{sample}.fq.gz'
    output:
        'qc/{sample}_fastqc.html'
    conda:
        'envs/qc.yaml'                                  # a FILE (pinnable), not a bare named env
    container:
        # PIN BY DIGEST, never a mutable tag - :latest or a re-pushed :0.7.17 silently changes the tool and busts the cache
        'docker://quay.io/biocontainers/fastqc@sha256:<digest>'
    shell:
        'fastqc {input} -o qc/'

snakemake --sdm conda --cores 8                         # build per-rule conda envs (was --use-conda in v7)
snakemake --sdm apptainer --cores 8                     # run each rule in its container (was --use-singularity)
snakemake --sdm conda apptainer --cores 8               # containerized conda: build the env INSIDE the pinned image

A bare environment.yml with samtools (no version) resolves differently over time; pin exact builds with a lockfile (conda-lock) for bit-reproducibility. --containerize auto-generates a Dockerfile baking all conda envs into one image. Between-workflow caching (cache: True + SNAKEMAKE_OUTPUT_CACHE) reuses results across workflows but ONLY for deterministic rules - a nondeterministic tool poisons the shared cache with wrong results silently.

Modularization and reuse

include: 'rules/x.smk' is textual inclusion sharing one namespace. For genuine composition of published workflows, use the module system (module other: snakefile: '...'; use rule * from other as other_*), which can import, prefix, and override rules. wrapper: 'v5.0.2/bio/bwa/mem' pulls a maintained, conda-shipping wrapper - PIN the leading version tag; an unpinned wrapper drifts silently.

include: 'rules/qc.smk'
include: 'rules/align.smk'

rule all:
    input:
        rules.qc_all.input,
        rules.call_all.input

Common Errors

| Symptom | Cause | Fix | |---------|-------|-----| | A rule silently never runs | its output pattern does not match any requested target (typo/path); the DAG dropped it | trace backward from rule all; run snakemake -n and inspect the DAG; fix the output path | | Wildcard captures too much / wrong sample | unconstrained greedy .+ swallowed a delimiter or path separator | add wildcard_constraints (e.g. sample='[^/]+') | | Aggregation after a split has zero inputs | forgot checkpoints.X.get(), or globbed at parse time, or input function on the wrong rule | call .get(**wildcards) first, glob the checkpoint's directory() output, attach the function to the consumer | | Everything reruns after a cosmetic edit | the code trigger - editing the shell/script body (even whitespace) counts | expected under provenance triggers; use --rerun-triggers mtime to opt out | | Expected a rerun, got none | old mtime mental model, or output is newer than input | check triggers; --forcerun rule / -R | | --cluster/S3RemoteProvider errors on v8 | removed from core in Snakemake 8 | install the executor/storage plugin; use --executor slurm and storage.s3(...) | | OOM-killed job (exit 137) | static mem_mb too low for the largest sample | retries + mem_mb=lambda wildcards, attempt: base*attempt | | Per-job scheduler meltdown on HPC | thousands of tiny jobs, submission overhead dominates | group: / --group-components to batch into one submission | | MissingOutputException on NFS/Lustre | networked filesystem lags after a job finishes | raise --latency-wait | | Workflow refuses to continue after a killed job ("Incomplete files") | a job died mid-write (SLURM kill, node crash), so its outputs are flagged incomplete | re-run with --rerun-incomplete (--ri); this is Snakemake's crash-resume, distinct from the rerun triggers | | Heavy run: block hangs the workflow | runs in the main process, shares the GIL, no isolation | move to script: | | "Reproducible" but results differ on a colleague's cluster | no --sdm, or a mutable :latest container tag | declare conda:/container:, pin by digest + conda lockfile |

Related Skills

• workflow-management/nextflow-pipelines - The reactive-dataflow alternative; author here when the DAG must emerge from runtime data
• workflow-management/nf-core-pipelines - Run a curated community Nextflow pipeline instead of authoring from scratch
• workflows/rnaseq-to-de - End-to-end RNA-seq-to-differential-expression pipeline this engine can orchestrate
• read-qc/quality-reports - The QC step a pipeline wraps as an early rule
• read-alignment/bwa-alignment - The alignment tool invoked inside a Snakemake shell: rule

References

• Köster J, Rahmann S. 2012. Snakemake - a scalable bioinformatics workflow engine. Bioinformatics 28(19):2520-2522.
• Mölder F, Jablonski KP, Letcher B, et al. 2021. Sustainable data analysis with Snakemake. F1000Research 10:33.
• Grüning B, Chilton J, Köster J, et al. 2018. Practical computational reproducibility in the life sciences. Cell Systems 6(6):631-635.
• Di Tommaso P, Chatzou M, Floden EW, et al. 2017. Nextflow enables reproducible computational workflows. Nature Biotechnology 35(4):316-319.