GPTomics/bio-workflow-management-cwl-workflows

Version Compatibility

Reference examples tested with: cwltool 3.1+, CWL spec v1.2, Docker 24+ (or Singularity/Apptainer 3.8+)

Before using code patterns, verify installed versions match. If versions differ:

• CLI: <tool> --version then <tool> --help to confirm flags

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Note: target cwlVersion: v1.2 for genomics (the record-form secondaryFiles with required: needs it). cwltool is the REFERENCE runner - correct, local, single-node, deliberately slow - NOT the spec; run Toil/Arvados/Calrissian at scale. In v1.2, ExpressionTool outputs are NOT type-checked (a known reference-impl gap, fix planned for v1.3), so do not lean on ExpressionTool for type safety.

CWL Workflows

"Write a portable pipeline that runs the same everywhere and proves what it ran" -> Describe each tool and their wiring as strongly-typed, machine-checkable CWL documents that any conforming runner honors identically, validate the contract before any compute, then execute locally (cwltool) or at scale (Toil/Arvados/Calrissian).

• CLI: cwltool --validate wf.cwl, cwltool wf.cwl job.yml, cwltool --provenance ro/ wf.cwl job.yml
• YAML: class: CommandLineTool (wrap a tool) and class: Workflow (wire tools) at cwlVersion: v1.2

The governing principle: CWL is a SPECIFICATION, not an engine

CWL deliberately splits the workflow DESCRIPTION (a portable, declarative, strongly-typed YAML/JSON document) from its EXECUTION (performed by any conforming runner). Nextflow and Snakemake are engines that happen to have a DSL; WDL is a language with a dominant engine (Cromwell). CWL alone is a community-governed open standard with multiple independent implementations and a formal conformance test suite (Crusoe 2022 Commun ACM 65(6):54-63). The whole value proposition - portability, auditability, vendor-neutrality, provenance, regulatory fit - is a consequence of "spec not engine." The correct frame: the author writes a portable, machine-checkable CONTRACT for a computation that any conforming platform must honor identically. That contract is why CWL is the most VERBOSE and most EXPLICIT of the four systems - the verbosity buys static analyzability and portability.

The most common conceptual error is conflating CWL with cwltool. cwltool is the reference implementation: correct, single-node, and slow by design (it prioritizes spec-conformance over speed). "CWL is slow" or "CWL can't scale" almost always means "I ran cwltool" - at scale the same unchanged document runs on Toil (HPC/cloud batch), Arvados (clinical/enterprise data management), or Calrissian (Kubernetes). Never equate a runner's limits with the spec's.

Adopting CWL buys reproducible workflow LOGIC and nothing else automatically. A clean typed DAG over unpinned tools is NOT reproducible - pin the software environment (containers by digest, not a moving :latest), the reference data and seeds, and control arch/thread/locale leaks separately. The type system closes the wiring-error layer; the author still owns the rest.

Decision: choose CWL vs the other engines

| Dimension | CWL | Nextflow | WDL | Snakemake | |-----------|-----|----------|-----|-----------| | Nature | open SPEC, many engines | engine + Groovy DSL | language + Cromwell/miniwdl | engine + Python DSL | | Typing | strong static (File/Dir/record/enum/optional) | dynamic | moderate | weak (paths/strings) | | Index companions | secondaryFiles = File type property | manual channel/tuple wiring | manual per-input | manual | | Provenance | CWLProv RO out of the box | report/trace/DAG | via platform | report/plugins | | Verbosity | HIGHEST (deliberate) | terse | moderate | moderate | | New-author mindshare | DECLINING | ASCENDANT (nf-core) | strong on Terra | strong in academia | | Sweet spot | multi-platform, provenance-critical, regulated/clinical, standards-driven | fast authoring, curated catalog | Terra/GATK ecosystem | single-lab Python/HPC |

Choose CWL when vendor-neutral portability across multiple platforms, a standardized provenance artifact for a regulated/audited setting, static type-checking before compute, or publishing to a multi-runner registry (Dockstore) drives the decision. Be honest about mindshare: for NEW pipeline authoring CWL has been losing ground to Nextflow/nf-core and WDL/Terra for years (they win on authoring speed and community momentum). CWL retains and deepens its hold where the SPEC is the point - not as a neutral default. Prefer Nextflow/WDL when authoring speed, an existing curated catalog (nf-core), or a specific hosted platform (Terra) dominates.

Decision: which runner

| Engine | For | Runtime model | Notes | |--------|-----|---------------|-------| | cwltool | authoring, --validate, --pack, --provenance, CI, local dev | local, single-node | the conformance yardstick; slow by design - do NOT read its limits as CWL's | | Toil | HPC and cloud batch scale | Python; Slurm/Kubernetes/AWS/Grid Engine | toil-cwl-runner; the workhorse for large CWL | | Arvados | enterprise/clinical data management + execution | cluster + content-addressed storage | arvados-cwl-runner; strong data provenance; regulated settings | | Calrissian | CWL on Kubernetes | one k8s pod per step | needs ReadWriteMany volumes; cloud-native parallelism | | Cromwell | primarily WDL, PARTIAL CWL | JVM | runs a subset only; do not rely on it for full CWL conformance |

Decision: requirements vs hints (get this wrong and jobs silently OOM)

requirements MUST be satisfied - if the runner cannot honor one, execution FAILS loudly (correctly). hints are advisory: the runner MAY honor or ignore them without error. The canonical failure is putting ResourceRequirement: {ramMin: 32000} under hints on a memory-hungry step - a runner is free to ignore a hint and schedule it on a small node, giving intermittent OOM kills. Anything whose absence would corrupt results or crash (the container, minimum RAM/cores, a required input layout, an env var a tool depends on) goes under requirements. Both inherit Workflow -> step -> tool with the INNERMOST declaration winning, so set a default DockerRequirement/ResourceRequirement at workflow scope and override per-step where a tool needs a different image or more RAM. A surprising container at a step is usually a forgotten override.

Decision: scatterMethod

scatter runs a step once per array element; when scattering over MULTIPLE inputs, scatterMethod decides how they combine. This is the most misunderstood CWL construct.

| scatterMethod | Combines by | Jobs | Output shape | Use when | |---------------|-------------|------|--------------|----------| | dotproduct | position-aligned zip | N (arrays MUST be equal length) | flat array of N | paired arrays that correspond 1:1 (R1[i] with R2[i]) | | flat_crossproduct | every combination | N x M | FLAT array of N x M | all pairs, want a flat result list | | nested_crossproduct | every combination | N x M | NESTED array (N of M) | all pairs, preserve the 2-D grid |

dotproduct requires equal-length arrays (unequal is an error, not truncation). The two cross-products run the SAME N x M jobs and differ only in output nesting - choosing flat_ vs nested_ wrong gives the right computations with a mis-shaped output that then mis-wires a downstream File[] step. ScatterFeatureRequirement must be declared regardless of scatterMethod.

CommandLineTool: wrap one tool

A CommandLineTool binds typed inputs to the command line (inputBinding) and captures outputs (outputBinding.glob or stdout). Tool outputs use outputBinding; workflow outputs use outputSource - mixing them is a validation error.

cwlVersion: v1.2
class: CommandLineTool
baseCommand: [bwa, mem]
requirements:
  DockerRequirement:
    dockerPull: quay.io/biocontainers/bwa:0.7.17--he4a0461_11   # digest-pin in production; :latest breaks reproducibility
  ResourceRequirement:            # under requirements: must hold, or the runner fails loudly (a hint could be ignored -> OOM)
    coresMin: 8
    ramMin: 16000                 # MB; bwa-mem index residency + reads, empirical floor for a human genome
inputs:
  reference:
    type: File
    secondaryFiles:               # the .amb/.ann/.bwt/.pac/.sa BWA index files are STAGED next to the fasta
      [.amb, .ann, .bwt, .pac, .sa]
    inputBinding: {position: 2}
  reads_1: {type: File, inputBinding: {position: 3}}
  reads_2: {type: File?, inputBinding: {position: 4}}   # File? is optional: [null, File]
  threads: {type: int, default: 8, inputBinding: {prefix: -t, position: 1}}
stdout: aligned.sam
outputs:
  sam: {type: stdout}

Workflow: wire tools by explicit typed connections

CWL is PULL/goal-oriented and fully declarative: dataflow is wired explicitly through source/outputSource, never through implicit channels (Nextflow) or filename wildcards (Snakemake). --validate type-checks every connection statically, before a byte of data moves.

cwlVersion: v1.2
class: Workflow
requirements:
  ScatterFeatureRequirement: {}
inputs:
  fastq_1: File
  fastq_2: File
  salmon_index: Directory
outputs:
  quant_results:
    type: Directory
    outputSource: salmon/quant_dir     # workflow output wires with outputSource (NOT outputBinding)
steps:
  fastp:
    run: fastp.cwl
    in: {reads_1: fastq_1, reads_2: fastq_2}
    out: [trimmed_1, trimmed_2, json_report]
  salmon:
    run: salmon_quant.cwl
    in: {index: salmon_index, reads_1: fastp/trimmed_1, reads_2: fastp/trimmed_2}   # source: other_step/output
    out: [quant_dir]

secondaryFiles: index and companion files as a type property

Genomics tools demand companion files that must sit next to the primary with a derived name: .bam needs .bai, .fasta needs .fai and .dict, .vcf.gz needs .tbi. CWL makes the companion a PROPERTY of the File type, so every conforming runner is OBLIGATED to co-stage them - the other engines leave this to hand-wiring. This is the single most genomics-relevant CWL feature and the strongest reason to target v1.2.

inputs:
  bam:
    type: File
    secondaryFiles: [.bai]         # append: sample.bam -> sample.bam.bai staged alongside
  reference:
    type: File
    secondaryFiles:                # v1.2 record form makes required-ness explicit
      - {pattern: .fai, required: true}
      - {pattern: ^.dict, required: false}   # caret ^ STRIPS one extension: genome.fasta -> genome.dict (NOT genome.fasta.dict)

The caret ^ removes one extension from the basename before appending; each leading ^ strips one more - the classic .dict gotcha. In v1.0/v1.1 a bare string pattern is required-by-default; only the v1.2 record {pattern, required} form can mark an index optional (a .tbi that may be absent). secondaryFiles are declared on OUTPUT File parameters too, so a produced BAM carries its .bai to the next step; on inputs required defaults true, on outputs the index is collected if present.

Expressions: $(...) is portable, ${...} is a portability debt

Parameter references $(...) are a restricted, safe subset - property/index access into inputs, self, runtime (e.g. $(inputs.reads.nameroot), $(runtime.outdir)). They need NO JavaScript engine, are statically analyzable, and are portable - prefer them. Full JavaScript ${ return ...; } runs only when InlineJavascriptRequirement is present; it is powerful but (a) needs a node engine wherever the workflow runs, (b) is opaque to static analysis, (c) is the leading cause of "works on my runner, breaks on theirs." Reach for ${...} only when a parameter reference genuinely cannot express the need, and treat every InlineJavascriptRequirement as a portability debt taken on knowingly. valueFrom transforms a step input before it reaches the tool (needs StepInputExpressionRequirement for expressions at step scope).

Portability leaks: "run anywhere" is real but DISCIPLINED

The promise is genuine for disciplined CWL, but it leaks - name the leaks. InlineJavascriptRequirement needs a JS engine (node). DockerRequirement carries arch/registry assumptions (an amd64-only image fails on arm64 Apple Silicon/Graviton; a private-registry image needs credentials the target may lack; :latest is not reproducible). Engine-specific extensions under custom namespaces (cwltool:, arv:) are portable only among runners that understand them - they live under hints so a naive runner can ignore them, but a workflow that DEPENDS on their behavior has forfeited portability. Conformance is graded (a coverage percentage per implementation), not binary - "valid CWL" does not guarantee "runs identically on engine X." The disciplined recipe: target v1.2; containerize every tool with a digest-pinned multi-arch image; minimize ${...} in favor of $(...); keep cwltool:/arv: items under hints and never depend on them for correctness; in the input (job) object prefer location: URIs over a local path: (a path: binds the job to one machine's filesystem); validate, then exercise the real target engine before trusting portability.

Provenance: CWL's high ground for regulated/clinical settings

cwltool --provenance ro/ wf.cwl job.yml produces a CWLProv Research Object (a W3C PROV + RO-Crate/BagIt bundle) capturing the workflow, the exact input object, all outputs, intermediates, container images, and the enactment trace (Khan 2019 GigaScience 8(11):giz095). No other mainstream system ships a standardized retrospective-provenance artifact out of the box - this is the concrete reason CWL wins in regulated/audited genomics: hand an auditor one object that answers "exactly what ran, on what inputs, in what containers, producing what outputs." CWL is also a first-class GA4GH citizen: TRS (Tool Registry Service, implemented by Dockstore) standardizes discovery, WES standardizes cross-platform execution - the strongest reproducibility+portability+provenance story of the four systems.

Run commands

cwltool --validate wf.cwl                     # static type-check the contract; no compute
cwltool wf.cwl job.yml                         # run locally (reference runner)
cwltool --singularity wf.cwl job.yml           # swap container runtime (Apptainer is Singularity-compatible)
cwltool --pack wf.cwl > packed.cwl             # bundle a multi-file workflow into one shareable JSON
cwltool --provenance ro/ wf.cwl job.yml        # emit a CWLProv Research Object
toil-cwl-runner --batchSystem slurm wf.cwl job.yml   # same document, at HPC scale

Common Errors

| Symptom | Cause | Fix | |---------|-------|-----| | "CWL is slow / can't scale" | ran cwltool (reference runner, single-node, slow by design) | run the same document on Toil/Arvados/Calrissian; do not equate runner limits with the spec | | "index not found" at runtime (e.g. no .bai/.fai) | secondaryFiles not declared, so the runner staged the primary but not its index | declare secondaryFiles on the indexed File input (and output) | | genome.fasta.dict produced instead of genome.dict | forgot the caret; .dict appends, ^.dict strips one extension | use ^.dict (each ^ strips one extension) | | Intermittent OOM / undersized node on a heavy step | ResourceRequirement placed under hints (advisory, may be ignored) | move anything that MUST hold under requirements | | Validation error on a workflow output | used outputBinding.glob on a workflow output | workflow outputs wire via outputSource: step/out; only tool outputs use outputBinding | | Scatter runs but produces N x M or wrong-shaped output | wrong scatterMethod (dotproduct vs flat_/nested_crossproduct) | pick deliberately: dotproduct=zip, flat_/nested_=all pairs (flat vs nested output) | | Works on cwltool, fails/differs on another engine | ${...} JS or cwltool:/arv: extension the target lacks; graded conformance | prefer $(...); keep engine extensions in hints; test the real target | | "ScatterFeatureRequirement not specified" | used scatter: without the feature flag | add requirements: [ScatterFeatureRequirement] | | Non-reproducible result across time | mutable :latest container tag | pin DockerRequirement by @sha256: digest |

Related Skills

• workflow-management/wdl-workflows - WDL/Cromwell alternative for the Terra/GATK ecosystem
• workflow-management/nextflow-pipelines - reactive-dataflow alternative with the nf-core catalog
• workflow-management/snakemake-workflows - Python/file-pattern alternative for single-lab HPC
• workflows/fastq-to-variants - an end-to-end variant-calling pipeline these engines orchestrate

References

• Crusoe MR, Abeln S, Iosup A, et al. 2022. Methods Included: Standardizing Computational Reuse and Portability with the Common Workflow Language. Commun ACM 65(6):54-63. DOI 10.1145/3486897.
• Amstutz P, Crusoe MR, Tijanic N, et al. 2016. Common Workflow Language, v1.0. figshare. DOI 10.6084/m9.figshare.3115156.v2.
• Khan FZ, Soiland-Reyes S, Sinnott RO, Lonie A, Goble C, Crusoe MR. 2019. Sharing interoperable workflow provenance: a review of best practices and their practical application in CWLProv. GigaScience 8(11):giz095. DOI 10.1093/gigascience/giz095.
• Wratten L, Wilm A, Göke J. 2021. Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers. Nat Methods 18:1161-1168. DOI 10.1038/s41592-021-01254-9.