14 Workflows and Reproducibility: Concepts

Concepts

14.1 The problem

An analysis is an argument. The figure is the conclusion, and the pipeline is the reasoning that got there. Reproducing it does not mean running the same commands again — it means being able to re-derive the conclusion, and knowing what would have had to be different for it to change.

Those are not the same goal, and confusing them is why so much effort spent on reproducibility buys so little. A pipeline that reruns perfectly on the machine that wrote it has demonstrated almost nothing. A pipeline that a stranger can run, on different hardware, two years later, and get an answer they can compare to yours, has demonstrated the thing that matters.

The distinction worth holding onto is between repeating and reproducing. Repeating is getting the same output from the same input on the same setup — a property of your machine. Reproducing is getting a scientifically equivalent answer from the same input somewhere else — a property of your method. Only the second is evidence, and the two require completely different work.

14.2 Three things a result depends on

Every computational result rests on three legs, and they fail in different ways and on different timescales.

The code is the leg everyone thinks of, and the one most nearly solved: version control has been standard practice for long enough that “which commit” is usually answerable. It is also the smallest of the three problems.

The environment is every piece of software the code calls, every library those call, and the versions of all of them. This is the leg that rots. Nothing about it is recorded by default, it changes underneath you without any action on your part, and by the time you notice, the state you want to describe is gone.

The data is the inputs, the reference files, and the annotations — and it is the leg most often left implicit. A reference genome build is a dependency exactly as much as a library version is, and an annotation release changes answers in ways that are easy to attribute to biology.

Which leg is actually load-bearing

If you can only pin one, pin the environment, because it is the one that changes without you and the one you cannot reconstruct after the fact. Code is usually recoverable from history, and data is usually recoverable from an archive. A software stack from three years ago is frequently gone for good.

14.3 A pipeline is a dependency graph

The step from “a script that runs things in order” to “a pipeline” is the step from describing how to describing what depends on what.

A script is imperative: do this, then that. It encodes the order you happened to think of, and the order is only correct by accident. A pipeline is a declaration of dependencies — this output requires those inputs — from which an order can be derived. Once the dependencies are explicit rather than implied by line numbers, several things become possible that were not before.

Independent work can run at the same time, because the graph says which steps do not depend on each other. Work that is already done can be skipped, because the graph says which outputs are current. And a failure part-way through becomes recoverable, because the graph says what still needs doing.

That last one is why the abstraction earns its cost. At the scale this book deals with, jobs fail: a node dies, a filesystem hiccups, a queue evicts you at hour nine of ten. A pipeline that must restart from the beginning after any failure is not merely inconvenient, it is a pipeline that will not finish.

Resumability requires idempotence: running a step twice must be indistinguishable from running it once. That is a real constraint on how steps are written. A step that appends to a file, mutates its input, or writes directly to its final output path breaks it — because a partial output from a crashed run is indistinguishable from a complete one. Writing to a temporary location and moving into place at the end is the whole discipline, and it is what makes “skip work that is already done” safe rather than dangerous.

14.4 Determinism is not reproducibility

It is tempting to want bit-identical outputs, and to treat any difference as a failure. That target is usually unattainable and rarely the right one.

Bit-identical results are hard because so much of what we run is legitimately non-deterministic. Multithreaded code sums floating-point values in whatever order the threads finish, and floating-point addition is not associative, so the same computation on the same data can differ in the last digits by thread count alone. Tools use random seeds. Hash ordering varies. Filesystems return directory entries in whatever order they like, and a tool that globs inputs inherits that order.

None of these mean the analysis is wrong. They mean bit-identity is the wrong acceptance test.

The useful target is that conclusions are stable under the variation you cannot eliminate. That reframes the question productively: instead of “do I get the same bytes”, you ask “how much would the answer have to move before I changed my mind, and is the run-to-run variation anywhere near that”. A result that depends on the thread count was never a result.

The exception

Regulated and clinical settings often do require bit-identity, and for a defensible reason: it makes any change detectable without having to judge whether the change mattered. If you are in that setting, single-threading the non-deterministic steps and fixing every seed is a legitimate cost. Just be clear that you are buying auditability, not correctness.

14.5 Portability is a spectrum

“It runs on my machine” and “it runs anywhere” are the ends of a scale, not a binary, and the intermediate points are where the real engineering choices are.

A written description of the software — a list of names and versions — records intent. It is better than nothing and it is not a specification, because the same names resolve to different things at different times and on different platforms.

A resolved environment records a solution: the exact set of packages that satisfied those constraints, pinned. This is a large step up, because it can be recreated rather than re-solved. It still assumes the packages remain available and that the host provides a compatible system underneath.

An image records the filesystem: the software, its dependencies, and the surrounding system libraries, as bytes. This removes the host from the equation almost entirely, at the cost of a much larger artifact and the need to be able to run images at all — which, on shared academic hardware, is a question about permissions rather than technology.

A standard records the workflow itself in a form more than one engine can execute (Crusoe et al. 2022). This is the least common choice and the one aimed furthest into the future: it accepts more friction now to avoid the workflow dying with the tool that ran it.

Each level pins strictly more than the one above and costs strictly more to produce. There is no correct point on the scale, only a defensible one for how long the result has to survive.

14.6 Provenance is part of the result

A number without its derivation is not a finding, it is a rumour. What has to travel alongside a result is: which version of the pipeline produced it, which environment, which inputs — identified by content rather than by filename — and which parameters.

The reason to have a workflow engine record this automatically rather than writing it down is not diligence, it is that manual provenance is always recorded at the moment of success and never at the moment of the eleven failed attempts before it. Automatic provenance describes what actually happened. Manual provenance describes what you remember happening.

14.7 What to take forward

Repeating a computation and reproducing a result are different goals; only the second is evidence.
Code, environment and data are all dependencies. The environment is the one that decays without you and cannot be reconstructed afterwards.
Writing a pipeline as a dependency graph rather than a sequence is what buys parallelism, caching and — most importantly — resumption after failure.
Resumption is only safe if steps are idempotent, which constrains how each step writes its output.
Bit-identity is usually the wrong target. Stability of the conclusion under unavoidable variation is the right one, except where auditability is the actual requirement.
Portability is a scale from “a list of versions” to “an executable standard”, and the right point on it depends on how long the result must outlive its tooling.

Section 15.1 names what to run.