Reproducibility and open science · Lesson 1 of 3

Reproducible, replicable, and the difference that matters

If a number cannot be regenerated from raw data by a recorded process, it cannot be checked by anybody, including you.

7 min read · Free — no account needed to read

Reproducible means somebody with your data and your code gets your numbers. This is the low bar, and a surprising amount of published work fails it — because the analysis was done in steps nobody recorded, or by hand, or in a file that was later edited.

Replicable means somebody running a new study with new data gets a consistent result. This is the high bar and it is where the "replication crisis" lives.

Failing to replicate is not by itself a scandal — real effects vary by context, and a single non-replication can be the new study's fault. Failing to *reproduce* is different: it means the original result cannot be checked at all.

What produced the crisis was a combination the earlier modules already named: underpowered studies, analysis decided after seeing data, selective reporting of outcomes, and publication bias. Each of those is individually understandable and together they generate a literature of confident findings that do not hold.

The remedies are correspondingly boring and effective: decide the analysis in advance, report everything measured, publish the materials, and make the path from raw data to reported number visible.

See it happen

A reviewer asks for the code behind Table 2 and you go to produce it.

The read that arrives on its own

The analysis is straightforward — you can rebuild the table quickly from the data file.

The same situation, with this idea

You try, and get a slightly different number. The difference turns out to be an exclusion applied by hand in the spreadsheet months ago and never written down. Nothing dishonest happened, and the result is now unreproducible by you — which means it was never checkable by anyone. The fix is upstream: every step that changes the data lives in a script, not in a cell.

After this, you can

keep every data-changing step in something that can be re-run.

The takeaway

If a number cannot be regenerated from raw data by a recorded process, it cannot be checked by anybody, including you.

Easy to take the wrong way

That this only applies to code-heavy work. A qualitative analytic trail — memos, codebook versions, decisions — is the same idea in a different form.

Try this

Try to regenerate one number from your own work, from raw data, following only what is written down.