Module 17 · 3 lessons

Reproducibility and open science

Work somebody else could check — starting with you, in six months. Several fields discovered that a large share of well-known findings did not replicate.

Free — no account needed to read

Work somebody else could check — starting with you, in six months.

Why it matters

Several fields discovered that a large share of well-known findings did not replicate. The practices that came out of that are now expected, and most of them help you before they help anybody else.

What it teaches

• Reproducible versus replicable
• Pre-registration and registered reports
• Documentation, data dictionaries and provenance
• Version control for research
• Open data, open methods, open access, FAIR
• What to do when data cannot be shared

When you reach for it

• You are about to start collecting or analysing
• A reviewer or supervisor asks for your data and code
• You return to your own analysis after three months

The common misreading

That this is about satisfying journals. The first beneficiary is you in six months, when you cannot remember which file produced the number in the table.

What this module will not tell you

This module will not tell you what your institution, funder or ethics approval permits you to share. Data sharing has real legal and consent constraints, and those are checked case by case.

Think first

A colleague leaves. Their results are in the paper, the data is on a shared drive in four versions, and nobody knows which one the figures came from. What was missing?

Think of an analysis you did more than three months ago. Could you produce the exact same number today? Write down what you would need that you do not have.

The lessons

• Reproducible, replicable, and the difference that matters — If a number cannot be regenerated from raw data by a recorded process, it cannot be checked by anybody, including you. (7 min)
• Pre-registration and registered reports — Pre-registration separates predicted from discovered. Both are publishable; only one carries the weight the statistics assume. (8 min)
• Documentation, versioning and openness — Keep raw data untouched, do everything in recorded steps, and share the layers you legitimately can. (8 min)

Practise it: Make one analysis reproducible

One analysis, taken from wherever it is now to something re-runnable.

1. Put the raw data somewhere it is never edited.
2. Write down every step between raw data and your reported number, including anything you did by hand.
3. Move each of those steps into a script, a documented procedure, or a memo.
4. Write the data dictionary for the variables involved.
5. Delete your output and regenerate it from raw data following only what is written.
6. Record what was missing when you tried.

What a considered answer tends to contain (not a mark scheme): A considered answer usually finds at least one undocumented step. Finding none on the first attempt is rare enough to be worth double-checking.

Use it on real work: Set the project up so this holds

Put the practices in place for a real project, at the start rather than the end.

1. Write and date the analysis plan before looking at data.
2. Record where raw data lives and that it is not edited.
3. Start a decision log for anything you change and why.
4. Write the data dictionary.
5. Note what you will be able to share and what you will not, and why.

Reflect

Which step in your own process exists only in your memory?