Causal inference · Lesson 2 of 3

Randomisation, and what to do when you cannot

When you cannot randomise, look for something in the world that assigned the treatment for reasons unrelated to the outcome.

10 min read · Free — no account needed to read

Randomisation is the strongest tool because it makes the groups comparable *on everything*, including what you never thought to measure. That is its real power — statistical control handles confounders you knew about and measured; randomising handles the ones you did not.

It fails in specific ways worth knowing: it is often impossible or unethical; small trials can be unbalanced by chance; people drop out non-randomly; and knowing you are in a trial changes behaviour.

When you cannot randomise, several designs create a credible comparison.

Difference-in-differences compares the change over time in a group that got the treatment with the change in one that did not. It removes anything fixed about the groups. Its assumption is that without the treatment both would have moved in parallel — which you support by showing they did so beforehand.

Regression discontinuity exploits a cutoff — a score of 60 gets the scholarship, 59 does not. People just either side are near-identical except for the treatment, so the comparison is strong near the threshold and says nothing about people far from it.

Instrumental variables use something that affects the treatment but has no other route to the outcome — distance to a clinic, a lottery, a rule change. The strength is that a good instrument mimics randomisation; the difficulty is that the "no other route" assumption cannot be verified, only argued for.

Natural experiments use external events — a policy that arrived in some regions first, a reorganisation, a weather shock — that assigned treatment in a way unrelated to the outcome.

Matching pairs treated and untreated cases with similar measured characteristics. It is genuinely useful and is worth being clear about: it handles only what you measured, so it does not do what randomisation does.

See it happen

A company rolled out a new onboarding process, region by region, over eighteen months.

The read that arrives on its own

Compare retention before and after the rollout to see whether it worked.

The same situation, with this idea

You notice the staggered rollout is a gift. Regions that had not yet received it are a comparison group for those that had, over the same period — so anything affecting the whole company is differenced out. You check the assumption first: did the regions track each other on retention before any rollout? If yes, you have a credible design from data you already had.

After this, you can

look for a staggered rollout, a cutoff or a rule change before settling for before-and-after.

The takeaway

When you cannot randomise, look for something in the world that assigned the treatment for reasons unrelated to the outcome.

Easy to take the wrong way

That these designs are statistical techniques. Each is an argument with an assumption attached, and the assumption is what a reader should be able to inspect.

Try this

For your causal question, look for a cutoff, a staggered rollout or a rule change in the setting. Write what its assumption would be.