Module 05 · 3 lessons

Thinking with evidence

Between the evidence and the conclusion there is always an argument. This is how to see it.

Free — no account needed to read

Between the evidence and the conclusion there is always an argument. This is how to see it.

Why it matters

Collecting good sources and still reaching a wrong conclusion is entirely possible, and it is the failure that is hardest to notice from the inside, because every individual piece checks out.

What it teaches

• What a correlation can and cannot support
• Confounders, and how to look for them
• Generating alternative explanations before choosing one
• Base rates, and why they are so easy to ignore
• Selection effects and survivorship
• Strength of evidence: what would change your mind

When you reach for it

• A finding fits your expectations so well that you are about to accept it quickly
• Two sources disagree and you need to work out why rather than pick one
• You are about to write the word "therefore"

The common misreading

That this is about being sceptical of everything. Indiscriminate doubt is as useless as indiscriminate belief — the skill is knowing which specific thing would have to be true for a conclusion to hold.

What this module will not tell you

This module will not tell you whether a particular finding is true. It tells you what would have to be ruled out, which is usually work that remains to be done rather than a verdict you can reach from the armchair.

Think first

Cities with more bookshops have higher literacy rates. A campaign proposes funding bookshops to raise literacy. What is being assumed?

Think of a finding you were pleased to read because it confirmed something you believed. Write down what would have had to be different in that study for you to have questioned it.

The lessons

• What an association actually supports — An association is a fact about the data. Which explanation it supports is a separate argument that has to be made. (9 min)
• Selection, survivorship, and who is missing — Ask who is missing from the data. Absent cases are the most reliably overlooked evidence there is. (8 min)
• Strength of evidence, and what would change your mind — Ask how much this should move you from where you were, and name what would move you further. (8 min)

Practise it: Take one finding apart

Choose a real finding — from your project, your field, or something you read this week — and work through it.

1. State the finding in one sentence, separating what was observed from what was concluded.
2. Write three alternative explanations for the observation.
3. For each alternative, name the check that would rule it out, and say whether the study did it.
4. Ask who is missing from the data, and whether they would look different.
5. Say how much this should move you, given what you believed beforehand.
6. Write what would change your mind.

What a considered answer tends to contain (not a mark scheme): A considered answer usually finds at least one alternative the source did not address, and can say specifically what evidence is still missing rather than concluding the study is bad.

Use it on real work: Stress-test a claim you are relying on

Find the claim your current project or decision most depends on, and put it through this.

1. Identify the claim your conclusion would not survive without.
2. Trace it to its primary source and read what was actually done.
3. Generate alternative explanations and check which the source addressed.
4. Look for anyone who has argued with it since.
5. Write down what you now think, and what would change it.

Reflect

Which alternative explanation did you not think of until you deliberately looked for one?