Thinking with evidence · Lesson 3 of 3

Strength of evidence, and what would change your mind

Ask how much this should move you from where you were, and name what would move you further.

8 min read · Free — no account needed to read

Evidence is not binary, and the useful question is not "is this proof?" but "how much should this move me, and from what starting point?"

Base rates are the starting point and the most commonly ignored input. A test that is 95% accurate for a condition affecting one person in ten thousand still produces mostly false alarms, because there are so many more healthy people to be wrong about. Any time you assess a specific case, ask how common the thing is in general before you weigh the specific signal.

Prior plausibility matters too. A surprising claim needs more support than an unsurprising one, not because surprising claims are false but because there are more ways to produce a spurious surprising result than a spurious mundane one.

Convergence across independent methods is worth far more than repetition within one. Three studies with the same design share the same blind spots; a trial, a longitudinal cohort and a natural experiment pointing the same way is a genuinely different situation.

Precision. "An effect somewhere between negligible and large" and "an effect of about this size" are different findings even when both are statistically significant.

The discipline that ties this together is simple to state and uncomfortable to practise: write down, in advance, what would change your mind. If nothing would, you are not weighing evidence. If you can name it, you have turned a belief into something that can be investigated.

See it happen

A single well-conducted study finds that a cheap, widely available supplement substantially reduces a common illness.

The read that arrives on its own

A good study with a clear result on something cheap and harmless — worth acting on.

The same situation, with this idea

You weigh it. The claim is surprising: cheap widely-used substances get studied constantly, so a large real effect would probably have surfaced already — that is a base rate about the literature, not about the study. One study, one design, no replication yet. You conclude it is a reason to watch the space rather than a settled finding, and you write down what would move you: two independent replications with different designs.

After this, you can

state in advance what would move you, so that later evidence actually counts.

The takeaway

Ask how much this should move you from where you were, and name what would move you further.

Easy to take the wrong way

That demanding a lot of evidence is rigour. Refusing to update on good evidence is the same failure as updating on bad evidence, pointing the other way.

Try this

For the central claim of your project, write one sentence: "I would abandon this if ___." Keep it somewhere you will see it when the evidence arrives.