Thinking with evidence · Lesson 2 of 3

Selection, survivorship, and who is missing

Ask who is missing from the data. Absent cases are the most reliably overlooked evidence there is.

8 min read · Free — no account needed to read

A selection effect happens when the way a sample was assembled is related to what you are measuring. The finding is then partly a fact about the selection rather than the world.

Survivorship is the most familiar form: you study what remains. Successful companies, surviving buildings, patients still attending, users who did not churn. The ones that disappeared held the information that would have changed the conclusion, and they are invisible precisely because they are absent.

Non-response is the same problem in surveys. The people who answer differ systematically from those who do not, and the difference is usually related to the topic — people with strong feelings answer surveys about that topic.

Publication bias is selection acting on the literature itself. Studies finding an effect are more likely to be written up, submitted and published than studies finding nothing, so the published record can show a clear effect where the full body of work would show none. It is a major reason that systematic reviews search for unpublished work.

The habit worth building is one question, asked routinely: who is not in this data, and would they have looked different? It costs seconds and catches a large share of the errors in this module.

See it happen

An analysis of 200 successful startups finds that 80% of the founders had a mentor, and concludes mentorship drives success.

The read that arrives on its own

A large sample and a strong pattern — mentorship looks like a real success factor.

The same situation, with this idea

You ask who is missing. Only successful startups were studied. If 80% of *failed* founders also had mentors, the figure says nothing about success at all — it is a fact about how common mentors are among founders. Without the comparison group the number cannot do the work it is being asked to do.

After this, you can

notice when a study of successes is being read as a study of what causes success.

The takeaway

Ask who is missing from the data. Absent cases are the most reliably overlooked evidence there is.

Easy to take the wrong way

That a study of one group is invalid. It is bounded. It becomes an error only when the conclusion is stated as though the missing group had been looked at.

Try this

Take a dataset or source in your project and write down who or what could not have appeared in it. Then ask whether their absence would change the finding.