Selection bias

Any situation where the sample in front of you was not assembled in a way that represents the whole population, so conclusions drawn from it do not generalise.

Example

A fund family advertising the record of its current funds has excluded the ones that were closed or merged away after poor results.

Important

Survivorship bias is one kind of selection bias; the broader habit worth keeping is to ask of any data set, 'how did these particular cases end up in front of me?'

Related terms

  • Survivorship bias — Drawing conclusions from the examples that lasted long enough to be visible, while the ones that failed have quietly left the sample.
  • Base rate — How often an outcome occurs across the whole population of similar cases, before you know anything specific about the one in front of you.

Lessons that explain it

  • The customers you find yourself tell you more than the ones you are given — Customer calls are the most direct evidence of demand. They are most useful when they ask about past behaviour and include customers the company did not choose. (10 min)
  • The references you find yourself are the ones that matter — Back-channel references come from people who worked with a founder but were not suggested by them. Structured, careful conversations with them reveal patterns that curated references cannot. (10 min)
  • Appraising one opportunity: evidence of the outcome, not the promise of it — Before paying for a course, qualification or programme, look for evidence that people like you achieved the outcome you want, check completion and employer demand, and compare cheaper routes. (12 min)
  • Survivorship bias — Almost every success story you can read about was selected for having succeeded. The ones that would have made the lesson honest are not in the sample. (8 min)
  • Record the decisions where you said no, or you only learn from half your judgement — Journals usually cover what was bought. The opportunities you declined are an equally large source of learning, and the only way to know whether your reasons for saying no were sound. (10 min)

All 578 terms in the Investing glossary