Statistics for researchers · Lesson 2 of 3

Sampling, and where bias gets in

Selection decides what a result can support. Size only decides how precise the wrong or right answer is.

8 min read · Free — no account needed to read

How you select cases decides what your results can support. It matters more than how many you have.

Random sampling gives everyone an equal chance and is what supports generalisation. Stratified sampling divides the population into groups first and samples within each, which guarantees small groups appear. Cluster sampling samples whole groups — schools, villages — which is practical when you cannot reach individuals. Systematic sampling takes every nth case.

Then the non-probability methods, which are extremely common and perfectly legitimate when their limits are stated. Convenience sampling takes whoever is reachable. Purposive sampling deliberately chooses cases that can speak to the question, and is the norm in qualitative work. Snowball sampling has participants recruit others, which is often the only way to reach hidden populations and builds in the bias that people recruit people like themselves.

Sampling bias is when selection is related to what you are measuring. Online polls reach people who are online and inclined to answer. Customer satisfaction surveys reach people who still use the product. Clinical studies reach people who attend appointments.

And the number: larger samples reduce random error, and they do nothing whatever about bias. A biased sample of a million is still biased — it is just precisely so. This is the single most useful thing in the module, because large numbers are persuasive in a way that has nothing to do with whether they are right.

See it happen

A product survey emailed to all users gets 2,400 responses, showing 78% satisfaction.

The read that arrives on its own

2,400 responses is a large sample, so 78% is a reliable picture of the user base.

The same situation, with this idea

You ask who responded. If 30,000 were emailed, 92% did not reply — and the people who do reply to a product survey are disproportionately those with strong feelings or active engagement. The dissatisfied who already stopped using the product are not in the data at all. The finding is "78% of the 8% who responded", and the 2,400 makes it precise, not representative.

After this, you can

ask what the response rate was and who is systematically absent.

The takeaway

Selection decides what a result can support. Size only decides how precise the wrong or right answer is.

Easy to take the wrong way

That non-probability sampling is invalid. It is standard and often necessary — it becomes an error when results are reported as though the sample were representative.

Try this

For any survey result you rely on, find the response rate. If it is not reported, note that as a limitation.