Statistics for researchers · Lesson 3 of 3

Significance, effect size, and what statistics cannot say

Significant means "probably not nothing". Effect size tells you whether to care. They are different questions.

10 min read · Free — no account needed to read

A p-value answers one narrow question: if there were genuinely no effect, how likely is data at least this extreme? A small p-value means the data would be surprising under "no effect". That is all it means.

It is not the probability that the hypothesis is true. It is not a measure of how large or important the effect is. And a large p-value is not evidence that there is no effect — it may only mean the study was too small to detect one.

Statistical significance and practical importance are different things. With a large enough sample, a difference far too small to matter to anyone becomes statistically significant. With a small sample, a large and important difference may not reach significance. The threshold is a convention, not a fact about nature, and a result just either side of it is nearly the same result.

Effect size is how big the difference actually is, and it is what tells you whether to care. A confidence interval shows the range the data is consistent with, which is more informative than a single estimate: "somewhere between a 1% and a 40% improvement" and "about 20%, give or take 2" can carry the same p-value and mean completely different things.

Power is the chance of detecting an effect that is really there. Underpowered studies both miss real effects and, when they do publish, overstate them.

Finally, the boundary. Statistics can tell you how much, how confident, how variable, and how likely under an assumption. It cannot tell you whether your measure captures the thing you care about, whether your sample represents anybody, whether an association is causal, whether the question was worth asking, or what any of it ought to mean for a decision. Those are all judgement, and no amount of computation supplies them.

See it happen

A trial of 40,000 users finds a redesign produced a statistically significant increase in engagement, p < 0.001.

The read that arrives on its own

Highly significant across a huge sample — the redesign works and should ship.

The same situation, with this idea

You look for the effect size. Sessions rose from 3.10 to 3.14 per week. With 40,000 users almost any difference clears the threshold, so the p-value is telling you the difference is probably real, not that it is worth anything. A 1.3% change may or may not justify the cost of the redesign — and that is a judgement about the business, which the statistics were never going to make.

After this, you can

look for the effect size before reacting to the p-value.

The takeaway

Significant means "probably not nothing". Effect size tells you whether to care. They are different questions.

Easy to take the wrong way

That p-values are useless. They answer a real and narrow question well; nearly all the trouble comes from reading them as answering a broader one.

Try this

Take a significant result in your field and find its effect size and confidence interval. Decide whether the size would change anything you do.