Evidence synthesis · Lesson 3 of 3

Weighting quality, and the studies that are not there

Weight by quality, and ask what is not in the literature. Absent studies are the most consequential evidence there is.

8 min read · Free — no account needed to read

Risk of bias assessment asks, for each study, whether its design and conduct could have produced the result independently of the truth: how were participants assigned, were outcomes assessed blind, how much attrition was there, were all pre-specified outcomes reported.

The reason to do it is that pooling regardless of quality lets a large, badly-conducted study dominate a careful small one. A synthesis that ignores quality is a vote count, and votes are not evidence.

Then the harder problem: publication bias. Studies finding an effect are more likely to be written up, submitted, and accepted than studies finding nothing. So the published literature can show a clear effect where the full body of work would show none — and every included study can be individually sound.

What is done about it: search for unpublished work, theses, trial registries and conference abstracts; contact authors; and check for asymmetry in the pattern of results, where small studies showing large effects and no small studies showing none is the signature.

Related and worth naming: outcome reporting bias, where a study measured six outcomes and reported the two that worked — which is why trial registries that record outcomes in advance exist, and why a review should check the registration against the paper.

The honest conclusion is sometimes that the evidence base cannot answer the question. That is a finding, and it is more useful than a pooled estimate built on a literature with a hole in it.

See it happen

Twelve studies of an intervention: ten find an effect, two find nothing.

The read that arrives on its own

Ten to two is a strong weight of evidence — the effect is real.

The same situation, with this idea

You look at what produced those twelve. The ten are small and the two are the largest; that pattern is the signature of publication bias, where small studies finding nothing were never published. You check trial registries and find four registered studies with no published results. The honest summary is that the evidence is consistent with a small effect or with none, and that a fifth of the registered work is missing — which is a more useful thing to tell a decision-maker than "ten out of twelve".

After this, you can

ask which studies would be missing before counting the ones present.

The takeaway

Weight by quality, and ask what is not in the literature. Absent studies are the most consequential evidence there is.

Easy to take the wrong way

That you can correct publication bias statistically. The methods detect asymmetry and estimate its size under assumptions; they do not recover the missing studies.

Try this

For a question you care about, search a trial or study registry and compare what was registered with what was published.