Module 16 · 3 lessons

Evidence synthesis

What does the whole literature say — established with a process, not an impression. An ordinary literature review reflects whatever you happened to find.

Free — no account needed to read

What does the whole literature say — established with a process, not an impression.

Why it matters

An ordinary literature review reflects whatever you happened to find. A synthesis has a method, which is why it can be checked, repeated and trusted with a decision.

What it teaches

• Narrative, scoping, systematic, rapid and umbrella reviews
• Meta-analysis and meta-synthesis
• The systematic review workflow, step by step
• Inclusion and exclusion criteria
• Risk of bias, and why quality is weighted
• Publication bias and the missing studies

When you reach for it

• You need to state what a field collectively shows
• Two studies disagree and you need to know which way the weight falls
• A decision is going to be made on the evidence and it should be all of it

The common misreading

That "systematic review" is a grander name for reading a lot. The method is the point: stated criteria, a documented search, and screening decisions somebody else could repeat.

What this module will not tell you

This module will not make a synthesis better than the studies in it. A careful review of weak work produces a careful account of weak work — which is a real finding, and must be reported as one.

Think first

Ten published studies find an effect; two find nothing. What would you need to know before concluding the effect is real?

Think of a question in your field where "the evidence is mixed". Write down how you would establish what the evidence actually is, rather than which studies you remember.

The lessons

• The kinds of review, and what each is for — Pick the review your question and your calendar can both support, and name it accurately. (8 min)
• The systematic review workflow — Criteria before searching, screening documented, exclusions counted. The process is what a reader is being asked to trust. (10 min)
• Weighting quality, and the studies that are not there — Weight by quality, and ask what is not in the literature. Absent studies are the most consequential evidence there is. (8 min)

Practise it: Write a review protocol

The protocol is the whole discipline. Writing one takes an hour and decides whether the review means anything.

1. State the question in structured form: population, intervention or exposure, comparison, outcome.
2. Write inclusion criteria: designs, populations, period, languages, outcomes.
3. Write exclusion criteria, and why each one exists.
4. List the databases and sources you will search, including grey literature.
5. Draft the search string for one database.
6. Say how screening will work and what you will do about disagreement.
7. List the fields you will extract from every study, decided now.

What a considered answer tends to contain (not a mark scheme): A considered protocol has criteria specific enough that two people would screen the same paper the same way, and states what it will do when they do not.

Use it on real work: Run the protocol into a real library

Take the protocol into a project and start the funnel.

1. Save the protocol into the project brief, dated, before searching.
2. Run the search and record the counts per database.
3. Screen into the library and set each source to included, excluded or duplicate.
4. Record why each full-text exclusion happened.
5. Write the numbers at each stage — found, screened, excluded, included.

Reflect

What would your review have concluded if you had written the criteria after searching instead of before?