Module 18 · 3 lessons

Peer review

Free — no account needed to read

How work gets checked — and how to do the checking well.

Why it matters

Reviewing teaches you to read critically faster than anything else, and being reviewed is much less bruising once you understand what a reviewer is actually trying to do.

What it teaches

• What peer review is for, and what it cannot do
• Single blind, double blind, open, and post-publication
• How reviewers evaluate a paper
• Writing review comments that help
• Responding to reviewers without capitulating or fighting
• Where peer review goes wrong

When you reach for it

• You are asked to review something for the first time
• You have received reviews and your first reaction is not calm
• You want to read a paper the way a reviewer would

The common misreading

That peer review certifies truth. It is a check on whether the work is competently done and the conclusions are supported — routinely useful, routinely imperfect, and not a guarantee.

What this module will not tell you

This module will not give you your venue's norms. Expectations differ enormously between fields and journals — read the reviewer guidelines and, if you can, two reviews somebody else has written.

Think first

A reviewer writes: "The methodology is weak and the conclusions are overstated." What is wrong with that review, regardless of whether it is right?

Write down what you would want from a reviewer if you had spent two years on a piece of work. Then write what you would want them to catch, even if it hurt.

The lessons

• What it is for, and the forms it takes — Peer review checks competence and support. It does not verify data or guarantee replication. (7 min)
• How to review a paper — Separate major from minor, number your points, name the location, and say what would fix it. (9 min)
• Being reviewed, and where review fails — Respond point by point, change what should change, and disagree with reasons rather than by silence. (8 min)

Practise it: Review a real paper properly

Pick a published paper in your field and review it as if asked to. This is the fastest critical-reading practice available.

1. Write what you understood the research question to be.
2. Say whether the method can answer that question, and why.
3. Assess the execution: sampling, measures, analysis, completeness of reporting.
4. Compare the results section with the abstract, and note any gap in confidence.
5. Write three major points and three minor ones, numbered, each naming a location and a possible fix.
6. Reread your comments and remove anything that is about the author rather than the work.

What a considered answer tends to contain (not a mark scheme): A considered review separates major from minor, names locations, and would tell the authors something they could act on tomorrow.

Use it on real work: Have your own work reviewed

Ask somebody to review a piece of your work using the questions above, and respond properly.

1. Send a draft and the six reviewer questions to a colleague, supervisor or peer.
2. Read what comes back, wait a day, then read it again as information.
3. Write a point-by-point response: changed, or not changed with reasons.
4. Make the changes you agree with.
5. Record which criticism was hardest to accept, and whether it was right.

Reflect

What did reviewing somebody else's work show you about your own?