Evidence and sources · Lesson 3 of 3

Evaluating a source on what it did

Evaluate on method, sample, comparison and what came after — not on the logo at the top.

8 min read · Free — no account needed to read

Surface signals — a journal name, a university logo, a confident tone, a large number of citations — correlate loosely with quality and are routinely wrong in both directions. The questions that actually discriminate are about what was done.

Who produced it, and what did they want? Not to dismiss anyone: funding and interest are normal and disclosed. You are calibrating, not accusing. A trial funded by a manufacturer is not thereby false; it is a reason to look harder at the outcome measure and the comparator.

What is the evidence behind the claim? Original data, somebody else's data, a model, an argument, or an assertion. All are legitimate; they support very different weights.

Who or what was studied, and how many? A finding about 30 undergraduates in one country may be entirely sound and is not a finding about everybody.

What was compared? A result with no comparison group usually cannot support a causal claim, whatever the discussion section says.

Do the conclusions exceed the evidence? Extremely common, and usually visible by reading the results section and the abstract and noticing they are making different claims.

Has anyone replicated it, or argued with it? Forward citation chasing answers this in minutes and is the single most informative check available.

See it happen

A widely shared paper reports that a brief intervention substantially improved outcomes.

The read that arrives on its own

Peer-reviewed, real journal, large effect, thousands of citations — solid enough to build on.

The same situation, with this idea

You check what was done: 42 participants, one site, self-reported outcome, no active control. All legitimate for an early study, and it makes the result a promising signal rather than an established effect. Then you look forward: three replication attempts since, two finding no effect. The paper is not discredited — the field has moved, and citing only the original would be reporting 2015.

After this, you can

check what happened *after* a paper before you rest anything on it.

The takeaway

Evaluate on method, sample, comparison and what came after — not on the logo at the top.

Easy to take the wrong way

That finding weaknesses means discarding the source. Almost every study has limitations, usually listed by its own authors. The job is weighting, not binary judgement.

Try this

Take the most important source in your project and answer the six questions above in writing. Note which one you could not answer without opening the methods section.