Module 07 · 3 lessons

Judging quality in each family

Quantitative and qualitative work are judged by different criteria, and applying the wrong ones is how good work gets dismissed.

Free — no account needed to read

Quantitative and qualitative work are judged by different criteria, and applying the wrong ones is how good work gets dismissed.

Why it matters

People trained in one family routinely evaluate the other by criteria it was never trying to meet — asking a case study about sample size, or asking an experiment what it means to the participants.

What it teaches

• Validity and reliability, and the kinds of each
• Generalisability, and what it does and does not require
• Credibility, transferability and the qualitative equivalents
• Sample size logic in both families, including saturation
• Where bias enters each, and what is done about it

When you reach for it

• Somebody dismisses a study for having "only" 20 participants
• You need to defend why your own sample is what it is
• Two studies disagree and you suspect it is about how they measured

The common misreading

That these are two sets of words for the same idea. They are not translations — they come from different assumptions about what the research is for.

What this module will not tell you

This module will not give you a threshold: no minimum sample size, no acceptable reliability figure. Those are field-specific conventions, and a number invented here would be worse than none.

Think first

A survey of 10,000 people and a study of 12 people are published in the same week on the same topic and reach different conclusions. What are the possible reasons, before you decide which to believe?

A study interviews 15 people in depth. Write down your first reaction to that number, then write down what the study would have to be asking for 15 to be exactly right.

The lessons

• Validity and reliability — Reliability asks whether the measure is steady. Validity asks whether it is about the thing you claim. (9 min)
• Generalisability and transferability — Generalisability is about how the sample was drawn. Transferability is about how well the context was described. (8 min)
• How many is enough, in each family — Ask "enough to detect what?" or "enough until what stops appearing?" — never "enough" on its own. (8 min)

Practise it: Judge two studies by their own criteria

Find one quantitative and one qualitative study on a topic you care about, and evaluate each on the criteria it was trying to meet.

1. For the quantitative one: how was the sample drawn, what population does it support, is the measure reliable, is it valid for the construct claimed?
2. For the qualitative one: who was chosen and why, is the context described well enough to judge transfer, can findings be traced to the material?
3. Write the one claim each can properly support.
4. Write the one question each cannot answer.
5. Say what the two together tell you that neither does alone.

What a considered answer tends to contain (not a mark scheme): A considered answer resists judging either study by the other's criteria, and ends with a claim narrower than the one in the abstract.

Use it on real work: State your own quality criteria

In a real project, write down how your work should be judged.

1. Name which family your approach belongs to.
2. Write the two or three criteria a reader should hold you to.
3. Write how you are addressing each.
4. Write the one you cannot fully meet, and why — this belongs in your limitations either way.

Reflect

Which criterion had you been applying to everything, regardless of what the study was trying to do?