Judging quality in each family · Lesson 1 of 3

Validity and reliability

Reliability asks whether the measure is steady. Validity asks whether it is about the thing you claim.

9 min read · Free — no account needed to read

Reliability is consistency: would the same measurement, repeated, give the same result? A bathroom scale that reads three kilos heavy every time is perfectly reliable and wrong.

Validity is whether you are measuring what you claim. That scale is reliable and not valid. The two are independent, and reliability is much easier to achieve, which is why unreliable-looking measures get more scrutiny than invalid ones.

Validity comes in kinds worth recognising. Construct validity — does the measure capture the idea? A reading-speed test is not a measure of comprehension. Internal validity — within this study, does the conclusion follow, or could something else explain the result? This is where confounders live. External validity — does it hold beyond this sample and setting? Ecological validity — does it hold outside the artificial conditions of the study? A laboratory task can be internally impeccable and tell you little about the same behaviour in a workplace.

These trade off against each other, which is the part that gets missed. Tight experimental control raises internal validity and often lowers ecological validity: the more you strip away the messy real conditions, the more confident you are about cause and the less sure you are it happens outside the room. Neither is a flaw. Recognising the trade-off is how you read a study without either over- or under-crediting it.

See it happen

A company measures "developer productivity" as commits per week and finds a tool increased it by 30%.

The read that arrives on its own

A clear, consistently recorded, objective measure with a large improvement — that is good evidence.

The same situation, with this idea

You separate the two. The measure is highly reliable: commits are counted the same way every time. Its construct validity is the question — do commits measure productive work, or do they measure the habit of committing? A tool that encourages smaller, more frequent commits would produce exactly this result with no change in output at all. The number is solid; what it is a number *about* is unresolved.

After this, you can

notice that a consistent measure and a meaningful one are different achievements.

The takeaway

Reliability asks whether the measure is steady. Validity asks whether it is about the thing you claim.

Easy to take the wrong way

That a study should maximise every kind of validity. They trade off; the reasonable expectation is that the researcher says which they prioritised and why.

Try this

Take the main measure in your project and ask: is it reliable, and is it valid? If it is one and not the other, say which in your notes.