Research design · Lesson 2 of 3

What each design buys, and what it costs

You are not choosing the strongest design; you are choosing which limitation you can live with and will declare.

9 min read · Free — no account needed to read

Every design is a trade. Knowing the trade is what lets you defend your choice and state your limitations honestly.

Experiment. Buys: the strongest basis for a causal claim, because randomising breaks the link between the condition and everything else. Costs: often artificial, frequently impossible for practical or ethical reasons, and usually narrow in scope.

Quasi-experiment / natural experiment. Buys: causal leverage in the real world where assignment cannot be controlled. Costs: groups may differ in ways you did not measure, so the argument rests on how convincingly you handle that.

Longitudinal / cohort. Buys: order in time, which is necessary for causation and settles reverse causation. Costs: slow, expensive, and people drop out in ways related to what you are studying.

Cross-sectional survey. Buys: breadth quickly and cheaply, and generalisation if sampled well. Costs: one moment only, self-report, and no direction of causation.

Case study. Buys: depth and context, and the ability to see a mechanism in operation. Costs: cannot establish how common anything is.

Ethnography / observation. Buys: what people actually do, as opposed to what they say. Costs: time, access, and the researcher's presence changing the setting.

Interviews. Buys: meaning, reasoning and experience in people's own terms. Costs: accounts are reconstructions, shaped by memory and by being asked.

Secondary data. Buys: scale and speed. Costs: you are limited to what somebody else decided to record, for their purposes.

Systematic review. Buys: what the field collectively shows, with a documented and repeatable process. Costs: months of work, and it inherits every weakness of the literature it reviews.

See it happen

You have access to two years of company data and want to know whether a policy change helped.

The read that arrives on its own

The data exists and covers the period, so analyse before versus after and report the difference.

The same situation, with this idea

You name the trade. Secondary data buys you scale and speed at no cost, and it limits you to what was recorded for operational reasons — which may not include the outcome you care about. Before-and-after gives no comparison group. So you look for one: a region or team where the policy arrived later, giving you a staggered comparison. If none exists, the honest output is an association with the alternatives named, not a claim that the policy worked.

After this, you can

state what your design costs you, in the same breath as what it buys.

The takeaway

You are not choosing the strongest design; you are choosing which limitation you can live with and will declare.

Easy to take the wrong way

That naming a limitation weakens your work. An undeclared limitation is a weakness. A declared one is a boundary, and it is what makes the rest credible.

Try this

Write the buy and the cost for the design you are leaning towards, and check the cost is one your question can survive.