Statistical methods · Lesson 3 of 3

Missing data, measurement error, robust methods

Ask why data is missing, what your measures could be attenuating, and decide outlier rules before seeing results.

9 min read · Free — no account needed to read

Missing data is a design problem wearing a technical costume. What matters is *why* it is missing. Missing completely at random is harmless and rare. Missing at random — explainable by other variables you have — is handleable. Missing not at random, where the missingness depends on the unobserved value itself, biases results and cannot be fixed by any technique.

The common default, deleting every incomplete case, quietly assumes the first and can discard most of the dataset. Multiple imputation is generally better under the second. Either way the reporting standard is the same: say how much was missing, where, and what you assumed.

Measurement error attenuates — an imprecise measure typically pulls an estimated relationship towards zero, so a real effect looks smaller or absent. This is why "we found no relationship" needs its measures examined before it is believed, and it is a direct continuation of the construct and operational definition work in module 03.

Outliers are a decision, not a category. Some are data-entry errors and should be corrected; some are real extreme cases carrying genuine information. The rule that keeps you honest is the one from module 10: decide the handling rule before seeing what it does to the result, and report the analysis both ways when it matters.

Robust methods — robust regression, non-parametric tests, bootstrapping — make fewer assumptions about distribution and are less influenced by extreme values. Bootstrapping in particular estimates uncertainty by resampling your own data, which is useful precisely when the usual formula assumptions do not hold.

And the summary that holds across all of it: no technique repairs a design. These methods handle known problems under stated assumptions — they do not manufacture information the data never contained.

See it happen

Your dataset has 30% missing values on the main outcome, so you analyse only complete cases.

The read that arrives on its own

Dropping incomplete rows is standard and leaves a clean dataset to work with.

The same situation, with this idea

You ask why those values are missing. They are concentrated among participants who disengaged — and disengagement is plausibly related to the outcome itself. Complete-case analysis then studies the people who stayed, which is survivorship from module 05 in numeric form. It may bias the result in a knowable direction; you report the missingness, its pattern, and an analysis under a different assumption to show how much the conclusion depends on it.

After this, you can

ask why data is missing before deciding how to handle it.

The takeaway

Ask why data is missing, what your measures could be attenuating, and decide outlier rules before seeing results.

Easy to take the wrong way

That imputation invents data. It represents uncertainty about missing values properly — the failure is using it while assuming a missingness mechanism that does not hold.

Try this

Report how much of your data is missing and where. If you do not know, that is the first thing to find out.