Statistical methods · Lesson 2 of 3

Structures: nesting, time and Bayesian thinking

Match the method to the structure in your data — nesting, censoring and time are design facts, not nuisances.

9 min read · Free — no account needed to read

Multilevel models — also called mixed or hierarchical models — are for data with structure: pupils in classrooms, patients in hospitals, repeated measures within a person. They model variation at each level instead of pretending it is not there, and they are the correct answer to the classroom problem above.

Survival analysis is for time-to-event data: how long until something happens, where some cases have not had the event yet when observation ends. That partial information — censoring — is exactly what makes ordinary methods unsuitable and what survival analysis is built to use rather than discard.

Time series methods handle observations ordered in time, where each point is related to the ones before it. They separate trend, seasonality and noise, and they are what makes "sales went up after the campaign" answerable, because they can ask whether the rise exceeds what the trend predicted anyway.

Bayesian methods ask a different question. The usual approach asks how surprising your data would be if there were no effect. The Bayesian approach combines a prior — what was believed beforehand — with the data to give a probability distribution over the effect itself, which is the thing people usually wanted in the first place.

Its advantages are that it states uncertainty directly, handles small samples and accumulating evidence naturally, and makes the prior explicit. Its cost is that the prior is a choice that has to be justified, and a sensitivity analysis showing how much the conclusion depends on it is good practice.

You do not have to choose a camp. Knowing that the p-value answers a narrow question is most of the benefit, however you then proceed.

See it happen

A pilot study of 18 patients shows a promising effect that is not statistically significant.

The read that arrives on its own

Not significant, so the study found nothing and cannot support continuing.

The same situation, with this idea

You separate two claims. "We did not detect an effect" is what happened; "there is no effect" is not, and with 18 patients the study had little chance of detecting anything but a large one. A Bayesian analysis would state what the data says about the plausible range of effects, which is what a decision about funding a larger trial actually needs. Either way, the honest report is an estimate with wide uncertainty, not a null result.

After this, you can

distinguish "we did not detect it" from "it is not there".

The takeaway

Match the method to the structure in your data — nesting, censoring and time are design facts, not nuisances.

Easy to take the wrong way

That Bayesian methods are a way to get a result when the usual approach gives none. Done properly they state uncertainty more honestly, which often means a wider range, not a friendlier one.

Try this

Describe the structure of your data in one sentence — what is nested in what, and what repeats — before choosing any method.