Module 19 · 3 lessons

Statistical methods

What the common techniques are actually for, so you can choose and read them. You do not need to compute any of these by hand.

Free — no account needed to read

What the common techniques are actually for, so you can choose and read them.

Why it matters

You do not need to compute any of these by hand. You do need to know which question each answers, what it assumes, and how it fails — otherwise the software will happily give you a number for the wrong thing.

What it teaches

• Comparing groups: t-tests, ANOVA, chi-square
• Relationships: correlation, regression, multiple and logistic
• Bayesian thinking alongside the usual approach
• Multilevel, survival and time series — what each is for
• Missing data and measurement error
• Robust methods, and when the usual assumptions fail

When you reach for it

• You know what you want to find out and not what to run
• You are reading a results section full of unfamiliar names
• Your data breaks an assumption you have just discovered exists

The common misreading

That the technique is the hard part. The hard parts are the design that produced the data and the assumptions the technique makes — both decided before any analysis runs.

What this module will not tell you

This module will not teach you to perform these analyses. It teaches you what each is for and what it assumes; the doing belongs in R, Python, JASP, jamovi, SPSS or Stata, and the Toolkit points at them.

Think first

You have test scores from 400 pupils across 20 classrooms. Why might treating them as 400 independent observations be wrong?

Write down a comparison you want to make with your data. Then write what would make that comparison unfair — not statistically, just in plain terms.

The lessons

• Comparing groups, and describing relationships — Choose by the question and the shape of your data, and plot it before you model it. (10 min)
• Structures: nesting, time and Bayesian thinking — Match the method to the structure in your data — nesting, censoring and time are design facts, not nuisances. (9 min)
• Missing data, measurement error, robust methods — Ask why data is missing, what your measures could be attenuating, and decide outlier rules before seeing results. (9 min)

Practise it: Choose the method by reasoning to it

Take your own data — or data you plan to collect — and reason from its shape to a method.

1. Write what you want to find out: a difference, a relationship, a prediction, a time to an event, a change over time.
2. Describe the structure: what is nested in what, what repeats, what is ordered in time.
3. Name your outcome type: continuous, binary, count, categorical, time-to-event.
4. From those three, write which methods become appropriate and what each would assume.
5. State how much data is missing and why.
6. Write the assumption most likely to be violated, and how you would check it.

What a considered answer tends to contain (not a mark scheme): A considered answer names a structure and an assumption, and keeps more than one method alive until something about the data rules it out.

Use it on real work: Write the analysis plan

Record the plan in a real project, before you see results.

1. Write the method, the outcome type and the structure into your project brief.
2. Write the assumptions it makes and how you will check each.
3. Write your rule for outliers and for missing data, now.
4. Date it, and treat any later deviation as something to report rather than to hide.

Reflect

Which assumption in your analysis had you not realised you were making?