Statistical methods · Lesson 1 of 3

Comparing groups, and describing relationships

Choose by the question and the shape of your data, and plot it before you model it.

10 min read · Free — no account needed to read

Most analyses answer one of two questions: do these groups differ, or do these things vary together?

Comparing groups. A t-test compares means between two groups. ANOVA extends that to three or more, and tells you that some difference exists without saying which pair — post-hoc comparisons do that. Chi-square tests whether two categorical variables are associated, such as whether outcome differs by group. Each assumes things about the data; the assumption that bites most often is independence, which is broken whenever observations are nested or repeated.

Relationships. Correlation measures how strongly two variables move together, between −1 and 1, and describes only straight-line relationships — a strong curved relationship can show a correlation near zero, which is why plotting the data first is not optional.

Regression predicts an outcome from one or more predictors, and reports how much the outcome changes per unit of predictor, holding the others constant. Multiple regression is the workhorse of observational research and carries module 14's warning in full: "holding constant" is only as good as the variables you chose, and choosing wrong introduces bias rather than removing it.

Logistic regression is for a binary outcome — happened or not — and reports odds ratios rather than differences in means. Odds ratios are widely misread as risk ratios; when the outcome is common the two diverge substantially.

The habit that prevents most errors: plot the data before modelling it. Distributions, outliers, curvature and grouping are visible in thirty seconds and invisible in a coefficient table.

See it happen

You compare exam results between two teaching methods across 20 classrooms using a t-test on 400 pupils.

The read that arrives on its own

400 observations and a clear difference between the two methods — a straightforward comparison.

The same situation, with this idea

You check independence. Pupils in the same classroom share a teacher, a room and each other, so their scores are not independent — the effective sample is closer to 20 classrooms than 400 pupils. Treating them as 400 makes the confidence interval far too narrow and the result look far more certain than the data supports. The design needs a method that accounts for nesting, which is the next lesson.

After this, you can

check independence before trusting any interval.

The takeaway

Choose by the question and the shape of your data, and plot it before you model it.

Easy to take the wrong way

That a significant coefficient in a regression means the predictor causes the outcome. Module 14 applies in full, and the software will not mention it.

Try this

Plot your main variables before running anything. Note anything you would not have seen in a table.