Chapter 12. Never mistake correlation for causation — thinking about causal inference

Key points

• "There is a correlation" does not mean "this is the cause". This is the biggest pitfall of all
• Three traps that mislead about causation: confounding, reverse causation, selection bias
• The royal road to showing effect correctly is the RCT; alternatives exist when that is impossible

The one line that matters most

If there is only one thing to carry away from this whole guide, it is this.

Correlation is not causation.

Two things moving together does not make one the cause of the other. Get this wrong and you convince yourself something works when it does not, and pour resources into the wrong intervention or the wrong service. In research and in business, the most frequent and most expensive mistakes start here.

The famous example — ice cream and heatstroke

Ice cream sales and heatstroke cases correlate strongly. The more ice cream sells, the more heatstroke. Does ice cream cause heatstroke?

Of course not. A third factor — temperature — is raising both. When it gets hot, ice cream sells and heatstroke increases. There is no direct causal link between them.

That hidden factor affecting both is the biggest trap of all.

Three traps

Trap 1: confounding. Another factor that affects both the intervention and the outcome creates an apparent causation. Example: "people who take a supplement have better cognitive function" → in reality, people who take supplements are already more health-conscious, and their whole lifestyle is better. What worked may be the lifestyle, not the supplement.

Trap 2: reverse causation. Cause and effect are the other way round. Example: "users of a brain-training app have lower cognitive function" → the app did not lower their function; people worried about their cognition were the ones who started using it. Cause and effect are reversed.

Trap 3: selection bias. Who took part distorts the result in the first place. Example: "users of a cognitive rehabilitation programme do better" → the people who used it were only the highly motivated ones. The programme's effect cannot be separated from the skew in who joined.

For a therapist this should feel close to differential diagnosis. Does this finding really come from this condition? Is there another cause? Am I reading the causation backwards? Turn the caution the clinic taught you onto data.

The royal road to showing effect — the RCT

So how do you avoid these traps and show that something really worked? The royal road is the randomised controlled trial (RCT).

Participants are divided at random into an intervention group and a control group, and the effect is compared. Randomising spreads every hidden factor — age, motivation, lifestyle — evenly across both groups. So if a difference appears, it can be attributed to the intervention. Confounding and selection bias are both minimised. That is why the RCT is the gold standard of evidence in medicine and pharmaceuticals.

Example: randomly assign 500 older adults to app-using and non-using groups, and compare cognitive function after three months.

The RCT of business — the A/B test

You may think RCTs belong to research and not to practice. Not so. The A/B test, familiar from web marketing, is an RCT.

Show designs A and B of an advertisement, landing page or product description to users at random, and compare click-through or purchase rates.

Randomise, keep a control, compare. Structurally identical. If you want to know whether something really worked in practice, simply carrying the A/B testing instinct lifts the quality of your decisions a level.

When an RCT is not possible

Cost, ethics and feasibility often make an RCT impossible. That is where methods for approaching causation from observational data come in.

• Propensity score matching (PSM) — match intervention and control cases so their background conditions are similar, then compare
• Difference-in-differences (DID) — compare the change before and after between intervention and control groups (needs time-series data)
• Instrumental variables (IV) — use an external variable to strip out confounding (common in economics)

The names are forbidding; the idea is one. If you cannot randomise, engineer a fair comparison by other means. These methods are advanced, so work with a specialist if you use them seriously. With AI, the right conversation is "which alternative looks suitable for my data and situation?"

A prompt to try (causal pitfall check)

You are a researcher who knows causal inference well. I obtained the following result and believe the intervention worked.

[Observed result] (e.g. participants in our facility's exercise class had higher life satisfaction than non-participants)

1. Point out concretely how confounding, reverse causation and selection bias could each be lurking in this conclusion.
2. Explain what study design (an RCT or otherwise) would ideally be needed to say the effect is real.
3. If an RCT is impractical, propose which of the observational alternatives (PSM, DID, IV) suits this situation.
4. Warn me about the risks of publishing this as "it worked" as it stands.

Chapter 12 summary

• Correlation is not causation — the most expensive mistake in data work starts here
• Three traps: confounding, reverse causation, selection bias (spot them as you would in differential diagnosis)
• The royal road is the RCT; a business A/B test shares its structure; alternatives exist when it is impossible