Chance vs Bias vs Confounding

Before an observed association can be called causal, three rivals must be ruled out. Chance is random error — addressed by p-values, confidence intervals and adequate power. Bias is systematic error built into the study’s design or conduct (e.g. selection or measurement bias) — not fixed by a bigger sample. Confounding is a third factor linked to both exposure and outcome — addressed by design (randomisation, matching, restriction) or statistical adjustment.

Two overlapping sampling distributions split by the critical value — the picture of random error. This is the only one of the three rivals that statistics directly quantify: the shaded tails are the false-positive and false-negative rates that arise purely from sampling. Bias and confounding sit outside this picture entirely.

Use it as a reminder of scope. A narrow confidence interval and a tiny p-value squeeze the curves and shrink the overlap — chance is handled. But the same diagram is silent on whether the wrong patients were selected (bias) or whether a third factor drove the result (confounding). Those need design and adjustment, not arithmetic.

It is the master checklist for appraising any reported association. Tackle the three in turn — could it be chance? bias? confounding? — before accepting causation. Skipping straight to the p-value is how spurious associations enter practice and how genuine ones get dismissed.

• Chance = random error → CIs, p-values, power
• Bias = systematic error → better design, NOT bigger n
• Confounding = third factor → randomise / adjust

Coffee, smoking and cancer — early observational studies repeatedly linked heavy coffee drinking to lung and pancreatic cancer. The association was real in the data but largely confounded by smoking: heavy coffee drinkers smoked more, and smoking drives the cancer. Once smoking was adjusted for, much of the apparent coffee effect dissolved — a textbook reminder that a robust, repeatable association can still be non-causal. No amount of extra sample size would have fixed this — only design or adjustment removes a confounder. A bigger biased study just yields a tighter wrong answer.

• Treating p<0.05 as proof of no bias — significance speaks only to chance.
• Thinking a larger sample cures bias — it shrinks random error, not systematic error.
• Confusing confounding (distorts the true effect, to be removed) with effect modification (a real difference in effect across subgroups, to be reported).