Sensitivity Analysis

Repeating an analysis under different but reasonable assumptions to test how robust the conclusion is — for example excluding studies at high risk of bias, trying different missing-data assumptions, using fixed- vs random-effects models, or per-protocol vs intention-to-treat. If the result barely changes, it is robust; if it flips or loses significance, it is fragile and must be interpreted with caution.

The same data, analysed two ways. The main (gold) diamond sits left of the null and excludes 1 — a significant benefit. The pale “sensitivity” diamond, recomputed after dropping the high-risk-of-bias trials, has slid back over the null line: the apparent benefit was being driven by the weaker studies and does not survive a stricter analysis.

Compare the two diamonds. If a reasonable change of assumption leaves the estimate sitting in much the same place — same side of the null, still excluding 1 — the conclusion is robust. If it swings across the null or loses significance, as here, the conclusion is fragile and rests on assumptions you may not believe.

A single headline estimate hides how much it leans on the choices behind it. A pre-specified sensitivity analysis is an honesty check: it tells you whether a “positive” result is solid enough to change ED practice or is an artefact of which studies, model, or missing-data assumption happened to be used.

• Result unchanged → robust; result flips/loses significance → fragile
• Common levers: high-bias exclusion · fixed vs random effects · ITT vs per-protocol · missing-data assumptions
• Pre-specify it — and report all versions, not just the flattering one

IV magnesium in acute MI — LIMIT-2 (Lancet 1992) and meta-analyses of small early trials suggested a striking ~24% reduction in mortality. Restricting the analysis to the large, low-risk-of-bias mega-trial ISIS-4 (Lancet 1995, ~58,000 patients) made the benefit disappear entirely — a textbook case of a pooled result that was fragile, driven by small-study effects rather than a real treatment benefit. If a meta-analytic result rests on small, lower-quality trials and evaporates when the robust evidence is isolated, treat the headline finding as a hypothesis — not a fact.

• Calling a result robust when it is in fact fragile to a reasonable change of assumption.
• Cherry-picking which analysis to report (only the version that stays significant).
• Confusing sensitivity analysis (robustness check, same population) with subgroup analysis (effect within a subset).