Appraising a Systematic Review

Structured appraisal of a systematic review / meta-analysis (CASP SR, AMSTAR-2 / ROBIS frameworks). Validity turns on: a focused question (PICO); a comprehensive, reproducible search; explicit inclusion criteria; risk-of-bias appraisal of the included studies; an appropriate synthesis (was pooling justified, was heterogeneity assessed); and consideration of publication bias. The pooled diamond is the eye-catching part — but its credibility is set entirely upstream of it.

A meta-analysis forest plot — the polished output a review is judged on at a glance. Here the studies scatter widely and I² is high (78%), a visible warning that the trials disagree. But the things that actually decide whether the diamond is believable — the question, the search, the inclusion criteria, the risk of bias within each trial, and publication-bias checks — all sit upstream and are invisible in the plot itself.

Resist the diamond. First confirm a focused PICO question and a search wide enough to be reproducible (multiple databases, no language limits, grey literature). Check the inclusion criteria are explicit and the authors appraised each included study’s risk of bias. Then ask whether pooling was even appropriate — with I² this high, a single combined estimate may mislead. Finally look for a funnel plot or other publication-bias check.

A systematic review sits atop the evidence hierarchy, so a flawed one carries outsized, misplaced authority. Meta-analysing biased or wildly heterogeneous primary studies produces a precise-looking but untrustworthy number — garbage in, garbage out. Appraising the method (search, inclusion, study-level bias, synthesis) rather than admiring the diamond is what separates real evidence from statistical theatre.

• Validity = PICO · comprehensive search · inclusion · study RoB · synthesis · pub-bias
• Tools: AMSTAR-2, ROBIS; reporting standard: PRISMA
• Synthesis quality is capped by the quality of the included studies

Appraising a Cochrane review — Cochrane reviews are the benchmark precisely because they make the upstream work explicit: a pre-registered PICO protocol, a documented multi-database search, PRISMA flow of included/excluded studies, formal RoB 2 tables for each trial, GRADE certainty ratings, and a funnel plot when enough studies allow it. Reading one, you can audit every appraisal step rather than trust the diamond on faith. Even a Cochrane review can pool heterogeneous or low-certainty trials — always read its risk-of-bias and GRADE tables, not just the headline pooled effect, before changing practice.

• Garbage in, garbage out — trusting a pooled estimate built from biased or low-quality primary studies.
• Ignoring unexplored heterogeneity or publication bias (no I² comment, no funnel plot).
• Confusing review-level bias (AMSTAR-2/ROBIS) with study-level bias (RoB 2) — they are different axes.