How to Pass the FRCEM Critical Appraisal Section

Process at a glance

Critical appraisal is no longer examined as a standalone FRCEM paper or viva, but it remains a high-yield part of emergency medicine exams. In current UK EM exams, evidence interpretation appears mainly in SBA questions and may also influence OSCE reasoning, justification of management, and safe application of guidelines. The skill being tested is practical: identify the study design, spot the main threat to validity, interpret the result correctly, and decide whether the evidence should change UK emergency practice.

For MRCEM SBA, the level is usually more basic: common study designs, simple statistics, and safe interpretation. For FRCEM SBA, the questions are broader and more nuanced: confidence intervals, diagnostic studies, non-inferiority, meta-analysis, and applicability to NHS practice. For FRCEM OSCE, you are unlikely to face a formal appraisal station, but you may need to justify decisions using evidence, guidelines, and risk-benefit reasoning.

Why the FRCEM Critical Appraisal Section Matters

Emergency medicine depends on rapid decisions made under uncertainty. Many of those decisions are driven by evidence: whether a troponin pathway is safe, whether age-adjusted D-dimer reduces unnecessary imaging, whether a risk score is ready for use, or whether a new intervention offers meaningful benefit.

In real practice, poor appraisal leads to two common errors:

• Adopting weak evidence too early
• Rejecting useful evidence because the statistics are misunderstood

In the exam, the same errors lose marks. Candidates often memorise isolated definitions but struggle when shown an abstract, a 2 x 2 table, a confidence interval, or a forest plot. The exam rewards structured judgement, not detached statistical trivia.

The core consultant skill is to answer five questions quickly:

• What is the clinical question?
• What study design is this?
• What is the biggest threat to validity?
• What do the results actually mean?
• Would this change UK emergency practice?

Key Definitions

Essential Pathophysiology

Critical appraisal has no biological pathophysiology, but there is an underlying logic to how evidence becomes trustworthy or misleading.

• Bias distorts the estimate away from the truth
• Confounding creates false associations in non-randomised studies
• Random error causes imprecision, often seen as wide confidence intervals
• Poor outcome choice can make a study statistically positive but clinically unhelpful
• Poor applicability means a valid study may still not fit UK ED practice

Think of evidence failure in three layers:

1. Was the study designed appropriately for the question?
2. Was it conducted well enough to trust the result?
3. Even if valid, does it apply to my patients, pathways, and resources?

This is the framework examiners expect.

Clinical Presentation

In exam terms, critical appraisal usually presents in one of a small number of ways:

• A short abstract asking for the study design
• A results table asking for the best interpretation
• A confidence interval asking whether the result is significant, precise, or clinically important
• A diagnostic 2 x 2 table asking for sensitivity, specificity, predictive values, or likelihood ratios
• A forest plot asking about pooled effect or heterogeneity
• A paper summary asking whether the authors’ conclusion is justified
• A clinical scenario asking whether evidence should change UK practice

The commonest paper types are:

• Randomised controlled trials
• Diagnostic accuracy studies
• Cohort studies
• Case-control studies
• Systematic reviews and meta-analyses
• Clinical prediction rule studies
• Before-and-after service evaluations
• Non-inferiority trials

Red Flags and High-Risk Features

These are the features that should make you cautious in both exams and practice.

• Observational study making causal claims
• Single-centre study with highly selected patients
• Surrogate outcomes instead of patient-important outcomes
• Large relative effect but tiny absolute benefit
• Wide confidence interval compatible with both benefit and harm
• Non-significant superiority trial described as proving equivalence
• Positive subgroup analysis despite neutral primary outcome
• Diagnostic study with poor or inconsistent reference standard
• Verification bias: not all patients receive the reference standard
• Spectrum bias: study population unlike real ED case mix
• Derivation study presented as if ready for implementation
• Systematic review pooling poor-quality or very heterogeneous studies
• Service redesign paper ignoring secular trends or co-interventions
• Single paper contradicting established NICE guidance without sufficient supporting evidence

Differential Diagnosis

When a question asks for the “best criticism” or “most likely interpretation”, your differential diagnosis is the list of possible flaws. The key is to choose the dominant one.

Initial ED Assessment

The fastest safe approach in the exam is a structured five-step appraisal.

Step 1: Define the clinical question

Decide whether the paper is about treatment, diagnosis, prognosis, risk prediction, harm, or service delivery. If you misclassify the question, you will often misjudge the design.

Step 2: Identify the study design

This is the highest-yield first move.

Step 3: Judge internal validity first

Ask what single flaw most threatens the result. In SBA questions, the best answer is usually the biggest problem, not a list of minor imperfections.

Step 4: Interpret the result in plain English

Translate statistics into a clinical statement.

• What is the direction of effect?
• How large is it?
• How precise is it?
• Is it clinically important?

Step 5: Assess applicability to UK emergency practice

Even a valid study may not fit your setting.

• Does the population resemble NHS ED patients?
• Does the comparator reflect current UK practice?
• Does the pathway depend on a specific assay or resource?
• Is it consistent with NICE, RCEM, BTS, SIGN, or local governance?

Investigations

For exam purposes, your “investigations” are the tools used to interrogate a paper.

High-yield statistics you must know

Confidence intervals: the exam-safe approach

• For ratio measures such as RR, OR, or HR, a CI crossing 1 is compatible with no effect
• For difference measures, a CI crossing 0 is compatible with no effect
• A narrow CI suggests precision
• A wide CI suggests imprecision and may be compatible with both clinically important benefit and harm
• A non-significant result does not prove no difference
• A superiority trial that fails to show significance does not prove equivalence or non-inferiority

Absolute versus relative effects

Examiners like papers that overstate benefit using relative measures.

Example:

• Control event rate 2%
• Treatment event rate 1%
• Relative risk reduction 50%
• Absolute risk reduction 1%
• NNT 100

The relative effect sounds dramatic. The absolute benefit is modest. In exam questions, the best answer often highlights this distinction.

Diagnostic test interpretation

Forest plots and meta-analysis basics

• Each study has a point estimate and confidence interval
• The pooled estimate is usually shown as a diamond
• If the pooled CI crosses the line of no effect, the meta-analysis is compatible with no overall effect
• Look for heterogeneity before trusting the pooled result
• Major clinical or methodological heterogeneity may make pooling inappropriate
• A meta-analysis of poor studies does not become strong evidence simply because it is pooled

Management in the Emergency Department

In practice and in the exam, management means how you handle evidence safely.

Immediate approach in the exam

1. Read the question stem before the abstract or table if possible
2. Identify the clinical question and study design
3. Look for the primary outcome, not just any positive result
4. Find the main effect estimate and confidence interval
5. Ask what the biggest validity problem is
6. Decide whether the conclusion is proportionate
7. Check UK applicability before choosing a practice-changing answer

Paper-type-specific appraisal

Randomised controlled trials

Use RCTs mainly for treatment questions.

• Was randomisation truly random?
• Was allocation concealed?
• Were groups similar at baseline?
• Was blinding adequate where feasible?
• Was follow-up complete enough?
• Were patients analysed by intention to treat?
• Was the primary outcome clinically important?
• Was the effect size meaningful, not just statistically significant?

Common traps:

• Focusing on a positive secondary outcome when the primary outcome was neutral
• Treating a non-significant result as proof of no difference
• Ignoring crossover or major loss to follow-up

Diagnostic accuracy studies

These are extremely common in emergency medicine.

• Was there an appropriate reference standard?
• Did all or nearly all patients receive both index test and reference standard?
• Were index test and reference standard interpreted independently?
• Was the population representative of real ED patients?
• Was the threshold pre-specified?
• Were indeterminate results handled properly?

Common biases:

• Verification bias
• Incorporation bias
• Spectrum bias
• Review bias from lack of blinding

Common traps:

• Assuming PPV and NPV will be the same in a different setting
• Accepting a case-control diagnostic study as if it reflects real-world test performance
• Ignoring whether the rule-out miss rate is acceptable in UK practice

Cohort studies

Useful for prognosis, risk factors, and some harms questions.

• Were exposed and unexposed groups comparable?
• Was follow-up adequate and complete?
• Were outcomes measured objectively?
• Were important confounders measured and adjusted for?
• Is the association biologically and clinically plausible?

Common trap: concluding causation from association.

Case-control studies

Often used for rare outcomes.

• Were cases and controls selected appropriately?
• Were they drawn from the same underlying population?
• Was exposure measured similarly in both groups?
• Is recall bias likely?
• Were important confounders addressed?

Common trap: interpreting odds ratio as if it were always the same as relative risk.

Systematic reviews and meta-analyses

• Was the review question focused?
• Was the search strategy adequate?
• Were inclusion criteria appropriate?
• Was study quality assessed?
• Were the included studies sufficiently similar to pool?
• Was heterogeneity explored?
• Is publication bias likely?

Common trap: assuming a pooled result is reliable despite poor included studies or marked heterogeneity.

Clinical prediction rules

These are highly relevant in ED practice.

• Was the rule derived in an appropriate cohort?
• Has it been externally validated?
• Does it show acceptable discrimination and calibration?
• Has it been tested in an impact analysis to show it improves care?
• Does it outperform or safely complement current practice?

Exam rule: derivation is not enough. Validation is better. Impact analysis is strongest.

Before-and-after service studies

Common in operational and pathway papers.

• Were there secular trends over time