Calibration & Discrimination
The two distinct facets of a prediction model’s accuracy — ranking patients, and getting the actual risks right.
Discrimination = can the model separate those who will from those who won’t have the event? (ROC / AUC, the c-statistic). Calibration = are the predicted risks themselves correct? (calibration plot; Brier score). A model can discriminate well yet be badly calibrated — systematically over- or under-predicting risk.
Good ranking, wrong numbers: this model over-predicts high risks even if its AUC is fine
Whether the model’s risks are right: predicted probability (x) against the frequency actually observed (y), point by point. The dashed diagonal is perfect agreement. Here the curve sags below the line at the high end — when the model says “high risk” the true event rate is lower, so it over-predicts. A curve above the line would mean under-prediction. Crucially, this can happen while the model still ranks patients well (good AUC).
Discrimination and calibration are read from different things. The AUC (ROC) asks only “are higher-risk patients ranked above lower-risk ones?” — it is unchanged if you shift every predicted risk by the same factor. The calibration plot asks “is a predicted 20% really a 20%?” Always ask for both; a model can ace one and fail the other.
Clinical thresholds run on absolute risk: “treat if 10-year risk >10%” only works if a predicted 10% is true. A miscalibrated model with a great AUC will still tip patients onto the wrong side of a threshold — over-treating, over-admitting, or falsely reassuring — even though it ranks them correctly.
- Discrimination = ranking · ROC /
AUC(c-statistic) - Calibration = risks correct · calibration plot,
Brier - AUC is blind to miscalibration — report both
Risk scores that travel by ranking but need re-calibrating by numbers. Cardiovascular risk equations are the classic case: a model can keep good discrimination (it still ranks high-risk above low-risk patients) yet over- or under-estimate absolute risk when moved to a population with a different baseline event rate — so guidelines recommend re-calibrating or re-fitting the intercept locally before the predicted percentages drive treatment thresholds. A good AUC does not guarantee the numbers are right: check the calibration plot — especially around the risk threshold your decision actually uses.
- Appraising a model on AUC only and never looking at calibration.
- Ignoring calibration drift across settings or over time.
- Forgetting that over/under-prediction can coexist with excellent discrimination.
Quick check
A model has a great AUC but over-predicts risk two-fold — fine to use?
Answer: No — poor calibration misleads thresholds and decisions despite good discrimination. A predicted 20% that’s really 10% will push patients across treat/no-treat cut-offs the wrong way; it needs recalibrating first.
Ready to build your plan? EMF Premium gives you all 40,000+ questions, 20 mocks and 1,215 OSCE stations from £29/month — or a one-off 3- or 6-month pass.