Iโd like to thank Dr. Andrew Vickers for his help with this presentation.
You can visit his page on mskcc.org/profile/andrew-vickers.
This lecture follows the article A simple, step-by-step guide to interpreting decision curve analysis.
Introduction: Motivation behind Decision Curve Analysis and how to draw a Decision Curve.
How to interpret a Decision Curve: Step by Step guide.
Standard software is available to run decision curves in R, SAS and Stata on decisioncurveanalysis.org. Python should be with us soon!
All interactive plots in this presentation were created with rtichoke R package (I am the author ๐), you are also invited to explore rtichoke blog for reproducible examples and some theory.
For ggplot2 outputs dcurves R package is available on CRAN.
Discrimination ๐: Modelโs ability to separate between events and non-events (ROC Curve, AUROC, Sensitivity, Specificity, NPV, PPV, Lift etc).
Calibration โ๏ธ: Agreement between predicted probabilities and the observed outcomes (Calibration Curve, Calibration in the large, Calibration in the small etc).
Problem๐ These metrics are not directly informative to clinical value, nor to full decision analytic approaches.
Solution๐ Decision Curve Analysis calculates a clinical โNet Benefitโ prediction models or diagnostic tests in comparison to default strategies of treating all or no patients.
X axis: pt (Probability Threshold)
Y axis: NB Model=TPNโFPNโpt1โpt
NB Treat None=0
NB Treat All=Prevalenceโ(1 - Prevalence)โpt1โpt
๐
๐คข
๐
๐คจ
๐คข
๐คจ
Probability Threshold is derived from Equivalence between
EVTreat=p(Disease)โUTP+p(NoDisease)โUFP
EVNoTreat=p(Disease)โUFN+p(NoDisease)โUTN
Probability Threshold is derived from Equivalence between
EVTreat=EVNoTreat
p(Disease)โUTP+p(NoDisease)โUFP=p(Disease)โUFN+p(NoDisease)โUTN
Probability Threshold is derived from Equivalence between
EVTreat=EVNoTreat
p(Disease)โUTP+(1โp(Disease))โUFP=p(Disease)โUFN+(1โp(Disease))โUTN
Probability Threshold is derived from Equivalence between
EVTreat=EVNoTreat
p(Disease)โ(UTPโUFN)=(1โp(Disease))โ(UTNโUFP)
Probability Threshold is derived from Equivalence between
EVTreat=EVNoTreat
p(Disease)(1โp(Disease))=(UTNโUFP)(UTPโUFN)
Probability Threshold is derived from Equivalence between
EV๐=EVNo๐
p(๐คข)(1โp(๐คข))=(U๐คจโU๐+๐คจ)(U๐+๐คขโU๐คข)
Sensitivity =TPTP + FN
Specificity=TNTN + FP
Prevalence=TP + FNN
Net Benefit=TPNโFPNโpt1โpt
Sensitivity =TPReal Positives
Specificity=TNReal Negatives
Prevalence=Real PositiveN
Net Benefit=TPNโFPNโpt1โpt
Sensitivity =TPReal Positives
Specificity=TNReal Negatives
Prevalence=Real PositiveN
1 - Prevalence=Real NegativesN
Net Benefit=TPNโFPNโpt1โpt
Sensitivity =TPReal Positives
Specificity=TNReal Negatives
Prevalence=Real PositiveN
1 - Prevalence=Real NegativesN
Net Benefit=TPNโFPNโpt1โpt
Net Benefit=SensitivityโPrevalenceโ(1 - Specificity)โ(1 - Prevalence)โpt1โpt
Sensitivity=TPReal Positives
1โSpecificity=FPReal Negatives
Prevalence=Real PositivesN
1 - Prevalence=Real NegativesN
NetBenefit=TPNโFPNโpt1โpt
NetBenefit=SensitivityโPrevalenceโ(1โSpecificity)โ(1 - Prevalence)โpt1โpt
Sensitivity=TPReal Positives
1โSpecificity=FPReal Negatives
Prevalence=Real PositivesN
1 - Prevalence=Real NegativesN
NetBenefit=TPNโFPNโpt1โpt
NetBenefit=SensitivityโPrevalenceโ(1โSpecificity)โ(1 - Prevalence)โpt1โpt
Net Benefit Treat All=1โPrevalenceโ(1โ0)โ(1 - Prevalence)โpt1โpt
Sensitivity=TPReal Positives
1โSpecificity=FPReal Negatives
Prevalence=Real PositivesN
1 - Prevalence=Real NegativesN
NetBenefit=TPNโFPNโpt1โpt
NetBenefit=SensitivityโPrevalenceโ(1โSpecificity)โ(1 - Prevalence)โpt1โpt
Net Benefit Treat All=Prevalenceโ(1 - Prevalence)โpt1โpt
Sensitivity=TPReal Positives
1โSpecificity=FPReal Negatives
Prevalence=Real PositivesN
1 - Prevalence=Real NegativesN
NetBenefit=TPNโFPNโpt1โpt
NetBenefit=SensitivityโPrevalenceโ(1โSpecificity)โ(1 - Prevalence)โpt1โpt
Net Benefit Treat All=Prevalenceโ(1 - Prevalence)โpt1โpt
Net Benefit Treat None=0โPrevalenceโ(1โ1)โ(1 - Prevalence)โpt1โpt
Sensitivity=TPReal Positives
1โSpecificity=FPReal Negatives
Prevalence=Real PositivesN
1 - Prevalence=Real NegativesN
NetBenefit=TPNโFPNโpt1โpt
NetBenefit=SensitivityโPrevalenceโ(1โSpecificity)โ(1 - Prevalence)โpt1โpt
Net Benefit Treat All=Prevalenceโ(1 - Prevalence)โpt1โpt
Net Benefit Treat None=0
Low Probability Threshold means that Iโm worried about the outcome:
High Probability Threshold means that Iโm worried about the Intervention:
Remember: almost always (specially in Health Care) -
Sensitivity does not have the same importance as Specificity and having 1 TP does not have the same clinical utility as having 1 FP.
A good example for a bad practice in evaluating performance of prediction model is the Youdenโs J statistics:
J=TPTP+FN+TNTN+FPโ1 Maximizing Youdenโs J statistic might lead to a Probability Threshold that will do more harm than good, even if the prediction model is accurate!
Remember: almost always (specially in Health Care) -
Sensitivity does not have the same importance as Specificity and having 1 TP does not have the same clinical utility as having 1 FP.
A good example for a bad practice in evaluating performance of prediction model is the Youdenโs J statistics:
J=Sensโ(1โSpec)
NB=SensโPrevโ(1โSpec)โ(1 - Prev)โpt1โpt
I wouldnโt give more than 4 antibiotics in order to help 1 infected patient.
If a patientโs risk was above 20% I will give him antibiotics, otherwise I wonโt.
The risk of 20% is an odds of 1:4, so in using a threshold probability of 20%, the doctor is telling us โmissing an infected patiant is 4 times worse than giving antibiotics to a healthy patient.โ
pt=11+4=0.2 pt1โpt=0.21โ0.2=14
I will be indifferent ๐
For having 1 TP for 4 FP pt=11+4=0.2 pt1โpt=0.21โ0.2=14
Net Benefit=15โ45โ14=0
I will be sad ๐
For having 1 TP for 5 FP pt=11+4=0.2 pt1โpt=0.21โ0.2=14 Net Benefit=16โ56โ14=โ0.04166โฒ
I will be happy ๐
For having 1 TP for 3 FP pt=11+4=0.2 pt1โpt=0.21โ0.2=14
Net Benefit=14โ34โ14=0.0625
Letโs think in terms of money ๐ธ
A wine importer buys โฌ1m of wine from France and sells it in the USA for $1.5m ๐ท
Net Benefit = Income - Expenditure
Net Benefit = 1.5m$ - 1mโฌ = ? ๐ค
Letโs say that 1โฌ is worth 1.25$
Exchange-Rate = 1.25 ($ / โฌ)
1mโฌ = 1m * 1.25$
Net Benefit = 1.5m$ - 1.25m$ = 250,000$
Which is the equivalent of being given 250,000$:
Net Benefit = 250,000$ - 0$ = 250,000$
๐๐
๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=TPNโFPNโpt1โpt
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=TPNโFPNโpt1โpt
๐๐
๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=TP10โFP10โ14
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=TP10โFP10โ14
๐๐
๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=210โFP10โ14
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=310โFP10โ14
๐๐
๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=210โ010โ14
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=310โ410โ14
๐๐
๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=210โ010โ14=210
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=310โ410โ14=210
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=TPNโFPNโpt1โpt
๐๐๐๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Net Benefit Treat All=TPNโFPNโpt1โpt
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=TP10โFP10โ14
๐๐๐๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Net Benefit Treat All=TP10โFP10โ14
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=310โFP10โ14
๐๐๐๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Net Benefit Treat All=310โFP10โ14
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=310โ410โ14
๐๐๐๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Net Benefit Treat All=310โ710โ14
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=1240โ440=840
๐๐๐๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Net Benefit Treat All=1240โ740=540
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Net Benefit=840
๐๐๐๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Net Benefit Treat All=540
(Net Benefit - Net Benefit All)โ1โptpt=(840โ540)โ4=310
Interventions Avoided (per 100 cases)=310โ100=30
๐๐๐๐๐๐๐
๐ท๐ท๐ท๐ท๐ท๐ท๐ท๐คจ๐คจ๐คจ
Interventions Avoided (per 100 cases)=(TNNโFNNโ1โptpt)โ100
๐๐๐๐๐๐๐
๐ท๐ท๐ท๐ท๐ท๐ท๐ท๐คจ๐คจ๐คจ
Interventions Avoided (per 100 cases)=(310โFN10โ4)โ100
๐๐๐๐๐๐๐
๐ท๐ท๐ท๐ท๐ท๐ท๐ท๐คจ๐คจ๐คจ
Interventions Avoided (per 100 cases)=(310โ010โ4)โ100
๐๐๐๐๐๐๐
๐ท๐ท๐ท๐ท๐ท๐ท๐ท๐คจ๐คจ๐คจ
Interventions Avoided (per 100 cases)=(310โ010โ4)โ100=30
๐๐
๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=TPNโFPNโpt1โpt
๐๐๐๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Net Benefit Treat All=TPNโFPNโpt1โpt
๐๐
๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=TP10โFP10โ14
๐๐๐๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Net Benefit Treat All=TP10โFP10โ14
๐๐
๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=210โFP10โ14
๐๐๐๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Net Benefit Treat All=310โFP10โ14
๐๐
๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=210โ010โ14
๐๐๐๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Net Benefit Treat All=310โ710โ14
๐๐
๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=840โ010โ14=840
๐๐๐๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Net Benefit Treat All=1240โ740=540
๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Net Benefit=840
๐๐๐๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Net Benefit Treat All=540
(Net Benefit - Net Benefit All)โ1โptpt=(840โ540)โ4=310
Interventions Avoided (per 100 cases)=310โ100=30
๐๐
๐ท๐ท๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Interventions Avoided (per 100 cases)=(TNNโFNNโ1โptpt)โ100
๐๐
๐ท๐ท๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Interventions Avoided (per 100 cases)=(710โFN10โ4)โ100
๐๐
๐ท๐ท๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Interventions Avoided (per 100 cases)=(710โ110โ4)โ100
๐๐
๐ท๐ท๐คข๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ๐คจ
Interventions Avoided (per 100 cases)=(710โ110โ4)โ100=30
Invasive Test might add information to the absolute risk with a possible Test Harm in forms of side affects or expenditure of time, effort or money.
I will be indifferent ๐ for having 1 TP for 10 Invasive Tests TestHarm=110=0.1
Net Benefit=TPNโFPNโpt1โptโTestHarm
Invasive Test might add information to the absolute risk with a possible Test Harm in forms of side affects or expenditure of time, effort or money.
I will be indifferent ๐ for having 1 TP for 10 Invasive Tests TestHarm=110=0.1
Net Benefit=TPNโFPNโpt1โptโTestHarm
The same is true for Implementation of Prediction Models ๐ฎ
๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช
๐๐๐
๐คข๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=TPNโFPNโpt1โptโTestHarm
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=TPNโFPNโpt1โpt
๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช
๐๐๐
๐คข๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=TP10โFP10โ14โTestHarm
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=TP10โFP10โ14
๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช
๐๐๐
๐คข๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=TP10โFP10โ14โ110โ1010
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=TP10โFP10โ14
๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช
๐๐๐
๐คข๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=310โFP10โ14โ110โ1010
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=310โFP10โ14
๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช
๐๐๐
๐คข๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=310โ010โ14โ110โ1010
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=310โ410โ14
๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช๐งช
๐๐๐
๐คข๐คข๐คข๐ท๐ท๐ท๐ท๐ท๐ท๐ท
Net Benefit=210
๐๐๐๐๐๐๐
๐คข๐คข๐คข๐คจ๐คจ๐คจ๐คจ๐ท๐ท๐ท
Net Benefit=210