ROC and AUC with a Binary Predictor: a Potentially Misleading Metric

In analysis of binary outcomes, the receiver operator characteristic (ROC) curve is heavily used to show the performance of a model or algorithm. The ROC curve is informative about the performance over a series of thresholds and can be summarized by the area under the curve (AUC), a single number. W...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Journal of classification Ročník 37; číslo 3; s. 696 - 708
Hlavní autor: Muschelli, John
Médium: Journal Article
Jazyk:angličtina
Vydáno: New York Springer US 01.10.2020
Témata:
Bioinformatics
Marketing
Mathematics and Statistics
Pattern Recognition
Psychometrics
Signal,Image and Speech Processing
Software Abstract
Statistical Theory and Methods
Statistics
R
Area under the curve
ROC
AUC
ISSN:0176-4268, 1432-1343
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:In analysis of binary outcomes, the receiver operator characteristic (ROC) curve is heavily used to show the performance of a model or algorithm. The ROC curve is informative about the performance over a series of thresholds and can be summarized by the area under the curve (AUC), a single number. When a predictor is categorical, the ROC curve has one less than number of categories as potential thresholds; when the predictor is binary, there is only one threshold. As the AUC may be used in decision-making processes on determining the best model, it important to discuss how it agrees with the intuition from the ROC curve. We discuss how the interpolation of the curve between thresholds with binary predictors can largely change the AUC. Overall, we show using a linear interpolation from the ROC curve with binary predictors corresponds to the estimated AUC, which is most commonly done in software, which we believe can lead to misleading results. We compare R, Python, Stata, and SAS software implementations. We recommend using reporting the interpolation used and discuss the merit of using the step function interpolator, also referred to as the “pessimistic” approach by Fawcett ( 2006 ).
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:0176-4268
1432-1343
DOI:10.1007/s00357-019-09345-1