ROC and AUC with a Binary Predictor: a Potentially Misleading Metric

In analysis of binary outcomes, the receiver operator characteristic (ROC) curve is heavily used to show the performance of a model or algorithm. The ROC curve is informative about the performance over a series of thresholds and can be summarized by the area under the curve (AUC), a single number. W...

Full description

Saved in:
Bibliographic Details
Published in:Journal of classification Vol. 37; no. 3; pp. 696 - 708
Main Author: Muschelli, John
Format: Journal Article
Language:English
Published: New York Springer US 01.10.2020
Subjects:
Bioinformatics
Marketing
Mathematics and Statistics
Pattern Recognition
Psychometrics
Signal,Image and Speech Processing
Software Abstract
Statistical Theory and Methods
Statistics
R
Area under the curve
ROC
AUC
ISSN:0176-4268, 1432-1343
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In analysis of binary outcomes, the receiver operator characteristic (ROC) curve is heavily used to show the performance of a model or algorithm. The ROC curve is informative about the performance over a series of thresholds and can be summarized by the area under the curve (AUC), a single number. When a predictor is categorical, the ROC curve has one less than number of categories as potential thresholds; when the predictor is binary, there is only one threshold. As the AUC may be used in decision-making processes on determining the best model, it important to discuss how it agrees with the intuition from the ROC curve. We discuss how the interpolation of the curve between thresholds with binary predictors can largely change the AUC. Overall, we show using a linear interpolation from the ROC curve with binary predictors corresponds to the estimated AUC, which is most commonly done in software, which we believe can lead to misleading results. We compare R, Python, Stata, and SAS software implementations. We recommend using reporting the interpolation used and discuss the merit of using the step function interpolator, also referred to as the “pessimistic” approach by Fawcett ( 2006 ).
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:0176-4268
1432-1343
DOI:10.1007/s00357-019-09345-1