Area under the ROC Curve has the most consistent evaluation for binary classification

The proper use of model evaluation metrics is important for model evaluation and model selection in binary classification tasks. This study investigates how consistent different metrics are at evaluating models across data of different prevalence while the relationships between different variables a...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:PloS one Jg. 19; H. 12; S. e0316019
1. Verfasser: Li, Jing
Format: Journal Article
Sprache:Englisch
Veröffentlicht: United States Public Library of Science 23.12.2024
Public Library of Science (PLoS)
Schlagworte:
Accuracy
Area Under Curve
Biology and Life Sciences
Classification
Computer and Information Sciences
Data analysis
Data Interpretation, Statistical
Datasets
Evaluation
Evaluation Studies as Topic
Machine Learning
Medicine and Health Sciences
Models, Statistical
Physical Sciences
Ranking
Research and Analysis Methods
ROC Curve
Sample Size
Simulation
Social Sciences
Thresholds
Variance
Variance analysis
United States
ISSN:1932-6203, 1932-6203
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:The proper use of model evaluation metrics is important for model evaluation and model selection in binary classification tasks. This study investigates how consistent different metrics are at evaluating models across data of different prevalence while the relationships between different variables and the sample size are kept constant. Analyzing 156 data scenarios, 18 model evaluation metrics and five commonly used machine learning models as well as a naive random guess model, I find that evaluation metrics that are less influenced by prevalence offer more consistent evaluation of individual models and more consistent ranking of a set of models. In particular, Area Under the ROC Curve (AUC) which takes all decision thresholds into account when evaluating models has the smallest variance in evaluating individual models and smallest variance in ranking of a set of models. A close threshold analysis using all possible thresholds for all metrics further supports the hypothesis that considering all decision thresholds helps reduce the variance in model evaluation with respect to prevalence change in data. The results have significant implications for model evaluation and model selection in binary classification tasks.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
Competing Interests: The authors have declared that no competing interests exist.
ISSN:1932-6203
1932-6203
DOI:10.1371/journal.pone.0316019