Reliable Accuracy Estimates from k-Fold Cross Validation

It is popular to evaluate the performance of classification algorithms by k -fold cross validation. A reliable accuracy estimate will have a relatively small variance, and several studies therefore suggested to repeatedly perform k -fold cross validation. Most of them did not consider the correlatio...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE transactions on knowledge and data engineering Ročník 32; číslo 8; s. 1586 - 1594
Hlavní autoři: Wong, Tzu-Tsung, Yeh, Po-Yang
Médium: Journal Article
Jazyk:angličtina
Vydáno: New York IEEE 01.08.2020
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Témata:
<italic xmlns:ali="http://www.niso.org/schemas/ali/1.0/" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">k -fold cross validation
Accuracy
Algorithms
Classification
Classification algorithms
Correlation
Dependence
Dependency relationship
Estimates
Forests
Journalists
Performance evaluation
Reliability
reliable estimate
replication
Roads
Statistical analysis
Statistical methods
Testing
Urban areas
Variance
Vocabulary development
ISSN:1041-4347, 1558-2191
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:It is popular to evaluate the performance of classification algorithms by k -fold cross validation. A reliable accuracy estimate will have a relatively small variance, and several studies therefore suggested to repeatedly perform k -fold cross validation. Most of them did not consider the correlation among the replications of k -fold cross validation, and hence the variance could be underestimated. The purpose of this study is to explore whether k -fold cross validation should be repeatedly performed for obtaining reliable accuracy estimates. The dependency relationships between the predictions of the same instance in two replications of k -fold cross validation are first analyzed for k -nearest neighbors with <inline-formula><tex-math notation="LaTeX">k= 1</tex-math> <mml:math><mml:mrow><mml:mi>k</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="wong-ieq1-2912815.gif"/> </inline-formula>. Then, statistical methods are proposed to test the strength of the dependency level between the accuracy estimates resulting from two replications of k -fold cross validation. The experimental results on 20 data sets show that the accuracy estimates obtained from various replications of k -fold cross validation are generally highly correlated, and the correlation will be higher as the number of folds increases. The k -fold cross validation with a large number of folds and a small number of replications should be adopted for performance evaluation of classification algorithms.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1041-4347
1558-2191
DOI:10.1109/TKDE.2019.2912815