Reliable Accuracy Estimates from k-Fold Cross Validation

It is popular to evaluate the performance of classification algorithms by k -fold cross validation. A reliable accuracy estimate will have a relatively small variance, and several studies therefore suggested to repeatedly perform k -fold cross validation. Most of them did not consider the correlatio...

Full description

Saved in:

Bibliographic Details
Published in:	IEEE transactions on knowledge and data engineering Vol. 32; no. 8; pp. 1586 - 1594
Main Authors:	Wong, Tzu-Tsung, Yeh, Po-Yang
Format:	Journal Article
Language:	English
Published:	New York IEEE 01.08.2020 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:	<italic xmlns:ali="http://www.niso.org/schemas/ali/1.0/" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">k -fold cross validation Accuracy Algorithms Classification Classification algorithms Correlation Dependence Dependency relationship Estimates Forests Journalists Performance evaluation Reliability reliable estimate replication Roads Statistical analysis Statistical methods Testing Urban areas Variance Vocabulary development
ISSN:	1041-4347, 1558-2191
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Abstract	It is popular to evaluate the performance of classification algorithms by k -fold cross validation. A reliable accuracy estimate will have a relatively small variance, and several studies therefore suggested to repeatedly perform k -fold cross validation. Most of them did not consider the correlation among the replications of k -fold cross validation, and hence the variance could be underestimated. The purpose of this study is to explore whether k -fold cross validation should be repeatedly performed for obtaining reliable accuracy estimates. The dependency relationships between the predictions of the same instance in two replications of k -fold cross validation are first analyzed for k -nearest neighbors with <inline-formula><tex-math notation="LaTeX">k= 1</tex-math> <mml:math><mml:mrow><mml:mi>k</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="wong-ieq1-2912815.gif"/> </inline-formula>. Then, statistical methods are proposed to test the strength of the dependency level between the accuracy estimates resulting from two replications of k -fold cross validation. The experimental results on 20 data sets show that the accuracy estimates obtained from various replications of k -fold cross validation are generally highly correlated, and the correlation will be higher as the number of folds increases. The k -fold cross validation with a large number of folds and a small number of replications should be adopted for performance evaluation of classification algorithms.
AbstractList	It is popular to evaluate the performance of classification algorithms by k -fold cross validation. A reliable accuracy estimate will have a relatively small variance, and several studies therefore suggested to repeatedly perform k -fold cross validation. Most of them did not consider the correlation among the replications of k -fold cross validation, and hence the variance could be underestimated. The purpose of this study is to explore whether k -fold cross validation should be repeatedly performed for obtaining reliable accuracy estimates. The dependency relationships between the predictions of the same instance in two replications of k -fold cross validation are first analyzed for k -nearest neighbors with [Formula Omitted]. Then, statistical methods are proposed to test the strength of the dependency level between the accuracy estimates resulting from two replications of k -fold cross validation. The experimental results on 20 data sets show that the accuracy estimates obtained from various replications of k -fold cross validation are generally highly correlated, and the correlation will be higher as the number of folds increases. The k -fold cross validation with a large number of folds and a small number of replications should be adopted for performance evaluation of classification algorithms. It is popular to evaluate the performance of classification algorithms by k -fold cross validation. A reliable accuracy estimate will have a relatively small variance, and several studies therefore suggested to repeatedly perform k -fold cross validation. Most of them did not consider the correlation among the replications of k -fold cross validation, and hence the variance could be underestimated. The purpose of this study is to explore whether k -fold cross validation should be repeatedly performed for obtaining reliable accuracy estimates. The dependency relationships between the predictions of the same instance in two replications of k -fold cross validation are first analyzed for k -nearest neighbors with <inline-formula><tex-math notation="LaTeX">k= 1</tex-math> <mml:math><mml:mrow><mml:mi>k</mml:mi><mml:mo>=</mml:mo><mml:mn>1</mml:mn></mml:mrow></mml:math><inline-graphic xlink:href="wong-ieq1-2912815.gif"/> </inline-formula>. Then, statistical methods are proposed to test the strength of the dependency level between the accuracy estimates resulting from two replications of k -fold cross validation. The experimental results on 20 data sets show that the accuracy estimates obtained from various replications of k -fold cross validation are generally highly correlated, and the correlation will be higher as the number of folds increases. The k -fold cross validation with a large number of folds and a small number of replications should be adopted for performance evaluation of classification algorithms.
Author	Yeh, Po-Yang Wong, Tzu-Tsung
Author_xml	– sequence: 1 givenname: Tzu-Tsung orcidid: 0000-0001-8132-0214 surname: Wong fullname: Wong, Tzu-Tsung email: tzutsung@mail.ncku.edu.tw organization: National Cheng Kung University, Tainan City, Taiwan, ROC – sequence: 2 givenname: Po-Yang surname: Yeh fullname: Yeh, Po-Yang email: zx9430@gmail.com organization: National Cheng Kung University, Tainan City, Taiwan, ROC
BookMark	eNp9kDFPwzAQhS1UJNrCD0AskZhTfHYc22NVWkBUQkIVq-U6tuSSJsV2hv57EloxMDDdDe-7e-9N0KhpG4vQLeAZAJYPm9fH5YxgkDMigQhgF2gMjImcgIRRv-MC8oIW_ApNYtxhjAUXMEbi3dZeb2ubzY3pgjbHbBmT3-tkY-ZCu88-81VbV9kitDFmH7r2lU6-ba7RpdN1tDfnOUWb1XKzeM7Xb08vi_k6N0TSlFdltTVGWqIrokmpS-YqYh3TFFdccmqIk9JJKqiWxhjuuCYMY4qNpEBKOkX3p7OH0H51Nia1a7vQ9B8VKUgfmJWc9ip-UpnBZbBOGZ9-bKagfa0Aq6ElNbSkhpbUuaWehD_kIfTpw_Ff5u7EeGvtr16UUggK9BtNInOl
CODEN	ITKEEH
CitedBy_id	crossref_primary_10_32604_cmc_2024_060837 crossref_primary_10_1016_j_biosystemseng_2025_104276 crossref_primary_10_1111_1541_4337_70082 crossref_primary_10_1002_ntls_20220016 crossref_primary_10_1371_journal_pone_0328880 crossref_primary_10_1111_exsy_13651 crossref_primary_10_1371_journal_pone_0326103 crossref_primary_10_1109_JBHI_2022_3207519 crossref_primary_10_1007_s11030_021_10245_z crossref_primary_10_3389_fenvs_2021_748734 crossref_primary_10_1111_jan_16591 crossref_primary_10_3390_diagnostics13223439 crossref_primary_10_1016_j_compgeo_2022_105094 crossref_primary_10_1007_s00477_025_03091_0 crossref_primary_10_1016_j_soilad_2025_100045 crossref_primary_10_3390_jimaging9100219 crossref_primary_10_1007_s11269_025_04211_9 crossref_primary_10_1016_j_tsep_2024_102882 crossref_primary_10_1109_ACCESS_2024_3470869 crossref_primary_10_1177_03611981251348446 crossref_primary_10_1016_j_eswa_2024_124384 crossref_primary_10_1088_2053_1591_ad87b1 crossref_primary_10_1016_j_engstruct_2024_118184 crossref_primary_10_1016_j_neunet_2025_107226 crossref_primary_10_1061__ASCE_CF_1943_5509_0001745 crossref_primary_10_3390_s22239542 crossref_primary_10_3390_agriculture13091784 crossref_primary_10_1109_ACCESS_2024_3480310 crossref_primary_10_1109_TAES_2024_3456090 crossref_primary_10_1016_j_conbuildmat_2023_134646 crossref_primary_10_3390_math12081267 crossref_primary_10_1007_s13198_022_01756_5 crossref_primary_10_1016_j_jafr_2025_102175 crossref_primary_10_3390_app14031023 crossref_primary_10_1016_j_bspc_2023_105624 crossref_primary_10_1016_j_comnet_2023_109871 crossref_primary_10_3390_app11114945 crossref_primary_10_1007_s42461_022_00678_x crossref_primary_10_1007_s41939_023_00154_z crossref_primary_10_1016_j_jtrangeo_2024_103828 crossref_primary_10_1371_journal_pone_0282429 crossref_primary_10_1007_s10661_023_11283_w crossref_primary_10_1016_j_asoc_2021_107249 crossref_primary_10_3389_fpsyg_2025_1579259 crossref_primary_10_1016_j_buildenv_2025_113287 crossref_primary_10_1016_j_envpol_2023_123105 crossref_primary_10_1016_j_optlastec_2024_112028 crossref_primary_10_1021_acs_energyfuels_5c00792 crossref_primary_10_1007_s41976_025_00242_3 crossref_primary_10_1016_j_future_2021_01_024 crossref_primary_10_1186_s12864_025_11553_6 crossref_primary_10_1016_j_neuroimage_2024_120727 crossref_primary_10_1109_TIM_2025_3544732 crossref_primary_10_1016_j_psep_2024_04_097 crossref_primary_10_32604_jpm_2024_053859 crossref_primary_10_1016_j_jrmge_2024_09_034 crossref_primary_10_3390_en13205464 crossref_primary_10_3390_rs14153547 crossref_primary_10_1007_s13349_025_00972_4 crossref_primary_10_1016_j_seares_2025_102613 crossref_primary_10_3390_bioengineering12060659 crossref_primary_10_1016_j_cstp_2025_101518 crossref_primary_10_1109_LCOMM_2020_2990188 crossref_primary_10_1109_TBME_2024_3509533 crossref_primary_10_1016_j_ultsonch_2024_107032 crossref_primary_10_3390_s24072297 crossref_primary_10_1016_j_bspc_2023_105609 crossref_primary_10_1080_10255842_2025_2553347 crossref_primary_10_1016_j_ijheatmasstransfer_2025_127531 crossref_primary_10_1002_adfm_202214271 crossref_primary_10_1016_j_heliyon_2024_e41084 crossref_primary_10_3847_2041_8213_ad1ceb crossref_primary_10_3390_su16229805 crossref_primary_10_3390_app13095260 crossref_primary_10_3390_app13042658 crossref_primary_10_3390_s24248111 crossref_primary_10_3390_jpm11050353 crossref_primary_10_1007_s13042_024_02133_3 crossref_primary_10_1002_adts_202401489 crossref_primary_10_1016_j_micpro_2020_103615 crossref_primary_10_1061__ASCE_IR_1943_4774_0001727 crossref_primary_10_1177_09544062251349003 crossref_primary_10_1016_j_engappai_2025_111357 crossref_primary_10_1007_s44290_025_00301_0 crossref_primary_10_1038_s41598_024_68651_w crossref_primary_10_3390_app13020773 crossref_primary_10_1007_s12145_024_01521_1 crossref_primary_10_3390_app12115413 crossref_primary_10_1016_j_conbuildmat_2023_133377 crossref_primary_10_1016_j_engappai_2025_111473 crossref_primary_10_1016_j_matbio_2023_03_003 crossref_primary_10_1007_s10712_024_09853_9 crossref_primary_10_1016_j_jobcr_2025_01_012 crossref_primary_10_3348_jksr_2020_0171 crossref_primary_10_1109_ACCESS_2024_3512495 crossref_primary_10_3389_fpsyt_2025_1648585 crossref_primary_10_3390_s22103700 crossref_primary_10_1002_rse2_291 crossref_primary_10_1016_j_jhydrol_2024_131625 crossref_primary_10_1371_journal_pone_0272946 crossref_primary_10_1016_j_dyepig_2022_110647 crossref_primary_10_1109_TIM_2025_3571088 crossref_primary_10_1016_j_jechem_2025_05_029 crossref_primary_10_13005_bpj_3163 crossref_primary_10_1016_j_csbj_2023_09_025 crossref_primary_10_1016_j_eswa_2023_122731 crossref_primary_10_3390_drones7080493 crossref_primary_10_1111_prd_70000 crossref_primary_10_1007_s41939_023_00315_0 crossref_primary_10_1016_j_atmosres_2024_107576 crossref_primary_10_1007_s41365_025_01684_6 crossref_primary_10_1016_j_jobe_2024_111248 crossref_primary_10_1177_15333175251339730 crossref_primary_10_1007_s00603_024_04287_6 crossref_primary_10_1016_j_fuel_2024_133323 crossref_primary_10_1016_j_tust_2025_106448 crossref_primary_10_1177_10591478251331143 crossref_primary_10_1111_exsy_12786 crossref_primary_10_1016_j_autcon_2024_105489 crossref_primary_10_1016_j_eswa_2023_121608 crossref_primary_10_1016_j_cej_2023_142619 crossref_primary_10_1177_01445987251318353 crossref_primary_10_1016_j_rineng_2025_105287 crossref_primary_10_3390_agronomy15010077 crossref_primary_10_1016_j_marpolbul_2025_118477 crossref_primary_10_1016_j_ecoenv_2025_119003 crossref_primary_10_1016_j_enbuild_2025_115991 crossref_primary_10_1109_JSEN_2020_2990864 crossref_primary_10_1109_ACCESS_2025_3539309 crossref_primary_10_1093_jamia_ocaf118 crossref_primary_10_1016_j_jobe_2025_113717 crossref_primary_10_1007_s44290_025_00204_0 crossref_primary_10_1007_s11696_024_03715_9 crossref_primary_10_1016_j_measurement_2022_111207 crossref_primary_10_1109_ACCESS_2020_2986232 crossref_primary_10_3390_plants14010083 crossref_primary_10_1016_j_fsidi_2024_301812 crossref_primary_10_1109_ACCESS_2021_3134330 crossref_primary_10_1109_TGRS_2025_3550554 crossref_primary_10_1016_j_jclepro_2023_138925 crossref_primary_10_3390_jintelligence12110111 crossref_primary_10_1002_cjce_25102 crossref_primary_10_1016_j_mtcomm_2025_113801 crossref_primary_10_1109_JAS_2024_124359 crossref_primary_10_1016_j_mtcomm_2024_109372 crossref_primary_10_3390_w17121776 crossref_primary_10_1016_j_sca_2025_100128 crossref_primary_10_3389_frai_2024_1467051 crossref_primary_10_3390_healthcare11070929 crossref_primary_10_1109_JSEN_2024_3408323 crossref_primary_10_1007_s42979_024_03622_6 crossref_primary_10_1371_journal_pone_0324204 crossref_primary_10_1016_j_csbj_2023_01_028 crossref_primary_10_1016_j_ifset_2021_102796 crossref_primary_10_3390_math10203771 crossref_primary_10_1109_LSP_2023_3337727 crossref_primary_10_1016_j_ssci_2024_106708 crossref_primary_10_3390_risks12040062 crossref_primary_10_3390_rs16132424 crossref_primary_10_1109_TUFFC_2024_3433407 crossref_primary_10_1016_j_ecoinf_2023_102084 crossref_primary_10_1016_j_compeleceng_2025_110117 crossref_primary_10_7235_HORT_20240022 crossref_primary_10_1016_j_rser_2023_113486 crossref_primary_10_1007_s40996_022_01015_4 crossref_primary_10_7717_peerj_cs_2516 crossref_primary_10_1038_s41598_021_03867_8 crossref_primary_10_1080_21655979_2023_2243416 crossref_primary_10_1109_TII_2020_2971014 crossref_primary_10_1371_journal_pone_0316830 crossref_primary_10_1016_j_agwat_2023_108416 crossref_primary_10_3390_rs17050797 crossref_primary_10_1016_j_asoc_2023_109987 crossref_primary_10_1002_ps_7788 crossref_primary_10_1016_j_measurement_2024_115270 crossref_primary_10_3390_w15223970 crossref_primary_10_3390_app14135660 crossref_primary_10_1016_j_compind_2023_103883 crossref_primary_10_1186_s42834_024_00225_x crossref_primary_10_1016_j_irfa_2023_103028 crossref_primary_10_3390_math12162542 crossref_primary_10_3390_app12178765 crossref_primary_10_3390_s23146613 crossref_primary_10_1109_ACCESS_2025_3590871 crossref_primary_10_1007_s41870_021_00707_w crossref_primary_10_3390_jmse12122267 crossref_primary_10_1016_j_tre_2024_103647 crossref_primary_10_1007_s00521_024_10147_9 crossref_primary_10_3390_cancers15184636 crossref_primary_10_3390_machines11121038 crossref_primary_10_1016_j_tafmec_2025_105129 crossref_primary_10_3390_electronics14101948 crossref_primary_10_1016_j_advwatres_2023_104569 crossref_primary_10_1016_j_aei_2025_103288 crossref_primary_10_1002_mp_17411 crossref_primary_10_3390_a18080465 crossref_primary_10_1088_1361_6501_ad3bdd crossref_primary_10_1016_j_jfoodeng_2025_112492 crossref_primary_10_3390_computers10090112 crossref_primary_10_1016_j_jhydrol_2022_128956 crossref_primary_10_3390_jpm11080727 crossref_primary_10_1007_s11042_025_21058_9 crossref_primary_10_1016_j_inffus_2025_103122 crossref_primary_10_1093_geronb_gbad061 crossref_primary_10_1016_j_landusepol_2022_106409 crossref_primary_10_1016_j_chaos_2024_114908 crossref_primary_10_1038_s41928_024_01134_y crossref_primary_10_1080_15389588_2025_2527849 crossref_primary_10_1016_j_engappai_2023_105991 crossref_primary_10_1016_j_eswa_2024_123934 crossref_primary_10_1016_j_cie_2022_108559 crossref_primary_10_3390_en15061966 crossref_primary_10_3389_fpls_2021_616689 crossref_primary_10_3390_w17131988 crossref_primary_10_1016_j_jag_2024_104013 crossref_primary_10_1007_s00432_023_05467_7 crossref_primary_10_1007_s11270_021_05036_z crossref_primary_10_1016_j_compbiomed_2021_105065 crossref_primary_10_1016_j_compag_2025_110680 crossref_primary_10_2166_ws_2023_248 crossref_primary_10_1016_j_ijplas_2023_103646 crossref_primary_10_1007_s42107_023_00698_y crossref_primary_10_1177_03611981231212162 crossref_primary_10_3390_app13074430 crossref_primary_10_3390_app12189321 crossref_primary_10_3390_agronomy15020492 crossref_primary_10_1007_s13178_022_00768_x crossref_primary_10_3390_data8120179 crossref_primary_10_1016_j_conbuildmat_2024_137619 crossref_primary_10_1016_j_engfailanal_2022_106647 crossref_primary_10_1016_j_ejor_2024_03_008 crossref_primary_10_1016_j_asr_2024_12_056 crossref_primary_10_1016_j_applanim_2021_105491 crossref_primary_10_3390_jpm11090893 crossref_primary_10_3390_e24030335 crossref_primary_10_3390_foods14111987 crossref_primary_10_1007_s11709_021_0785_x crossref_primary_10_1186_s12911_022_02026_x crossref_primary_10_1016_j_cscm_2025_e05303 crossref_primary_10_1186_s13638_020_01818_x crossref_primary_10_1016_j_seppur_2025_134691 crossref_primary_10_1016_j_jece_2025_115946 crossref_primary_10_1016_j_chemosphere_2024_143781 crossref_primary_10_1080_24725854_2024_2302368 crossref_primary_10_1155_2021_9304925 crossref_primary_10_1016_j_est_2023_107063 crossref_primary_10_1016_j_jenvman_2025_125688 crossref_primary_10_1016_j_envpol_2025_126849 crossref_primary_10_1016_j_indcrop_2024_119934 crossref_primary_10_3390_s24216815 crossref_primary_10_1007_s42979_024_02828_y crossref_primary_10_1016_j_ress_2023_109141 crossref_primary_10_3390_app131810157 crossref_primary_10_1016_j_engappai_2022_105445 crossref_primary_10_1007_s11663_024_03191_2 crossref_primary_10_1001_jamanetworkopen_2024_50260 crossref_primary_10_3390_rs15102626 crossref_primary_10_1007_s00521_021_06440_6 crossref_primary_10_3390_s22093234 crossref_primary_10_1016_j_desal_2025_119041 crossref_primary_10_3390_f13091350 crossref_primary_10_1016_j_infrared_2023_104653 crossref_primary_10_1016_j_jbi_2023_104581 crossref_primary_10_1007_s13369_024_09034_1 crossref_primary_10_3390_a17020078 crossref_primary_10_1016_j_buildenv_2023_110047 crossref_primary_10_1016_j_compag_2024_109062 crossref_primary_10_3390_buildings14092894 crossref_primary_10_1016_j_identj_2025_100883 crossref_primary_10_1016_j_future_2022_08_007 crossref_primary_10_1007_s41348_023_00784_y crossref_primary_10_1016_j_agrformet_2022_109275 crossref_primary_10_1007_s13206_024_00186_8 crossref_primary_10_1016_j_asoc_2022_109266 crossref_primary_10_1016_j_engappai_2025_111425 crossref_primary_10_1109_JSEN_2025_3545035 crossref_primary_10_3390_healthcare12242527 crossref_primary_10_1016_j_jvs_2025_06_023 crossref_primary_10_1088_1742_6596_2835_1_012011 crossref_primary_10_1177_10775463241276645 crossref_primary_10_1061_AJRUA6_RUENG_1632 crossref_primary_10_1111_aogs_14623 crossref_primary_10_2118_217452_PA crossref_primary_10_1016_j_mtcomm_2025_112282 crossref_primary_10_1108_SASBE_12_2024_0558 crossref_primary_10_1016_j_compbiomed_2023_107723 crossref_primary_10_1007_s42114_024_01113_z crossref_primary_10_1016_j_cej_2024_156687 crossref_primary_10_1016_j_ress_2025_110885 crossref_primary_10_1007_s10726_025_09920_5 crossref_primary_10_1016_j_eswa_2025_128461 crossref_primary_10_1016_j_jobe_2023_106263 crossref_primary_10_1016_j_aiepr_2025_06_002 crossref_primary_10_1016_j_snb_2024_136924 crossref_primary_10_1149_1945_7111_adeb2c crossref_primary_10_1002_admt_202301316 crossref_primary_10_1016_j_ijhydene_2024_08_257 crossref_primary_10_1016_j_jhazmat_2024_133560 crossref_primary_10_3390_make5030057 crossref_primary_10_1016_j_ejrh_2025_102651 crossref_primary_10_1016_j_compbiomed_2023_106649 crossref_primary_10_1016_j_oceaneng_2025_121589 crossref_primary_10_1016_j_wasman_2024_12_034 crossref_primary_10_1177_21582440241266370 crossref_primary_10_1016_j_ecolind_2025_113572 crossref_primary_10_1038_s43247_025_02624_3 crossref_primary_10_1039_D4EW00111G crossref_primary_10_7717_peerj_cs_2474 crossref_primary_10_1038_s41545_024_00361_2 crossref_primary_10_1007_s11356_023_30443_6 crossref_primary_10_1111_ijfs_16915 crossref_primary_10_3390_diagnostics12061396 crossref_primary_10_1007_s11042_024_18280_2 crossref_primary_10_1108_ACMM_12_2023_2935 crossref_primary_10_1109_ACCESS_2020_3032173 crossref_primary_10_1016_j_ijdrr_2025_105204 crossref_primary_10_1061__ASCE_SU_1943_5428_0000411 crossref_primary_10_1016_j_engstruct_2025_121175 crossref_primary_10_1109_ACCESS_2022_3170897 crossref_primary_10_1016_j_childyouth_2024_107644 crossref_primary_10_3389_fbioe_2022_1097363 crossref_primary_10_1002_smll_202204719 crossref_primary_10_1016_j_ymeth_2022_08_002 crossref_primary_10_1109_TVLSI_2025_3530956 crossref_primary_10_1016_j_pss_2024_105894 crossref_primary_10_1016_j_scico_2024_103140 crossref_primary_10_1016_j_ijnonlinmec_2024_104857 crossref_primary_10_1016_j_jnoncrysol_2023_122693 crossref_primary_10_1016_j_powtec_2023_118480 crossref_primary_10_1145_3733051 crossref_primary_10_1007_s42979_023_02208_y crossref_primary_10_3390_f16010065 crossref_primary_10_1016_j_scitotenv_2024_171556 crossref_primary_10_1016_j_undsp_2023_01_006 crossref_primary_10_1007_s41870_020_00592_9 crossref_primary_10_1109_ACCESS_2024_3442979 crossref_primary_10_1016_j_ijepes_2023_109579 crossref_primary_10_1109_TIM_2024_3375980 crossref_primary_10_1038_s41598_023_50164_7 crossref_primary_10_3390_su151914357 crossref_primary_10_3390_electronics13244905 crossref_primary_10_1038_s41598_025_12129_w crossref_primary_10_3390_s21196412 crossref_primary_10_1007_s11227_023_05817_9 crossref_primary_10_1038_s41598_024_54990_1 crossref_primary_10_1016_j_saa_2024_125608 crossref_primary_10_1016_j_ejim_2025_02_009 crossref_primary_10_1007_s10115_025_02533_z crossref_primary_10_3390_f14030616 crossref_primary_10_3390_su16145901 crossref_primary_10_1016_j_bspc_2025_107987 crossref_primary_10_1016_j_cie_2024_110590 crossref_primary_10_1063_5_0136830 crossref_primary_10_1109_ACCESS_2023_3326075 crossref_primary_10_1177_10790632231200838 crossref_primary_10_1093_jigpal_jzae020 crossref_primary_10_3390_s23239550 crossref_primary_10_1016_j_apenergy_2024_123673 crossref_primary_10_1029_2022WR032395 crossref_primary_10_1016_j_ress_2022_108753 crossref_primary_10_1016_j_socscimed_2025_117690 crossref_primary_10_1007_s12145_024_01582_2 crossref_primary_10_3390_math11143204 crossref_primary_10_1134_S0040579523070102 crossref_primary_10_3390_ijms26178423 crossref_primary_10_1016_j_ndteint_2022_102782 crossref_primary_10_1186_s12871_025_02987_2 crossref_primary_10_1016_j_enbenv_2025_02_001 crossref_primary_10_3390_axioms11080374 crossref_primary_10_3390_rs13224577 crossref_primary_10_1016_j_autcon_2023_104767 crossref_primary_10_3390_agronomy13051277 crossref_primary_10_1016_j_oceaneng_2023_115271 crossref_primary_10_1016_j_jhydrol_2025_133506 crossref_primary_10_1016_j_jmapro_2025_07_083 crossref_primary_10_3390_su17104287 crossref_primary_10_1007_s44291_025_00054_1 crossref_primary_10_3390_s21248378 crossref_primary_10_1016_j_vibspec_2025_103816 crossref_primary_10_1007_s11468_025_02781_3 crossref_primary_10_1016_j_bonr_2024_101821 crossref_primary_10_3390_app15169209 crossref_primary_10_3390_s25175380 crossref_primary_10_1016_j_jenvman_2024_121295 crossref_primary_10_1007_s40747_022_00938_9 crossref_primary_10_1016_j_enbuild_2025_116251 crossref_primary_10_1016_j_energy_2024_133863 crossref_primary_10_1016_j_saa_2025_126934 crossref_primary_10_1016_j_neuri_2021_100034 crossref_primary_10_1016_j_rineng_2025_105715 crossref_primary_10_1016_j_asoc_2024_112088 crossref_primary_10_1002_ece3_70003 crossref_primary_10_3389_fpubh_2024_1414209 crossref_primary_10_3390_coatings13061060 crossref_primary_10_1016_j_ress_2024_110148 crossref_primary_10_1016_j_jss_2024_112109 crossref_primary_10_1177_00405175231162643 crossref_primary_10_1016_j_epsr_2023_109486 crossref_primary_10_1016_j_seizure_2024_03_013 crossref_primary_10_1088_1741_2552_ad5b19 crossref_primary_10_1016_j_eswa_2024_124871 crossref_primary_10_1016_j_imavis_2024_105081 crossref_primary_10_3758_s13428_025_02718_y crossref_primary_10_1007_s11069_025_07317_w crossref_primary_10_1016_j_psep_2025_107594 crossref_primary_10_1155_2024_4616609 crossref_primary_10_1016_j_jallcom_2023_172664 crossref_primary_10_1177_03019233251356229 crossref_primary_10_1109_TIM_2025_3597675 crossref_primary_10_1016_j_ress_2025_111309 crossref_primary_10_1016_j_cma_2024_116812 crossref_primary_10_1007_s42979_023_01840_y crossref_primary_10_1016_j_rse_2024_114026 crossref_primary_10_3390_rs17121974 crossref_primary_10_1080_10837450_2023_2231074 crossref_primary_10_3390_jpm12050768 crossref_primary_10_1016_j_health_2022_100121 crossref_primary_10_3390_a17030108 crossref_primary_10_3390_polym15051324 crossref_primary_10_1016_j_asoc_2024_112185 crossref_primary_10_1088_1361_6501_abc6e3 crossref_primary_10_1016_j_jgsce_2025_205782 crossref_primary_10_1016_j_jlumin_2025_121388 crossref_primary_10_1007_s42243_024_01198_2 crossref_primary_10_1007_s41939_023_00220_6 crossref_primary_10_3390_a16040182 crossref_primary_10_3390_jimaging10060131 crossref_primary_10_1016_j_eswa_2024_124780 crossref_primary_10_1109_ACCESS_2024_3457922 crossref_primary_10_3390_mi13081352 crossref_primary_10_1109_ACCESS_2025_3584890 crossref_primary_10_3390_land13111903 crossref_primary_10_1007_s11042_023_16423_5 crossref_primary_10_3390_app13053125 crossref_primary_10_3390_s25144264 crossref_primary_10_1007_s13369_024_09284_z crossref_primary_10_3390_app9183723 crossref_primary_10_1016_j_istruc_2024_107649 crossref_primary_10_1097_j_jcrs_0000000000001419 crossref_primary_10_1039_D5AY00304K crossref_primary_10_1117_1_NPh_9_2_025003 crossref_primary_10_1016_j_compstruc_2023_107114 crossref_primary_10_1016_j_jss_2025_112459 crossref_primary_10_1016_j_psep_2024_05_036 crossref_primary_10_3390_biomimetics10090567 crossref_primary_10_1007_s12040_023_02217_8 crossref_primary_10_3390_s22176709 crossref_primary_10_1007_s10479_022_04531_8 crossref_primary_10_1007_s40430_025_05458_4 crossref_primary_10_1016_j_pnmrs_2025_101562 crossref_primary_10_1007_s13246_022_01106_6 crossref_primary_10_1038_s41529_025_00638_y crossref_primary_10_1038_s41598_025_07450_3 crossref_primary_10_1016_j_jenvman_2025_127132 crossref_primary_10_1002_minf_202400193 crossref_primary_10_1007_s10895_023_03499_3 crossref_primary_10_1016_j_surfcoat_2025_132494 crossref_primary_10_1016_j_xjmad_2025_100116 crossref_primary_10_1049_tje2_12362 crossref_primary_10_1093_bib_bbae380 crossref_primary_10_3390_e27010083 crossref_primary_10_1007_s10044_024_01369_7 crossref_primary_10_3390_f16050783 crossref_primary_10_3390_su17136193 crossref_primary_10_1080_15599612_2023_2225573 crossref_primary_10_1016_j_habitatint_2025_103518 crossref_primary_10_1109_TED_2024_3474618 crossref_primary_10_1016_j_jafr_2024_101605 crossref_primary_10_1021_acs_analchem_5c03126 crossref_primary_10_1007_s11416_021_00385_z crossref_primary_10_1016_j_gsme_2025_09_006 crossref_primary_10_1016_j_tust_2024_105802 crossref_primary_10_1038_s41598_025_12026_2 crossref_primary_10_1016_j_jhazmat_2022_128807 crossref_primary_10_1186_s40359_025_02768_z crossref_primary_10_1007_s12265_024_10534_6 crossref_primary_10_1007_s12652_025_04978_0 crossref_primary_10_1007_s40747_024_01399_y crossref_primary_10_1016_j_ast_2025_110251 crossref_primary_10_1038_s41598_025_09957_1 crossref_primary_10_1063_5_0142198 crossref_primary_10_1007_s40996_023_01138_2 crossref_primary_10_1049_ell2_13086 crossref_primary_10_1007_s11116_023_10447_4 crossref_primary_10_1016_j_csite_2024_105048 crossref_primary_10_2478_remav_2025_0001 crossref_primary_10_1371_journal_pone_0297615 crossref_primary_10_1007_s10489_022_03314_9 crossref_primary_10_1016_j_irfa_2023_102953 crossref_primary_10_1109_TPWRS_2024_3443105 crossref_primary_10_3390_rs14153536 crossref_primary_10_3390_f15122136 crossref_primary_10_1080_00102202_2025_2540313 crossref_primary_10_3389_fdata_2024_1402384 crossref_primary_10_1002_int_22471 crossref_primary_10_1016_j_ijsolstr_2025_113637 crossref_primary_10_1007_s10207_023_00684_0 crossref_primary_10_3390_rs15184534 crossref_primary_10_3390_app15010352 crossref_primary_10_1038_s41598_024_83902_6 crossref_primary_10_3390_s21010002 crossref_primary_10_1016_j_imed_2024_01_001 crossref_primary_10_1016_j_microc_2024_111396 crossref_primary_10_1177_01445987251365291 crossref_primary_10_3390_rs17010105 crossref_primary_10_1016_j_inffus_2023_102023 crossref_primary_10_1093_bib_bbae674 crossref_primary_10_3390_su142416692 crossref_primary_10_3390_agriengineering6040233 crossref_primary_10_1007_s11356_024_32158_8 crossref_primary_10_1007_s40846_023_00777_0 crossref_primary_10_1086_733509 crossref_primary_10_1109_TIM_2022_3165740 crossref_primary_10_3390_su17093804 crossref_primary_10_1007_s10661_025_14461_0 crossref_primary_10_1016_j_ress_2023_109736 crossref_primary_10_12688_f1000research_75469_1 crossref_primary_10_3390_computation10070104 crossref_primary_10_1088_1361_6668_ace3fb crossref_primary_10_1007_s11760_021_01907_4 crossref_primary_10_3390_agriculture15010071 crossref_primary_10_12688_f1000research_75469_2 crossref_primary_10_3390_app112412135 crossref_primary_10_1111_nph_19955 crossref_primary_10_1371_journal_pone_0292185 crossref_primary_10_3390_s24041159 crossref_primary_10_1016_j_imu_2020_100508 crossref_primary_10_1016_j_jag_2024_103986 crossref_primary_10_3390_su142215180 crossref_primary_10_3389_fpubh_2022_959667 crossref_primary_10_1016_j_heliyon_2024_e40748 crossref_primary_10_3390_app15116312 crossref_primary_10_1002_ente_202300735 crossref_primary_10_1016_j_ocemod_2025_102601 crossref_primary_10_3390_diagnostics12092132 crossref_primary_10_1016_j_cclet_2024_109870 crossref_primary_10_1371_journal_pone_0276814 crossref_primary_10_1007_s42417_022_00780_w crossref_primary_10_1016_j_envpol_2023_121484 crossref_primary_10_1007_s00542_024_05776_y crossref_primary_10_1007_s40964_025_01276_1 crossref_primary_10_1016_j_apenergy_2023_121439 crossref_primary_10_1063_5_0277974 crossref_primary_10_1016_j_envres_2025_121916 crossref_primary_10_32604_cmc_2021_016907 crossref_primary_10_1016_j_inffus_2024_102424 crossref_primary_10_1109_TIA_2023_3234935 crossref_primary_10_1007_s00170_024_13311_6 crossref_primary_10_1109_JSEN_2025_3596131 crossref_primary_10_1038_s41598_025_18172_x crossref_primary_10_1007_s13198_025_02951_w crossref_primary_10_3390_brainsci11070902 crossref_primary_10_1038_s41598_025_13634_8 crossref_primary_10_1080_10298436_2023_2270765 crossref_primary_10_1016_j_heliyon_2023_e21828 crossref_primary_10_3390_agriengineering5040145 crossref_primary_10_1016_j_coal_2023_104294 crossref_primary_10_1049_gtd2_12610 crossref_primary_10_1016_j_cie_2025_110967 crossref_primary_10_1021_acssensors_5c01439
Cites_doi	10.1016/j.patrec.2015.08.002 10.1162/089976699300016007 10.1016/j.patcog.2015.03.009 10.1109/TPAMI.2009.187 10.1162/NECO_a_00532 10.1109/TNNLS.2012.2199516 10.1214/aos/1032181158 10.1162/089976698300017197 10.1016/j.csda.2009.04.009 10.1109/TKDE.2017.2740926
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2020
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2020
DBID	97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
DOI	10.1109/TKDE.2019.2912815
DatabaseName	IEEE All-Society Periodicals Package (ASPP) 2005-present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional
DatabaseTitle	CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional
DatabaseTitleList	Technology Research Database
Database_xml	– sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering Computer Science
EISSN	1558-2191
EndPage	1594
ExternalDocumentID	10_1109_TKDE_2019_2912815 8698831
Genre	orig-research
GrantInformation_xml	– fundername: Ministry of Science and Technology, Taiwan grantid: 106-2410-H-006-020 funderid: 10.13039/501100004663
GroupedDBID	-~X .DC 0R~ 29I 4.4 5GY 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACIWK AENEX AGQYO AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 EBS EJD F5P HZ~ IEDLZ IFIPE IPLJI JAVBF LAI M43 MS~ O9- OCL P2P PQQKQ RIA RIE RNS RXW TAE TN5 UHB AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
ID	FETCH-LOGICAL-c293t-d6dbcc9e2ad2a26a65fd2ef5a30d7973c2f99f9383a9ccc7f7a250030c931263
IEDL.DBID	RIE
ISICitedReferencesCount	675
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000546878300011&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	1041-4347
IngestDate	Mon Jun 30 04:50:31 EDT 2025 Sat Nov 29 04:46:48 EST 2025 Tue Nov 18 22:28:51 EST 2025 Wed Aug 27 02:36:38 EDT 2025
IsPeerReviewed	true
IsScholarly	true
Issue	8
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c293t-d6dbcc9e2ad2a26a65fd2ef5a30d7973c2f99f9383a9ccc7f7a250030c931263
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0001-8132-0214
PQID	2422015673
PQPubID	85438
PageCount	9
ParticipantIDs	proquest_journals_2422015673 crossref_citationtrail_10_1109_TKDE_2019_2912815 crossref_primary_10_1109_TKDE_2019_2912815 ieee_primary_8698831
PublicationCentury	2000
PublicationDate	2020-08-01
PublicationDateYYYYMMDD	2020-08-01
PublicationDate_xml	– month: 08 year: 2020 text: 2020-08-01 day: 01
PublicationDecade	2020
PublicationPlace	New York
PublicationPlace_xml	– name: New York
PublicationTitle	IEEE transactions on knowledge and data engineering
PublicationTitleAbbrev	TKDE
PublicationYear	2020
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	bengio (ref2) 2004; 5 ref12 witten (ref3) 2011 kohavi (ref13) 1995 ref15 ref14 vanwinckelen (ref5) 2012 demšar (ref7) 2006; 7 ref10 lichman (ref17) 2013 ref1 ref18 ref8 wang (ref11) 2014; 26 cramer (ref16) 1946 ref9 ref6 bouckaert (ref4) 2003
References_xml	– ident: ref12 doi: 10.1016/j.patrec.2015.08.002 – ident: ref10 doi: 10.1162/089976699300016007 – year: 2011 ident: ref3 publication-title: Data Mining Practical Machine Learning Tools and Techniques – start-page: 39 year: 2012 ident: ref5 article-title: On estimating model accuracy with repeated cross-validation publication-title: Proc Belgian-Dutch Conf Mach Learn – start-page: 1137 year: 1995 ident: ref13 article-title: A study of cross-validation and bootstrap for accuracy estimation and model selection publication-title: Proc Int Joint Conf Artif Intell – ident: ref8 doi: 10.1016/j.patcog.2015.03.009 – year: 1946 ident: ref16 publication-title: Mathematical Methods of Statistics – volume: 7 start-page: 1 year: 2006 ident: ref7 article-title: Statistical comparisons of classifiers over multiple data sets publication-title: J Mach Learn Res – ident: ref14 doi: 10.1109/TPAMI.2009.187 – volume: 26 start-page: 208 year: 2014 ident: ref11 article-title: Blocked 3×2 cross-validated t-Test for comparing supervised classification learning algorithms publication-title: Neural Comput doi: 10.1162/NECO_a_00532 – ident: ref18 doi: 10.1109/TNNLS.2012.2199516 – ident: ref1 doi: 10.1214/aos/1032181158 – ident: ref9 doi: 10.1162/089976698300017197 – ident: ref6 doi: 10.1016/j.csda.2009.04.009 – year: 2013 ident: ref17 – ident: ref15 doi: 10.1109/TKDE.2017.2740926 – volume: 5 start-page: 1089 year: 2004 ident: ref2 article-title: No unbiased estimator of the variance of k-fold cross-validation publication-title: J Mach Learn Res – start-page: 51 year: 2003 ident: ref4 article-title: Choosing between two learning algorithms based on calibrated tests publication-title: Proc 20th Int Conf Mach Learn
SSID	ssj0008781
Score	2.717646
Snippet	It is popular to evaluate the performance of classification algorithms by k -fold cross validation. A reliable accuracy estimate will have a relatively small...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	1586
SubjectTerms	<italic xmlns:ali="http://www.niso.org/schemas/ali/1.0/" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">k -fold cross validation Accuracy Algorithms Classification Classification algorithms Correlation Dependence Dependency relationship Estimates Forests Journalists Performance evaluation Reliability reliable estimate replication Roads Statistical analysis Statistical methods Testing Urban areas Variance Vocabulary development
Title	Reliable Accuracy Estimates from k-Fold Cross Validation
URI	https://ieeexplore.ieee.org/document/8698831 https://www.proquest.com/docview/2422015673
Volume	32
WOSCitedRecordID	wos000546878300011&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1558-2191 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0008781 issn: 1041-4347 databaseCode: RIE dateStart: 19890101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB5UPOjBt1hf5OBJ3Hab7G4yx1JbBEE8FPG2xNkERG2ltoL_3ky6LYoieNtDAmFmJzOTmW8-gDPpLDrjgiF5rZKMjE_QZSap2mQyF1IioymSTeibG3N_j7dLcLHAwjjnYvOZa_JnrOVXI5ryU1nLFGgMg6aXtdYzrNbi1jU6EpKG7CLkRCrTdQWznWJrcH3Z4yYubErkwlH-zQdFUpUfN3F0L_3N_x1sCzbqMFJ0ZnrfhiU33IHNOUWDqC12B9a_zBvcBcMNyIyVEh2i6djSh-gFE3_heFMw0EQ8Jf3RcyW6fG5xF2L0GeXSHgz6vUH3KqmpExIK_nuSME0UETppK2llYYvcV9L53Kq00qgVSY_oMejCIhFpr22IhYLBE6q2LNQ-rAxHQ3cAwktboFfGU2UzLdUDylyFKNAW3uVUpA1I57IsqR4rzuwWz2VML1IsWfwli7-sxd-A88WW19lMjb8W77K8FwtrUTfgeK6wsra6tzKEG5Kh4Vod_r7rCNYk58uxge8YVibjqTuBVXqfPL6NT-MP9QmS08ca
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1RTxsxDLYqQBo8UFZAFBjLw56mHVyTu0v8iFirorJqD9XUt1PmSyQEtKi0SPx74vRagTZN2ts9JFJkn2M79ucP4It0Fp1xwZC8VklGxifoMpNUHTKZCymR0RTJJvRwaMZj_NmAb2ssjHMuNp-5c_6MtfxqSgt-KrswBRrDoOnNPMtkZ4nWWt-7RkdK0pBfhKxIZbquYXZSvBgNvne5jQvPJXLpKH_nhSKtyh93cXQwveb_HW0PdutAUlwuNf8RGm7SguaKpEHUNtuCnTcTB_fBcAsyo6XEJdFiZulFdIORP3DEKRhqIu6S3vS-Eld8bvErROlL0qUDGPW6o6t-UpMnJBQ8-DxhoigidNJW0srCFrmvpPO5VWmlUSuSHtFj0IZFItJe2xANBZMnVB1ZqEPYmEwn7giEl7ZAr4ynymZaqt8ocxXiQFt4l1ORtiFdybKkerA481vclzHBSLFk8Zcs_rIWfxu-rrc8Lqdq_GvxPst7vbAWdRtOVwora7t7KkPAIRkcrtXx33d9hg_90Y-b8uZ6ODiBbcnZc2znO4WN-WzhPsEWPc9vn2Zn8ed6BRrJymE
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Reliable+Accuracy+Estimates+from+k+-Fold+Cross+Validation&rft.jtitle=IEEE+transactions+on+knowledge+and+data+engineering&rft.au=Wong%2C+Tzu-Tsung&rft.au=Yeh%2C+Po-Yang&rft.date=2020-08-01&rft.issn=1041-4347&rft.eissn=1558-2191&rft.volume=32&rft.issue=8&rft.spage=1586&rft.epage=1594&rft_id=info:doi/10.1109%2FTKDE.2019.2912815&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TKDE_2019_2912815
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1041-4347&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1041-4347&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1041-4347&client=summon