Zobrazit v EDS

A public benchmark for human performance in the detection of focal cortical dysplasia

Uloženo v:
Podrobná bibliografie
Název: A public benchmark for human performance in the detection of focal cortical dysplasia
Autoři: Lennart Walger, Matthias H. Schmitz, Tobias Bauer, David Kügler, Fabiane Schuch, Christophe Arendt, Tobias Baumgartner, Johannes Birkenheier, Valeri Borger, Christoph Endler, Franziska Grau, Christian Immanuel, Markus Kölle, Patrick Kupczyk, Asadeh Lakghomi, Sarah Mackert, Elisabeth Neuhaus, Julia Nordsiek, Anna‐Maria Odenthal, Karmele Olaciregui Dague, Laura Ostermann, Jan Pukropski, Attila Racz, Klaus von der Ropp, Frederic Carsten Schmeel, Felix Schrader, Aileen Sitter, Alexander Unruh‐Pinheiro, Marilia Voigt, Martin Vychopen, Philip von Wedel, Randi von Wrede, Ulrike Attenberger, Hartmut Vatter, Alexandra Philipsen, Albert Becker, Martin Reuter, Elke Hattingen, Alexander Radbruch, Rainer Surges, Theodor Rüber
Zdroj: Epilepsia Open
Epilepsia Open, Vol 10, Iss 3, Pp 778-786 (2025)
Epilepsia open 10(3), 778-786 (2025). doi:10.1002/epi4.70028
Informace o vydavateli: Wiley, 2025.
Rok vydání: 2025
Témata: Male, Adult, human performance, diagnostic imaging [Malformations of Cortical Development], computer‐aided detection, artificial intelligence, Magnetic Resonance Imaging, diagnosis [Malformations of Cortical Development], reader study, Focal Cortical Dysplasia, Benchmarking, Humans, Female, Original Article, ddc:610, Neurology. Diseases of the nervous system, RC346-429
Popis: ObjectiveThis study aims to report human performance in the detection of Focal Cortical Dysplasias (FCDs) using an openly available dataset. Additionally, it defines a subset of this data as a “difficult” test set to establish a public baseline benchmark against which new methods for automated FCD detection can be evaluated.MethodsThe performance of 28 human readers with varying levels of expertise in detecting FCDs was originally analyzed using 146 subjects (not all of which are openly available), we analyzed the openly available subset of 85 cases. Performance was measured based on the overlap between predicted regions of interest (ROIs) and ground‐truth lesion masks, using the Dice‐Soerensen coefficient (DSC). The benchmark test set was chosen to consist of 15 subjects most predictive for human performance and 13 subjects identified by at most 3 of the 28 readers.ResultsExpert readers achieved an average detection rate of 68%, compared to 45% for non‐experts and 27% for laypersons. Neuroradiologists detected the highest percentage of lesions (64%), while psychiatrists detected the least (34%). Neurosurgeons had the highest ROI sensitivity (0.70), and psychiatrists had the highest ROI precision (0.78). The benchmark test set revealed an expert detection rate of 49%.SignificanceReporting human performance in FCD detection provides a critical baseline for assessing the effectiveness of automated detection methods in a clinically relevant context. The defined benchmark test set serves as a useful indicator for evaluating advancements in computer‐aided FCD detection approaches.Plain Language SummaryFocal cortical dysplasias (FCDs) are malformations of cortical development and one of the most common causes of drug‐resistant focal epilepsy. Once found, FCDs can be neurosurgically resected, which leads to seizure freedom in many cases. However, FCDs are difficult to detect in the visual assessment of magnetic resonance imaging. A myriad of algorithms for automated FCD detection have been developed, but their true clinical value remains unclear since there is no benchmark dataset for evaluation and comparison to human performance. Here, we use human FCD detection performance to define a benchmark dataset with which new methods for automated detection can be evaluated.
Druh dokumentu: Article
Other literature type
Jazyk: English
ISSN: 2470-9239
DOI: 10.1002/epi4.70028
Přístupová URL adresa: https://pubmed.ncbi.nlm.nih.gov/40167314
https://doaj.org/article/5e20f184f5d14b218036bb109ddaf5e4
https://pub.dzne.de/record/279188
Rights: CC BY
Přístupové číslo: edsair.doi.dedup.....ef7d4751946396ab1954c0ca7c0f7641
Databáze: OpenAIRE
Popis
Abstrakt:ObjectiveThis study aims to report human performance in the detection of Focal Cortical Dysplasias (FCDs) using an openly available dataset. Additionally, it defines a subset of this data as a “difficult” test set to establish a public baseline benchmark against which new methods for automated FCD detection can be evaluated.MethodsThe performance of 28 human readers with varying levels of expertise in detecting FCDs was originally analyzed using 146 subjects (not all of which are openly available), we analyzed the openly available subset of 85 cases. Performance was measured based on the overlap between predicted regions of interest (ROIs) and ground‐truth lesion masks, using the Dice‐Soerensen coefficient (DSC). The benchmark test set was chosen to consist of 15 subjects most predictive for human performance and 13 subjects identified by at most 3 of the 28 readers.ResultsExpert readers achieved an average detection rate of 68%, compared to 45% for non‐experts and 27% for laypersons. Neuroradiologists detected the highest percentage of lesions (64%), while psychiatrists detected the least (34%). Neurosurgeons had the highest ROI sensitivity (0.70), and psychiatrists had the highest ROI precision (0.78). The benchmark test set revealed an expert detection rate of 49%.SignificanceReporting human performance in FCD detection provides a critical baseline for assessing the effectiveness of automated detection methods in a clinically relevant context. The defined benchmark test set serves as a useful indicator for evaluating advancements in computer‐aided FCD detection approaches.Plain Language SummaryFocal cortical dysplasias (FCDs) are malformations of cortical development and one of the most common causes of drug‐resistant focal epilepsy. Once found, FCDs can be neurosurgically resected, which leads to seizure freedom in many cases. However, FCDs are difficult to detect in the visual assessment of magnetic resonance imaging. A myriad of algorithms for automated FCD detection have been developed, but their true clinical value remains unclear since there is no benchmark dataset for evaluation and comparison to human performance. Here, we use human FCD detection performance to define a benchmark dataset with which new methods for automated detection can be evaluated.
ISSN:24709239
DOI:10.1002/epi4.70028