Lessons from a breast cell annotation competition series for school pupils

Due to COVID-19 outbreaks, most school pupils have had to be home-schooled for long periods of time. Two editions of a web-based competition “Beat the Pathologists” for school age participants in the UK ran to fill up pupils’ spare time after home-schooling and evaluate their ability on contributing...

Full description

Saved in:
Bibliographic Details
Published in:Scientific reports Vol. 12; no. 1; pp. 7792 - 11
Main Authors: Lu, Wenqi, Miligy, Islam M., Minhas, Fayyaz, Park, Young Saeng, Snead, David R. J., Rakha, Emad A., Verrill, Clare, Rajpoot, Nasir
Format: Journal Article
Language:English
Published: London Nature Publishing Group UK 12.05.2022
Nature Publishing Group
Nature Portfolio
Subjects:
631/114/2400
692/308
Annotations
Breast cancer
Cell differentiation
Competition
COVID-19
Humanities and Social Sciences
Labeling
multidisciplinary
Neural networks
Pathology
Public concern
Science
Science (multidisciplinary)
Training
Tumors
ISSN:2045-2322, 2045-2322
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Due to COVID-19 outbreaks, most school pupils have had to be home-schooled for long periods of time. Two editions of a web-based competition “Beat the Pathologists” for school age participants in the UK ran to fill up pupils’ spare time after home-schooling and evaluate their ability on contributing to AI annotation. The two editions asked the participants to annotate different types of cells on Ki67 stained breast cancer images. The Main competition was at four levels with different level of complexity. We obtained annotations of four kinds of cells entered by school pupils and ground truth from expert pathologists. In this paper, we analyse school pupils’ performance on differentiating different kinds of cells and compare their performance with two neural networks (AlexNet and VGG16). It was observed that children tend to get very good performance in tumour cell annotation with the best F1 measure 0.81 which is a metrics taking both false positives and false negatives into account. Low accuracy was achieved with F1 score 0.75 on positive non-tumour cells and 0.59 on negative non-tumour cells. Superior performance on non-tumour cell detection was achieved by neural networks. VGG16 with training from scratch achieved an F1 score over 0.70 in all cell categories and 0.92 in tumour cell detection. We conclude that non-experts like school pupils have the potential to contribute to large-scale labelling for AI algorithm development if sufficient training activities are organised. We hope that competitions like this can promote public interest in pathology and encourage participation by more non-experts for annotation.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ISSN:2045-2322
2045-2322
DOI:10.1038/s41598-022-11782-9