TY - GEN
T1 - Crowd disagreement about medical images is informative
AU - Cheplygina, Veronika
AU - Pluim, Josien P.W.
PY - 2018/1/1
Y1 - 2018/1/1
N2 - Classifiers for medical image analysis are often trained with a single consensus label, based on combining labels given by experts or crowds. However, disagreement between annotators may be informative, and thus removing it may not be the best strategy. As a proof of concept, we predict whether a skin lesion from the ISIC 2017 dataset is a melanoma or not, based on crowd annotations of visual characteristics of that lesion. We compare using the mean annotations, illustrating consensus, to standard deviations and other distribution moments, illustrating disagreement. We show that the mean annotations perform best, but that the disagreement measures are still informative. We also make the crowd annotations used in this paper available at https://figshare.com/s/5cbbce14647b66286544.
AB - Classifiers for medical image analysis are often trained with a single consensus label, based on combining labels given by experts or crowds. However, disagreement between annotators may be informative, and thus removing it may not be the best strategy. As a proof of concept, we predict whether a skin lesion from the ISIC 2017 dataset is a melanoma or not, based on crowd annotations of visual characteristics of that lesion. We compare using the mean annotations, illustrating consensus, to standard deviations and other distribution moments, illustrating disagreement. We show that the mean annotations perform best, but that the disagreement measures are still informative. We also make the crowd annotations used in this paper available at https://figshare.com/s/5cbbce14647b66286544.
UR - http://www.scopus.com/inward/record.url?scp=85055808469&partnerID=8YFLogxK
U2 - 10.1007/978-3-030-01364-6_12
DO - 10.1007/978-3-030-01364-6_12
M3 - Conference contribution
AN - SCOPUS:85055808469
SN - 9783030013639
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 105
EP - 111
BT - Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis - 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018 Held in Conjunction with MICCAI 2018
A2 - Lee, Su-Lin
A2 - Trucco, Emanuele
A2 - Maier-Hein, Lena
A2 - Moriconi, Stefano
A2 - Albarqouni, Shadi
A2 - Jannin, Pierre
A2 - Balocco, Simone
A2 - Zahnd, Guillaume
A2 - Mateus, Diana
A2 - Taylor, Zeike
A2 - Demirci, Stefanie
A2 - Stoyanov, Danail
A2 - Sznitman, Raphael
A2 - Martel, Anne
A2 - Cheplygina, Veronika
A2 - Granger, Eric
A2 - Duong, Luc
PB - Springer-Verlag
T2 - 7th Joint International Workshop on Computing and Visualization for Intravascular Imaging and Computer Assisted Stenting, CVII-STENT 2018, and the 3rd International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, LABELS 2018, held in conjunction with the 21th International Conference on Medical Imaging and Computer-Assisted Intervention, MICCAI 2018
Y2 - 16 September 2018 through 16 September 2018
ER -