TY - JOUR
T1 - Development and clinical impact assessment of a machine-learning model for early prediction of late-onset sepsis
AU - van den Berg, Merel A M
AU - Medina, O'Jay O A G
AU - Loohuis, Ingmar I P
AU - van der Flier, Michiel M
AU - Dudink, Jeroen J
AU - Benders, Manon M J N L
AU - Bartels, Richard R T
AU - Vijlbrief, Daniel D C
N1 - Funding Information:
The authors acknowledge support from the Applied Data Analytics in Medicine (ADAM) program of the UMC Utrecht in the early stages of this project. We thank Annemarie van ‘t Veen, Saskia Haitjema, Lieke van Schaijk and Ruben Peters for their contributions to this project.
Publisher Copyright:
© 2023 The Author(s)
PY - 2023/9
Y1 - 2023/9
N2 - Background and aim: Preterm infants are prone to neonatal infections such as late-onset sepsis (LOS). The consequences of LOS can be severe and potentially life-threatening. Unfortunately, LOS often presents with unspecific symptoms, and early screening laboratory tests have limited diagnostic value and are often late. This study aimed to build a predictive algorithm to aid doctors in the early detection of LOS in very preterm infants. Methods: In a retrospective cohort study, all consecutively admitted preterm infants (GA ≤ 32 weeks) from 2008 until 2019 were included. They were classified as LOS or control according to blood culture results, currently the gold standard. To generate features, routine and continuously measured oxygen saturation and heart rate data with a minute-by-minute sampling rate were extracted from electronic medical records. Care was taken not to include variables indicative of existing LOS suspicion. The timing of a positive blood culture served as a proxy for LOS-onset. An equivalent timestamp was generated in gestational-age-matched control patients without a positive blood culture. Three machine learning (ML) techniques (generalized additive models, logistic regression, and XGBoost) were used to build a classification algorithm. To simulate the performance of the algorithm in clinical practice, a simulation using multiple alarm thresholds was performed on hourly predictions for the total hospitalization period. Results: 292 infants with LOS were matched to 1497 controls. The median gestational age before matching was 28.1 and 30.3 weeks, respectively. Evaluation of the overall discriminative power of the LR algorithm yielded an AUC of 0.73 (p < 0.05) at the moment of clinical suspicion (t = 0). In the longitudinal simulation, our algorithm detects LOS in at least 47% of the patients before clinical suspicion without exceeding the alarm fatigue threshold of 3 alarms per day. Furthermore, medical experts evaluated the algorithm as clinically relevant regarding the feature contributions in the model explanations. Conclusions: An ML algorithm was trained for the early detection of LOS. Performance was evaluated on both prediction horizons and in a clinical impact simulation. To the best of our knowledge, our assessment of clinical impact with a retrospective simulation on longitudinal data is the most extensive in the literature on LOS prediction to date. The clinically relevant algorithm, based on routinely collected data, can potentially accelerate clinical decisions in the early detection of LOS, even with limited inputs.
AB - Background and aim: Preterm infants are prone to neonatal infections such as late-onset sepsis (LOS). The consequences of LOS can be severe and potentially life-threatening. Unfortunately, LOS often presents with unspecific symptoms, and early screening laboratory tests have limited diagnostic value and are often late. This study aimed to build a predictive algorithm to aid doctors in the early detection of LOS in very preterm infants. Methods: In a retrospective cohort study, all consecutively admitted preterm infants (GA ≤ 32 weeks) from 2008 until 2019 were included. They were classified as LOS or control according to blood culture results, currently the gold standard. To generate features, routine and continuously measured oxygen saturation and heart rate data with a minute-by-minute sampling rate were extracted from electronic medical records. Care was taken not to include variables indicative of existing LOS suspicion. The timing of a positive blood culture served as a proxy for LOS-onset. An equivalent timestamp was generated in gestational-age-matched control patients without a positive blood culture. Three machine learning (ML) techniques (generalized additive models, logistic regression, and XGBoost) were used to build a classification algorithm. To simulate the performance of the algorithm in clinical practice, a simulation using multiple alarm thresholds was performed on hourly predictions for the total hospitalization period. Results: 292 infants with LOS were matched to 1497 controls. The median gestational age before matching was 28.1 and 30.3 weeks, respectively. Evaluation of the overall discriminative power of the LR algorithm yielded an AUC of 0.73 (p < 0.05) at the moment of clinical suspicion (t = 0). In the longitudinal simulation, our algorithm detects LOS in at least 47% of the patients before clinical suspicion without exceeding the alarm fatigue threshold of 3 alarms per day. Furthermore, medical experts evaluated the algorithm as clinically relevant regarding the feature contributions in the model explanations. Conclusions: An ML algorithm was trained for the early detection of LOS. Performance was evaluated on both prediction horizons and in a clinical impact simulation. To the best of our knowledge, our assessment of clinical impact with a retrospective simulation on longitudinal data is the most extensive in the literature on LOS prediction to date. The clinically relevant algorithm, based on routinely collected data, can potentially accelerate clinical decisions in the early detection of LOS, even with limited inputs.
KW - Algorithm
KW - Early-warning
KW - Impact assessment
KW - Late-onset sepsis
KW - Machine learning
KW - NICU
KW - Preterm infants
UR - http://www.scopus.com/inward/record.url?scp=85162901553&partnerID=8YFLogxK
U2 - 10.1016/j.compbiomed.2023.107156
DO - 10.1016/j.compbiomed.2023.107156
M3 - Article
C2 - 37369173
SN - 0010-4825
VL - 163
SP - 1
EP - 10
JO - Computers in Biology and Medicine
JF - Computers in Biology and Medicine
M1 - 107156
ER -