Validation of an AI prediction model for massive blood loss during surgery for metastatic spinal disease: a multi-institutional study

BACKGROUND CONTEXT: Recently, a 2024 TSJ article by Shi et al. described the development of a machine learning (ML) model for predicting massive (>2500mL) intraoperative blood loss during posterior decompressive surgery for spinal metastasis. Surgery included en-bloc or partial resection of vertebrae with stabilization or palliative decompression with stabilization. The model was developed in a Chinese cohort (n = 200) and performed well in an external validation cohort (n = 76) within the same region. We sought to validate the ML model across two new geographical areas (North America and Europe) and patient cohorts. PURPOSE: N/A STUDY DESIGN/SETTING: N/A PATIENT SAMPLE: N/A OUTCOME MEASURES: N/A METHODS: We retrospectively included patients across two institutions between 2016 and 2022. Inclusion and exclusion criteria were consistent with the development study, with additional inclusion of (1) patients undergoing palliative decompression without stabilization, (2) patients with multiple myeloma and lymphoma, and (3) patients who continued anticoagulants perioperatively. Massive intraoperative blood loss was defined as >2500mL and in addition to the Gross formula used in the development study, we quantified this outcome using 6 other methods as there is no gold standard for estimating blood loss. ML model predictions were compared against all intraoperative blood loss quantification methods. A sub-analysis, excluding the additional patient groups, assessed the ML model's performance with the same in- and exclusion criteria as the development cohort. Missing values were imputed using missForest in R. RESULTS: In total, 539 patients were included. There were notable differences in baseline characteristics between our cohort and the development cohort. We included more patients with partial or en-bloc resection, suggesting higher risk of blood loss, and lower Eastern Cooperative Oncology Group (ECOG) scores. However, depending on the method used, massive intraoperative blood loss incidence ranged from 7.7% (p <0.01) to 23% (p >0.999), which is less or equal to the development study respectively. The ML model scored poorly on performance metrics and overestimated the intraoperative blood loss across all outcome methods with the best performance using the Lopez-Picado nadir formula (specificity: 0.352, precision: 0.261), overall performance (F1 Score: 0.394, accuracy: 0.453, Brier score: 0.298, log loss: 0.848), and discriminatory ability (AUC: 0.634 [CI: 0.578, 0.690]), except for recall (0.875). Sub-analysis (n = 421) showed consistent poor performance. CONCLUSIONS: To our knowledge, this is the first external ML model validation within spine surgery demonstrating poor performance. The ML model displayed a strong tendency to overestimate massive intraoperative blood loss risk in our cohort. This overestimation might be attributed to a distributional shift as our cohort had key differences in variables used by the ML model to predict massive intraoperative blood loss, such as ECOG scores and surgery type. Overall, our results emphasize the importance of external validation and the development of ML algorithms in multiple geographical areas with varying baseline characteristic distributions, before implementing them in clinical practice. FDA Device/Drug Status: This abstract does not discuss or include any applicable devices or drugs.