External evaluation of the SORG machine learning algorithm predicting 90-day and 1-year mortality in a Midwest cohort of patients with spinal metastasis

BACKGROUND CONTEXT: The Skeletal Oncology Research Group (SORG) machine learning algorithm was developed in 2019 to predict 90-day and 1-year mortality for patients with spinal metastatic disease. With an increasing prevalence of spinal metastasis and the associated rise in surgical intervention, our group sought to improve accuracy in estimating intermediate and long-term post-operative survival to better inform the shared decision-making process of the patient and surgeon regarding the next steps in care. The SORG algorithm was developed based on a cohort of patients residing in the Northeastern United States but has yet to be evaluated in a rural Midwest cohort.

PURPOSE: The purpose of this study was to externally evaluate the accuracy of the SORG algorithm among a cohort of patients in the Midwest region of the United States.

STUDY DESIGN/SETTING: This external validation study is a retrospective study with data from a Midwest cohort at a single institution.

PATIENT SAMPLE: Patients aged 18 and older who underwent surgical treatment for spinal metastasis between 2010 and 2022 at [redacted] were included in this retrospective study.

OUTCOME MEASURES: Outcome measures included discrimination (c-statistic and receiver operating curve), calibration (calibration slope, intercept, calibration plot, and observed proportions by predicted risk groups), overall performance (Brier score), and decision curve analysis.

METHODS: Baseline characteristics of the validation cohort were obtained and compared to the developmental cohort. Discrimination, calibration, overall performance, and decision curve analysis were used to assess the SORG machine learning algorithm in the validation cohort.

RESULTS: Overall, there were 247 patients included in this study with 90-day and 1-year mortality rates of 63 (25%) and 134 (54%), respectively. The validation cohort and the developmental cohort differed significantly regarding primary tumor histology, the presence of visceral metastasis, and pre-operative hemoglobin levels. The SORG algorithm for 90-day and 1-year mortality showed strong discrimination ability (AUC 0.85 [95% confidence interval [CI] 0.74 to 0.94] and 0.82 [95% CI 0.71 to 0.91] respectively), decision curve analysis, calibration, and Brier score. The 90-day and 1-year mortality showed almost perfect calibration demonstrated by a calibration intercept of 0.06 (95% CI -0.09 to 0.21) and 0.02 (95% CI -0.12 to 0.16) respectively.

CONCLUSION: The SORG machine learning algorithm demonstrated strong generalizability in predicting 90-day and 1-year survival for patients with spinal metastatic disease in a Midwest cohort. Further validation with international patient populations and a prospective, multicenter cohort would be helpful moving forward to confirm these findings and ensure reliable integration into clinical practice.