Not all risk scores are created equal: A comparison of risk scores for abdominal aortic aneurysm repair in administrative data and quality improvement registries

OBJECTIVE: Accurate and contemporary prognostic risk prediction is essential to inform clinical decision-making surrounding abdominal aortic aneurysm (AAA) care. Therefore, we validated and compared three different in-hospital mortality risk scores in one administrative and two quality improvement registries.

METHODS: We included patients who had undergone elective AAA repair from 2012 to 2015 in the National Inpatient Sample (NIS), Vascular Quality Initiative (VQI; excluding the New England region), and the National Surgical Quality Improvement Program (NSQIP) datasets to validate three risk scores: Medicare, the Vascular Study Group of New England (VSGNE), and Glasgow Aneurysm Score (GAS). The receiver operating characteristic area under the curve (AUC) of all risk scores was calculated, and their discrimination was compared within a dataset using the Delong test and between datasets using a Z test. We constructed graphic calibration curves for the Medicare and VSGNE risk scores and compared the calibration using an integrated calibration index, which indicates the weighted average of the absolute difference between the calibration curve and the diagonal line of perfect calibration.

RESULTS: We identified a total of 25,461 NIS, 18,588 VQI, and 8051 NSQIP patients who had undergone elective open or endovascular AAA repair. Overall, the Medicare risk score was more likely to overestimate mortality in the quality improvement registries and the VSGNE risk score underestimated mortality in all the databases. After endovascular AAA repair, the Medicare risk score had a higher AUC in the NIS than in the GAS (P < .001) but not compared with the VSGNE risk score (P = .54). The VSGNE risk score was associated with a significantly higher receiver operating characteristic AUC compared with the Medicare (P < .001) and GAS (P < .001) risk scores in the VQI registry. Also, the VSGNE risk score showed improved calibration compared with the Medicare risk score across all three databases (P < .001 for all). After open repair, the Medicare risk score showed improved calibration compared with the VSGNE risk score in the NIS (P < .001). However, in the VQI registry, the VSGNE risk score compared with the Medicare risk score had significantly better discrimination (P = .008) and calibration (P < .001).

CONCLUSIONS: Overall, the VSGNE risk score performed best in the quality improvement registries but underestimated mortality. However, the Medicare risk score demonstrated better calibration in the administrative dataset after open repair. Although the VSGNE risk score appeared to perform better in the quality improvement registries, its overly optimistic mortality estimates and its reliance on detailed anatomic and clinical variables reduces its broader applicability to other databases.