TY - JOUR
T1 - Poor performance of ChatGPT in clinical rule-guided dose interventions in hospitalized patients with renal dysfunction
AU - van Nuland, Merel
AU - Snoep, JaapJan D
AU - Egberts, Toine
AU - Erdogan, Abdullah
AU - Wassink, Ricky
AU - van der Linden, Paul D
N1 - Publisher Copyright:
© The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2024.
PY - 2024/8
Y1 - 2024/8
N2 - PURPOSE: Clinical decision support systems (CDSS) are used to identify drugs with potential need for dose modification in patients with renal impairment. ChatGPT holds the potential to be integrated in the electronic health record (EHR) system to give such dosing advices. In this study, we aim to evaluate the performance of ChatGPT in clinical rule-guided dose interventions in hospitalized patients with renal impairment.METHODS: This cross-sectional study was performed at Tergooi Medical Center, the Netherlands. CDSS alerts regarding renal dysfunction were collected from the electronic health record (EHR) during a 2-week period and were presented to ChatGPT and an expert panel. Alerts were presented with and without patient variables. To evaluate the performance, suggested medication interventions were compared.RESULTS: In total, 172 CDDS alerts were generated for 80 patients. Indecisive responses by ChatGPT to alerts were excluded. For alerts presented without patient variables, ChatGPT provided "correct and identical" responses to 19.9%, "correct and different" responses to 26.7%, and "incorrect responses to 53.4% of the alerts. For alerts including patient variables, ChatGPT provided "correct and identical" responses to 16.7%, "correct and different" responses to 16.0%, and "incorrect responses to 67.3% of the alerts. Accuracy was better for newer drugs such as direct oral anticoagulants.CONCLUSION: The performance of ChatGPT in clinical rule-guided dose interventions in hospitalized patients with renal dysfunction was poor. Based on these results, we conclude that ChatGPT, in its current state, is not appropriate for automatic integration into our EHR to handle CDSS alerts related to renal dysfunction.
AB - PURPOSE: Clinical decision support systems (CDSS) are used to identify drugs with potential need for dose modification in patients with renal impairment. ChatGPT holds the potential to be integrated in the electronic health record (EHR) system to give such dosing advices. In this study, we aim to evaluate the performance of ChatGPT in clinical rule-guided dose interventions in hospitalized patients with renal impairment.METHODS: This cross-sectional study was performed at Tergooi Medical Center, the Netherlands. CDSS alerts regarding renal dysfunction were collected from the electronic health record (EHR) during a 2-week period and were presented to ChatGPT and an expert panel. Alerts were presented with and without patient variables. To evaluate the performance, suggested medication interventions were compared.RESULTS: In total, 172 CDDS alerts were generated for 80 patients. Indecisive responses by ChatGPT to alerts were excluded. For alerts presented without patient variables, ChatGPT provided "correct and identical" responses to 19.9%, "correct and different" responses to 26.7%, and "incorrect responses to 53.4% of the alerts. For alerts including patient variables, ChatGPT provided "correct and identical" responses to 16.7%, "correct and different" responses to 16.0%, and "incorrect responses to 67.3% of the alerts. Accuracy was better for newer drugs such as direct oral anticoagulants.CONCLUSION: The performance of ChatGPT in clinical rule-guided dose interventions in hospitalized patients with renal dysfunction was poor. Based on these results, we conclude that ChatGPT, in its current state, is not appropriate for automatic integration into our EHR to handle CDSS alerts related to renal dysfunction.
KW - CDSS
KW - ChatGPT
KW - Clinical rule-guided dose interventions
KW - Language model
KW - Renal dysfunction
UR - http://www.scopus.com/inward/record.url?scp=85189885006&partnerID=8YFLogxK
U2 - 10.1007/s00228-024-03687-5
DO - 10.1007/s00228-024-03687-5
M3 - Article
C2 - 38592470
SN - 0031-6970
VL - 80
SP - 1133
EP - 1140
JO - European Journal of Clinical Pharmacology
JF - European Journal of Clinical Pharmacology
IS - 8
ER -