(1214) Evaluating the Ability of Artificial Intelligence Platforms to Create Postoperative Instructions Following Rhinologic Surgery

Introduction: Recently, artificial intelligence (AI) platforms such as ChatGPT and Google Bard have progressed at a rapid pace; to allow for optimal medical outcomes and patient safety, it is crucial that patients have clearly written and easy to follow postoperative instructions. Patients are increasingly turning to AI platforms for medical information. The goal of this study is to evaluate the ability of ChatGPT versus Google Bard to develop accurate and understandable postoperative instructions for patients undergoing sinus surgery.

Methods: Postoperative instructions on 4 rhinologic procedures (endoscopic sinus surgery, balloon sinuplasty, Caldwell Luc surgery, and septoplasty) were generated using ChatGPT 4.0 and Google Bard (January 2024). Measures of readability, Flesch Kincaid Grade Level (FKGL) and Flesch Kincaid Reading Ease (FKRE), were calculated. The Patient Education Materials Assessment Tool (PEMAT) was utilized to determine understandability and actionability. Instructions are deemed understandable when patients of diverse health literacy can process them, and actionable when patients of diverse health literacy can identify steps based on the information. Accepted standards for adequate patient comprehension are FKGL < 6, FKRE > 60, and PEMAT > 80%.

Results: The different AI platforms were not uniformly optimal for patient comprehension. ChatGPT had higher understandability than Bard (mean 98 vs 92, p = 0.038). Google Bard had higher actionability than ChatGPT (mean 55 vs 40, p = 0.024) There were no significant differences in FKGL and FKRE between ChatGPT and Bard (mean 9 vs 8, mean 56 vs 52, p = 0.093, respectively).

Conclusions: ChatGPT and GoogleBard were capable of generating understandable postoperative instructions but were poor for actionability and readability. While AI has a promising ability to generate accessible and tailored medical information, patients should be aware of its drawbacks especially when seeking important postoperative instructions following rhinologic surgery.