(1004) Artificial Intelligence Chatbot Performance in Triage of Pediatric Otolaryngologic Emergencies

Introduction: Artificial intelligence (AI) is a powerful tool that has become increasingly integrated into everyday life. Large language models (LLMs), such as Open AI’s ChatGPT, use natural language processing to generate human-like responses to text prompts. The development of LLMs has transformed AI into an accessible and user-friendly tool for the public. In this study, we focus specifically on the accuracy and reliability of ChatGPT to diagnose and triage pediatric otolaryngologic conditions.

Methods: In total, 25 patient vignettes were created describing a variety of pediatric otolaryngologic conditions. ChatGPT3.5/4 and otolaryngology trainees were asked to generate diagnosis, treatment, and triage level. Triage options included ER, clinic within 24 hours, clinic in 1-2 weeks, and home management. Differential was deemed correct if it included main diagnosis provided by attending and triage was compared to attending recommendation. AI responses were additionally evaluated for any misleading or dangerous information provided.

Results: Chat GPT3.5 and 4 were able to generate diagnoses, treatment, and triage level for all scenarios. A total of 8 otolaryngology trainees completed the survey. Correct diagnosis was 100% for Chat GPT4, 96% for ChatGPT3.5 and average 97% for residents. Triage concordance with attending 70% for residents compared to 60% and 52% for ChatGPT4 and 3.5. AI triage was generally more conservative. ChatGPT3.5 recommended ER or clinic within 24 hours for 100% of scenarios, ChatGPT4 92%, resident 84% and attending 76%. A potentially dangerous recommendation (clinic rather than ER) was noted 3 times with ChatGPT4 and 3 times with 3.5.

Conclusions: This study looks to assess the performance of ChatGPT in diagnosis and triage of pediatric otolaryngologic emergencies. AI performed well in generating an appropriate differential diagnosis and generally made appropriate recommendations for triage. With the rapidly growing scope of AI in medicine along with its accessibility to patients and parents, it is imperative to understand the capabilities and limitations of this tool.