Assessing Artificial Intelligence Chatbot Responses for Trainee Education on Inpatient Otolaryngology Consult Management

Introduction: The role of artificial intelligence (AI) continues to expand throughout healthcare, namely with regard to medical education. Our objective was to compare responses generated by popular AI-supported chatbots about common inpatient otolaryngology (ENT) consults to determine their utility as educational resources for junior trainees.

Methods: Three AI chatbots (ChatGPT 3.5, Google Bard, Bing Chat) were provided 16 clinical scenarios from the perspective of an on-call ENT resident looking for management advice for patient chief complaints. Chief complaints included epistaxis, dysphagia, odynophagia, dysphonia, otalgia, dizziness, hearing loss, facial trauma, facial laceration, peritonsillar abscess, stridor, dyspnea, angioedema, facial swelling, bleeding from tracheostomy tube, and bleeding from tracheostomy stoma. Responses were evaluated by an attending otolaryngologist and given a modified Global Quality Scale (GQS) score. Score descriptions corresponded to response quality and flow as follows: poor, generally poor, moderate, good, excellent. Response length (word count) was calculated for each response. Chi-square tests (categorical variables) and analysis of variance (continuous variables) were utilized when applicable with statistical significance defined as p< 0.05.

Results: Responses generated by ChatGPT were of at least good quality more often than Google Bard and Bing Chat responses (81.25% vs. 6.25% vs. 6.25%, p< 0.0001). Responses generated by ChatGPT were of at least moderate quality more often than Google Bard and Bing Chat responses (100% vs. 75% vs. 25%, p< 0.0001). A singular response was given a GQS rating of perfect (ChatGPT, facial swelling). Bing Chat responses were significantly shorter than ChatGPT responses (215 vs. 385 words, p< 0.0001).

Conclusions: ChatGPT responses on inpatient ENT consult topics are superior in quality compared to Google Bard or Bing Chat responses. ChatGPT may benefit junior trainees learning to manage common ENT clinical scenarios, however it should only supplement the guidance of senior residents, fellows, and attending otolaryngologists.