(1008) Comparing the Utility of Google Web Search and ChatGPT Responses in Pediatric Otolaryngology

Introduction: The rising popularity of artificial intelligence tool ChatGPT potentiates its role in answering medical inquiries from the general population. We aimed to compare the accuracy of ChatGPT and Google search responses to common questions regarding pediatric tonsillectomy and tympanostomy tube placement.

Methods: For this qualitative content analysis, 10 popular questions regarding pediatric tonsillectomy and tympanostomy tube placement were generated through ChatGPT and Google searches. Question categorizations and response sources were recorded. Responses to 20 general and 18 discreet pediatric otolaryngology questions were recorded. Two board-certified otolaryngologists independently graded responses on a 4-point scale with 1 indicating greatest accuracy. Flesh-Kincaid reading levels of responses were recorded. A Mann-Whitney U-test compared means of accuracy and reading grade levels.

Results: The most frequent question classification for Google and ChatGPT were indications/management and technical details, respectively. Google utilized academic sources and ChatGPT utilized government sources primarily. Cohen’s K-coefficient for interrater reliability was 0.72. The average accuracy of ChatGPT and Google responses were 1.33 (SD: 0.65) and 2 (0.76) respectively. ChatGPT responses scored significantly higher in accuracy than Google responses (p-value = 0.0006). ChatGPT and Google responses mean reading levels were 14.32 (1.83) and 9.88 (3.2) respectively; reading level was significantly higher for ChatGPT (p-value < 0.0001).

Conclusions: Google and ChatGPT differed in question types and response sources. ChatGPT frequently responded accurately to questions about pediatric otolaryngology procedures and received higher accuracy grades than Google responses. However, the significantly higher average reading level for ChatGPT may preclude comprehension and utilization of its responses for medical inquiries by the general population. Further research should investigate patient perceptions and understanding of responses.