(0103) Did ChatGPT Write This Paper: Exploring the Use of Artificial Intelligence in Published Otolaryngology Abstracts

Introduction: With the launch of ChatGPT in recent years, artificial intelligence (AI) has had a growing and substantial impact in the academic community. The use of AI language tools in writing scientific abstracts and manuscripts has been reported within other fields, however the use of AI in otolaryngology publications has yet to be investigated. The purpose of this study was to examine the trend of AI use over 3 years of published abstracts from the largest otolaryngology scientific conference in the nation.

Methods: All published oral presentation abstracts from the 2021-2023 American Academy of Otolaryngology-Head and Neck Surgery annual meeting were run through multiple freely online AI detection models (Sapling, GPTZero, and Copyleaks). Sapling and GPTZero generate a probability that the abstract is written by an AI software. We use a cutoff value of 88% to determine if the abstract is written by AI (> 88%) or a human ( < 88%). Copyleaks definitively classifies abstracts as written by a human or AI.

Results: A total of 1,062 abstracts were included with 253 (23%) from 2021, 399 (38%) from 2022, and 410 (39%) from 2023. Sapling flagged 15% of abstracts in 2021 as having been written by AI, 14% in 2022, and 10.5% in 2023. GPTZero flagged less than 1% of abstracts over all three years as having been written by AI. Copyleaks had an increasing trend of flagged abstracts with less than 1% in 2021, 2.3% in 2022, and 4.6% in 2023 as having been written by AI.

Conclusions: AI language tools are readily accessible to the public and can assist students and physicians in the creation of scientific publications. This has yet to be truly examined within recent otolaryngology literature, nor have any best practices or guidelines for its use in scientific publications been addressed by national societies. Furthermore, the accuracy of AI detectors is unclear as demonstrated by the variable results among 3 commonly used AI detectors. This study shows that only a minority of publications are implicated in the use of AI language tools. However, AI detection models themselves must be used with caution as they are as much of a black box as the AI software itself.