(1105) Detecting Artificial Intelligence-Generated Content in Personal Statements of Otolaryngology Residency Applications: An Observation Study

Introduction: The recent advancements of artificial intelligence (AI) has produced a new concern for AI-generated documents that appear both professionally written and unique. Residency applicant selection is a rigorous process and AI-generated personal statements may risk obviating the role of these important essays. The goal of this project is to investigate the rate of AI generated content in the personal statements of otolaryngology applications to an allopathic residency program in the United States.

Methods: Personal statements from 40 applicants in 2019 and 39 applicants in 2023 were collected and coded numerically, then analyzed with GPTZero, an AI-detector application. Values reported by the program included probability that the text was entirely generated by AI, confidence in the model’s accuracy, and readability scores for each document. The independent variables for this project were the year in which essays were written, either preceding or after the advent of publicly available AI programs.

Results: A lower proportion of personal statements from the 2023 cohort were rated as solely human written (79.5%) as compared to personal statements from the 2019 cohort (97.5%; X2 = 6.347, p = 0.012). Personal statements from the 2023 cohort also had, on average, lower readability scores (M = 28.55, SD = 10.66) as compared to personal statements from the 2019 cohort (M = 42.65, SD = 7.94; F = 10.199, p < 0.001). However, the confidence of the AI-detector in its analysis was lower for the 2023 cohort (69.2% medium/high confidence) compared to the 2021 cohort (100% medium/high confidence; X2 = 26.755, p < 0.001).

Conclusions: While chatbot detector may distinguish AI-generated text from human-generated text, it is not absolute. As AI introduces additional challenges to the authenticity and utility of personal statements, residency admissions committees along with national organizations may need to re-evaluate the role of personal statements in applications.