Skip to main content
AAO-HNSF 2024 Annual Meeting & OTO EXPO Main banner
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.
Back

Scientific Oral Presentations

Business of Medicine/Practice Management

Session: Scientific Oral Presentations 3 - Comprehensive Otolaryngology

A Critical Analysis of a Specialized ENT GPT and Top Large-Language Models’ Clinical Response Accuracies

Monday, September 30, 2024
1:36 PM – 1:42 PM EDT
Location: MBCC, 203-205
Disclosure(s):
Brenton T. Bicknell:

faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Delaney H. Sheehan, MD:

faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Benjamin J. Greene, MD:

faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Panuganti A. Bharat, MD:

faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Introduction: Studies in otolaryngology (ORL) have begun to explore the utility of top large language models (LLMs) in clinical care. However, few critical analyses have included recent updates, such as the Claude series and OpenAI’s ChatGPT builder platform, which can be used to build custom ChatGPT LLMs.

Methods: Utilizing OpenAI's builder platform, we developed a ChatGPT tailored to ORL: ENT GPT Assistant (E-GPT-A). We then presented 240 ORL multiple choice questions (MCQs) to our custom version of GPT (E-GPT-A) as well as other publicly available AI LLMs (ChatGPT 3.5 (GPT-3.5), ChatGPT 4 (GPT-4), Claude 2.0 (Cld-2.0), and Claude 2.1 (Cld-2.1). An answer was considered accurate if it corresponded with the answer provided by BoardVitals and OTOQuest, as assessed by three reviewers. Performance of each AI LLM was compared with respect to ORL subspecialty, question difficulty, and in 3 question categories (differential diagnosis, diagnostics, and management).

Results: Overall AI LLM accuracy was 66.3% (782/1200; 95% Confidence Interval, 63.6%-69.0%). E-GPT-A (95% CI: 69.1%-80.1%) exhibited higher response accuracy than GPT-3.5 (Odds Ratio=0.327, 95% CI: 54.2%-66.6%, p< 0.001) and GPT-4 (OR=0.307, 95% CI: 62.4%-74.2%, p=0.130), though this increase was not seen from Cld-2.0 (95% CI: 55.5-67.8%) to Cld-2.1 (OR=-0.018, 95% CI: 54.6%-67.0%, p=0.851). In comparison, GPT-3.5 provided the broadest range of differential diagnoses on average (11±2.23). Increased question difficulty yielded a decrease in response accuracy (p < 0.001). GPT-4 and E-GPT-A exhibited higher response accuracies overall and in diagnostics and management. Though not significant, overall AI LLM average response accuracy was highest in laryngology (69.5%) and allergy and rhinology (67.5%), while lowest in facial plastics (55.5%) and pediatrics (64.5%).

Conclusions: This study emphasizes the latest developments of AI LLMs and potentiates further exploration and development of field-specific AI LLM GPTs, such as the ENT GPT Assistant, as a clinical tool for otolaryngologists.

    Presenting Author(s)

  • Brenton T. Bicknell

    Medical Student
    Heersink School of Medicine, University of Alabama at Birmingham, Birmingham, AL
    Huntsville, Alabama, United States

    Disclosure(s):

    faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

    • Co-Author(s)

  • NR

    Nicholas J. Rivers

    Department of Otolaryngology – Head and Neck Surgery, University of Alabama at Birmingham, Birmingham, AL, Alabama, United States

    Disclosure information not submitted.

  • Delaney H. Sheehan, MD

    Resident
    Department of Otolaryngology – Head and Neck Surgery, University of Alabama at Birmingham, Birmingham, AL
    Birmingham, AL, United States

    Disclosure(s):

    faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

  • AS

    Adam Skelton, n/a

    Heersink School of Medicine, University of Alabama at Birmingham, Birmingham, AL, United States

    Disclosure information not submitted.

  • SF

    Stevan Fairburn

    Heersink School of Medicine, University of Alabama at Birmingham, Birmingham, AL, United States

    Disclosure information not submitted.

  • Benjamin J. Greene, MD

    Associate Professor
    University of Alabama at Birmingham
    Birmingham, AL, United States

    Disclosure(s):

    faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

    • Senior Author(s)

  • PB

    Panuganti A. Bharat, MD

    Assistant Professor
    University of Alabama at Birmingham
    Birmingham, AL, United States

    Disclosure(s):

    faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.