(0656) How Well Can ChatGPT Interpret Head and Neck Oncologic CT Scans?

Introduction: Machine learning and natural language processing tools, notably ChatGPT, have become immensely popular recently for their ability to create and respond to textual information. The latest iteration of ChatGPT-4 can interpret images as part of prompts. Here, we seek to understand the implications and capabilities of this in medicine, particularly for head and neck oncology.

Methods: GPT-4 was provided 5 de-identified images of computed tomography (CT) scan slices with abnormalities suspicious for oncologic processes as corroborated by initial radiologist report, 4 images with non-oncologic abnormalities, and to elicit false positives, 5 scans with no significant abnormalities. Two fellowship trained head and neck surgeons, blinded to radiologists’ report, graded GPT’s responses on a Likert scale (0 (completely inaccurate) to 4 (accurate)). All prompts were written to convey research and not medical advice purposes. Initial prompts were broad, but subsequent (up to 3 subsequent attempts) prompts were more specific i.e. – “are there abnormalities of the submandibular gland,” where GPT4 failed to provide a satisfactory initial response or deflected to a medical disclaimer.

Results: All responses from GPT-4 contained a “disclaimer” indicating that interpretation should come from a radiologist or medical professional. Mean overall accuracy was 1.21 (SD: 1.13) for all scans. Images with findings suspicious for oncologic processes received an average accuracy rating of 1.1 (SD:1.29). Images of non-oncologic pathologies received a mean accuracy rating of 0.75 (SD: 0.76). Scans with no obvious pathologies received a mean accuracy rating of 1.5 (SD: 1.27).

Conclusions: GPT-4 image analysis performs poorly in analysis of head and neck CT scans, and GPT-4 expresses hesitancy in interpreting imaging. Although there is potential for use, patients nor physicians should rely on GPT-4’s ability to interpret images at this time. As a dynamic tool, further work with alternative imaging modalities, prompt engineering, and subsequent updates will be necessary to determine the capabilities of GPT-4 in medical imaging.