A Picture Is Not Worth One Thousand Words: Improving Artificial Intelligence Diagnoses Using Clinical Vignettes

Introduction: While utilization of artificial intelligence (AI) in medicine shows promise as a helpful adjunct to physicians, it’s role in diagnosis remains unfounded. The purpose of this study is to compare the diagnostic accuracy of advanced web based artificial intelligence when provided a photo of laryngeal pathology versus a photo combined with a clinical vignette.

Methods: Twenty-seven standardized flexible laryngoscopy images, two normal control images and 25 pathologic were collected. The images were provided to ChatGPT with either a clinical vignette followed by the question “What is the most likely diagnosis?” or solely the question. The complexity of each case was assessed by the staff laryngologist. The accuracy of each response was rated by 5 otolaryngologists using a validated AI scale from 0-3. To verify stability of AI responses, the ChatGPT image evaluation was repeated for five consecutive days on the same account.

Results: The complexity of the cases were rated as: 1 low complexity, 2 moderate complexity and 3 high complexity. The mean correctness and consistency ratings of the ChatGPT responses to the picture alone compared to the picture with a prompt were 0.51, 1.57 (p= 0.00) and 0.67, 1.56 (p=0.00), respectively. The mean correctness for low complexity versus high complexity cases for the picture alone versus the picture with the prompt was 0.65 vs1.28 (p=0.02) and 0.4 vs 1.8 (p=0.00). There was no statistically significant difference in the overall consistency and correctness between complexities.

Conclusions: The results presented here reveal that web-based AI diagnosis of laryngoscopy pathologic images can be significantly improved with an accompanying clinical prompt. Specific programming to guide AI has shown promise and with appropriate clinic prompting it can be trained to find correct diagnoses consistently.