ChatGPT vs Human Clinical Coders in Determining Appropriate CPT Billing Codes for Otologic Surgeries

Introduction: Evaluate the efficiency and appropriateness of utilizing ChatGPT for the analysis of operative notes and the generation of CPT billing codes in comparison to human clinical coders within the context of an otology practice at a tertiary academic center.

Methods: A total of 191 operative notes from September 2022 to October 2023, with their corresponding billing data, were collected. These operative notes were entered into ChatGPT prompting CPT codes and modifiers as output. The analysis centered on assessing agreement between the existing billing data and the billing data derived from ChatGPT, with a specific focus on variations in Work Relative Value Units (wRVUs).

Results: Among the analyzed operative notes, ChatGPT achieved an exact match with the clinical coders in 22.5% (43/191) of cases, demonstrated some level of matching in 37.2% of cases (71/191), and did not exhibit any degree of matching in 62.8% (120/191) of cases. Of the exact matches, 88.4% (38/43) of cases corresponded to cochlear device implantation. Subgroup analysis on the partial and non-matching groups revealed that ear cartilage grafting was the most overlooked procedure by ChatGPT, with accurate billing occurring only 4.2% (3/72) of the time. Analysis focused solely on the primary CPT code for each procedure revealed a fair level of agreement between ChatGPT and clinical coders with a Cohen's kappa value of 0.31 and match rate of 33% (63/191). Operative notes assessed by clinical coders and ChatGPT resulted an average procedural wRVU of 20.1 and 17.0, respectively.

Conclusions: ChatGPT shows promise in facilitating medical billing practices but requires further refinement and validation of the AI model. By enhancing its recognition of component procedures, such as cartilage grafting, the accuracy and utility of the model should improve. Currently, AI-assisted medical billing is not fully reliable as it tends to generate lower wRVUs and disparate billing practices compared to human coders. However, the potential benefits of increased efficiency and reduced administrative costs may outweigh these differences in the future, warranting further investigation.