ChatGPT for Automated Cross-Checking of Authors' Conflicts of Interest Against Industry Payments

Monday, September 30, 2024

1:06 PM – 1:12 PM EDT

Location: MBCC, 203-205

Disclosure(s):

Conrad W. Safranek, BS:

faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Trinithas J. Boyi, MD, MA:

faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

R. Peter Manes, MD:

faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Introduction: The CMS "OpenPayments" database tracks industry payments to US physicians to improve research conflicts of interest (COI) transparency, but manual cross-checking of articles' authors against this database is labor-intensive. This study investigates the potential of large language models (LLMs) like ChatGPT for automating COI data analysis in publications, with a proof-of-concept focusing on publications regarding FDA-approved biologics for chronic rhinosinusitis with nasal polyposis (CRwNP).

Methods: We assessed ChatGPT's accuracy in automating the cross-checking of COI disclosures against the OpenPayments database across publications on CRwNP concerning omalizumab, mepolizumab, or dupilumab. First, ChatGPT evaluated author affiliations from PubMed to identify which authors are based in the US. Second, for author names matching one or multiple payment recipients in OpenPayments, ChatGPT undertook a comparative analysis between author affiliation and OpenPayments recipient metadata. Third, ChatGPT scrutinized full article COI statements, producing an intricate matrix of disclosures for each author against each relevant company (Sanofi/Regeneron/Genentech/Novartis/GlaxoSmithKline). A random subset of responses was manually checked for accuracy.

Results: In total, 78 relevant articles and 294 unique US authors were included, leading to 980 LLM queries. Manual verification showed accuracies of 100% (200/200; 95%CI 98.1-100%) for country analysis, 97.4% (113/116; 95%CI 92.7-99.1%) for matching author affiliations with OpenPayments metadata, and 99.2% (1091/1100; 95% CI: 98.5-99.6%) for COI statement data extraction.

Conclusions: LLMs have robust potential to automate author-company-specific COI cross-checking against the OpenPayments database. Our findings pave the way for streamlined, efficient, and accurate COI assessment that could be widely employed across medical research.

Presenting Author(s)

Conrad W. Safranek, BS

Medical Student
Otolaryngology, Yale School of Medicine
New Haven, CT, United States

Disclosure(s):
faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.

Co-Author(s)

CL

CHRISTINA Y. Liu, MS

Medical Student
Yale School of Medicine
New Haven, CT, United States

Disclosure information not submitted.
RR

Rhys Richmond, BS

Division of Otolaryngology, Department of Surgery, Yale School of Medicine
New Haven, Connecticut, United States

Disclosure information not submitted.
Trinithas J. Boyi, MD, MA

Resident Physician
University of California San Francisco - East Bay
Oakland, California, United States

Disclosure(s):
faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.
Ryan A. Rimmer, MD

Assistant Professor of Surgery
Division of Otolaryngology, Department of Surgery, Yale School of Medicine
Cheshire, CT, United States

Disclosure information not submitted.

Senior Author(s)

R. Peter Manes, MD

Associate Professor
Yale School of Medicine
New Haven, CT, United States

Disclosure(s):
faculty for this accredited education activity has no relationship(s) with ineligible companies to disclose.