Fervers, Philipp, Hahnfeldt, Robert, Kottlors, Jonathan, Wagner, Anton, Maintz, David, Pinto dos Santos, Daniel, Lennartz, Simon and Persigehl, Thorsten (2024). ChatGPT yields low accuracy in determining LI-RADS scores based on free-text and structured radiology reports in German language. Frontiers in Radiology, 4. pp. 1-9. Frontiers. ISSN 2673-8740

[thumbnail of fradi-04-1390774.pdf] PDF
fradi-04-1390774.pdf
Bereitstellung unter der CC-Lizenz: Creative Commons Attribution.

Download (3MB)
Identification Number:10.3389/fradi.2024.1390774

Abstract

[Artikel-Nr.: 1390774] Background: To investigate the feasibility of the large language model (LLM) ChatGPT for classifying liver lesions according to the Liver Imaging Reporting and Data System (LI-RADS) based on MRI reports, and to compare classification performance on structured vs. unstructured reports. Methods: LI-RADS classifiable liver lesions were included from German written structured and unstructured MRI reports with report of size, location, and arterial phase contrast enhancement as minimum inclusion requirements. The findings sections of the reports were propagated to ChatGPT (GPT-3.5), which was instructed to determine LI-RADS scores for each classifiable liver lesion. Ground truth was established by two radiologists in consensus. Agreement between ground truth and ChatGPT was assessed with Cohen's kappa. Test-retest reliability was assessed by passing a subset of n = 50 lesions five times to ChatGPT, using the intraclass correlation coefficient (ICC). Results: 205 MRIs from 150 patients were included. The accuracy of ChatGPT at determining LI-RADS categories was poor (53% and 44% on unstructured and structured reports). The agreement to the ground truth was higher ( k = 0.51 and k = 0.44), the mean absolute error in LI-RADS scores was lower (0.5 ± 0.5 vs. 0.6 ± 0.7, p < 0.05), and the test-retest reliability was higher (ICC = 0.81 vs. 0.50), in free-text compared to structured reports, respectively, although structured reports comprised the minimum required imaging features significantly more frequently (Chi-square test, p < 0.05). Conclusions: ChatGPT attained only low accuracy when asked to determine LI-RADS scores from liver imaging reports. The superior accuracy and consistency throughout free-text reports might relate to ChatGPT's training process. Clinical relevance statement: Our study indicates both the necessity of optimization of LLMs for structured clinical data input and the potential of LLMs for creating machine-readable labels based on large free-text radiological databases.

Item Type: Article
Creators:
Creators
Email
ORCID
ORCID Put Code
Fervers, Philipp
UNSPECIFIED
UNSPECIFIED
UNSPECIFIED
Hahnfeldt, Robert
UNSPECIFIED
UNSPECIFIED
UNSPECIFIED
Kottlors, Jonathan
UNSPECIFIED
UNSPECIFIED
UNSPECIFIED
Wagner, Anton
UNSPECIFIED
UNSPECIFIED
UNSPECIFIED
Maintz, David
UNSPECIFIED
UNSPECIFIED
UNSPECIFIED
Pinto dos Santos, Daniel
UNSPECIFIED
UNSPECIFIED
UNSPECIFIED
Lennartz, Simon
UNSPECIFIED
UNSPECIFIED
UNSPECIFIED
Persigehl, Thorsten
UNSPECIFIED
UNSPECIFIED
UNSPECIFIED
URN: urn:nbn:de:hbz:38-817066
Identification Number: 10.3389/fradi.2024.1390774
Journal or Publication Title: Frontiers in Radiology
Volume: 4
Page Range: pp. 1-9
Number of Pages: 9
Date: 5 July 2024
Publisher: Frontiers
ISSN: 2673-8740
Language: English
Faculty: Faculty of Medicine
Divisions: Faculty of Medicine > Radiologische Diagnostik > Institut und Poliklinik für Radiologische Diagnostik
Subjects: Medical sciences Medicine
Uncontrolled Keywords:
Keywords
Language
diagnostic imaging ; neoplasms ; liver ; diagnosis ; LI-RADS (liver imaging reporting and data system) ; MRI
English
['eprint_fieldname_oa_funders' not defined]: Publikationsfonds UzK
Refereed: Yes
URI: http://kups.ub.uni-koeln.de/id/eprint/81706

Downloads

Downloads per month over past year

Altmetric

Export

Actions (login required)

View Item View Item