Universität zu Köln

Böke, Annkathrin ORCID: 0009-0006-8016-5703, Hacker, Hannah ORCID: 0009-0000-4709-5180, Chakraborty, Millennia ORCID: 0009-0009-4353-2641, Baumeister-Lingens, Luise ORCID: 0000-0002-5070-2137, Vöckel, Jasper ORCID: 0000-0002-6306-3346, Koenig, Julian ORCID: 0000-0003-1009-9625, Vogel, David HV ORCID: 0000-0003-0645-9034, Lichtenstein, Theresa Katharina ORCID: 0000-0001-5573-1212, Vogeley, Kai ORCID: 0000-0002-5891-5831, Kambeitz-Ilankovic, Lana ORCID: 0000-0002-8218-0425 and Kambeitz, Joseph ORCID: 0000-0002-8988-3959 (2025). Observer-Independent Assessment of Content Overlap in Mental Health Questionnaires: Large Language Model–Based Study. JMIR AI, 4. pp. 1-17. JMIR Publications. ISSN 2817-1705

PDF ai-2025-1-e79868.pdf Bereitstellung unter der CC-Lizenz: Creative Commons Attribution. Download (671kB)

Abstract

[Artikel-Nr. e79868] Background: Mental disorders are frequently evaluated using questionnaires, which have been developed over the past decades for the assessment of different conditions. Despite the rigorous validation of these tools, high levels of content divergence have been reported for questionnaires measuring the same construct of psychopathology. Previous studies that examined the content overlap required manual symptom labeling, which is observer-dependent and time-consuming. Objective: In this study, we used large language models (LLMs) to analyze content overlap of mental health questionnaires in an observer-independent way and compare our results with clinical expertise. Methods: We analyzed questionnaires from a range of mental health conditions, including adult depression (n=7), childhood depression (n=15), clinical high risk for psychosis (CHR-P; n=11), mania (n=7), obsessive-compulsive disorder (n=7), and sleep disorder (n=12). Two different LLM-based approaches were tested. First, we used sentence Bidirectional Encoder Representations from Transformers (sBERT) to derive numerical representations (embeddings) for each questionnaire item, which were then clustered using k-means to group semantically similar symptoms. Second, questionnaire items were prompted to a Generative Pretrained Transformer to identify underlying symptom clusters. Clustering results were compared to a manual categorization by experts using the adjusted rand index. Further, we assessed the content overlap within each diagnostic domain based on LLM-derived clusters. Results: We observed varying degrees of similarity between expert-based and LLM-based clustering across diagnostic domains. Overall, agreement between experts was higher than between experts and LLMs. Among the 2 LLM approaches, GPT showed greater alignment with expert ratings than sBERT, ranging from weak to strong similarity depending on the diagnostic domain. Using GPT-based clustering of questionnaire items to assess the content overlap within each diagnostic domain revealed a weak (CHR-P: 0.344) to moderate (adult depression: 0.574; childhood depression: 0.433; mania: 0.419; obsessive-compulsive disorder [OCD]: 0.450; sleep disorder: 0.445) content overlap of questionnaires. Compared to the studies that manually investigated content overlap among these scales, the results of this study exhibited variations, though these were not substantial. Conclusions: These findings demonstrate the feasibility of using LLMs to objectively assess content overlap in diagnostic questionnaires. Notably, the GPT-based approach showed particular promise in aligning with expert-derived symptom structures.

Item Type:	Article
Creators:	Creators Email ORCID ORCID Put Code Böke, Annkathrin UNSPECIFIED https://orcid.org/0009-0006-8016-5703 UNSPECIFIED Hacker, Hannah UNSPECIFIED https://orcid.org/0009-0000-4709-5180 UNSPECIFIED Chakraborty, Millennia UNSPECIFIED https://orcid.org/0009-0009-4353-2641 UNSPECIFIED Baumeister-Lingens, Luise UNSPECIFIED https://orcid.org/0000-0002-5070-2137 UNSPECIFIED Vöckel, Jasper UNSPECIFIED https://orcid.org/0000-0002-6306-3346 UNSPECIFIED Koenig, Julian UNSPECIFIED https://orcid.org/0000-0003-1009-9625 UNSPECIFIED Vogel, David HV UNSPECIFIED https://orcid.org/0000-0003-0645-9034 UNSPECIFIED Lichtenstein, Theresa Katharina UNSPECIFIED https://orcid.org/0000-0001-5573-1212 UNSPECIFIED Vogeley, Kai UNSPECIFIED https://orcid.org/0000-0002-5891-5831 UNSPECIFIED Kambeitz-Ilankovic, Lana UNSPECIFIED https://orcid.org/0000-0002-8218-0425 UNSPECIFIED Kambeitz, Joseph UNSPECIFIED https://orcid.org/0000-0002-8988-3959 UNSPECIFIED
URN:	urn:nbn:de:hbz:38-799511
Identification Number:	10.2196/79868
Journal or Publication Title:	JMIR AI
Volume:	4
Page Range:	pp. 1-17
Number of Pages:	1
Date:	11 December 2025
Publisher:	JMIR Publications
ISSN:	2817-1705
Language:	English
Faculty:	Faculty of Medicine
Divisions:	Faculty of Medicine > Kinder- und Jugendmedizin > Klinik und Poliklinik für Kinder- und Jugendmedizin Faculty of Medicine > Psychiatrie und Psychotherapie > Klinik und Poliklinik für Psychiatrie und Psychotherapie
Subjects:	Medical sciences Medicine
['eprint_fieldname_oa_funders' not defined]:	Publikationsfonds UzK
Refereed:	Yes
URI:	http://kups.ub.uni-koeln.de/id/eprint/79951