Universität zu Köln

Martinez, Joel E., Funk, Friederike ORCID: 0000-0002-6514-3235 and Todorov, Alexander (2020). Quantifying idiosyncratic and shared contributions to judgment. Behav. Res. Methods, 52 (4). S. 1428 - 1445. NEW YORK: SPRINGER. ISSN 1554-3528

Full text not available from this repository.

Abstract

Identifying relative idiosyncratic and shared contributions to judgments is a fundamental challenge to the study of human behavior, yet there is no established method for estimating these contributions. Using edge cases of stimuli varying in intrarater reliability and interrater agreement-faces (high on both), objects (high on the former, low on the latter), and complex patterns (low on both)-we showed that variance component analyses (VCAs) accurately captured the psychometric properties of the data (Study 1). Simulations showed that the VCA generalizes to any arbitrary continuous rating and that both sample and stimulus set size affect estimate precision (Study 2). Generally, a minimum of 60 raters and 30 stimuli provided reasonable estimates within our simulations. Furthermore, VCA estimates stabilized given more than two repeated measures, consistent with the finding that both intrarater reliability and interrater agreement increased nonlinearly with repeated measures (Study 3). The VCA provides a rigorous examination of where variance lies in data, can be implemented using mixed models with crossed random effects, and is general enough to be useful in any judgment domain in which agreement and disagreement are important to quantify and in which multiple raters independently rate multiple stimuli.

Item Type:	Journal Article
Creators:	Creators Email ORCID ORCID Put Code Martinez, Joel E. UNSPECIFIED UNSPECIFIED UNSPECIFIED Funk, Friederike UNSPECIFIED orcid.org/0000-0002-6514-3235 UNSPECIFIED Todorov, Alexander UNSPECIFIED UNSPECIFIED UNSPECIFIED
URN:	urn:nbn:de:hbz:38-349368
DOI:	10.3758/s13428-019-01323-0
Journal or Publication Title:	Behav. Res. Methods
Volume:	52
Number:	4
Page Range:	S. 1428 - 1445
Date:	2020
Publisher:	SPRINGER
Place of Publication:	NEW YORK
ISSN:	1554-3528
Language:	English
Faculty:	Unspecified
Divisions:	Unspecified
Subjects:	no entry
Uncontrolled Keywords:	Keywords Language SENTENCING GUIDELINES; PARTICIPANTS RESPOND; STATISTICAL POWER; BEAUTY; IMPRESSIONS; PREFERENCES; CONSISTENCY; PSYCHOLOGY; SAMPLES; DESIGN Multiple languages Psychology, Mathematical; Psychology, Experimental Multiple languages
URI:	http://kups.ub.uni-koeln.de/id/eprint/34936