Martinez, Joel E., Funk, Friederike ORCID: 0000-0002-6514-3235 and Todorov, Alexander (2020). Quantifying idiosyncratic and shared contributions to judgment. Behav. Res. Methods, 52 (4). S. 1428 - 1445. NEW YORK: SPRINGER. ISSN 1554-3528

Full text not available from this repository.

Abstract

Identifying relative idiosyncratic and shared contributions to judgments is a fundamental challenge to the study of human behavior, yet there is no established method for estimating these contributions. Using edge cases of stimuli varying in intrarater reliability and interrater agreement-faces (high on both), objects (high on the former, low on the latter), and complex patterns (low on both)-we showed that variance component analyses (VCAs) accurately captured the psychometric properties of the data (Study 1). Simulations showed that the VCA generalizes to any arbitrary continuous rating and that both sample and stimulus set size affect estimate precision (Study 2). Generally, a minimum of 60 raters and 30 stimuli provided reasonable estimates within our simulations. Furthermore, VCA estimates stabilized given more than two repeated measures, consistent with the finding that both intrarater reliability and interrater agreement increased nonlinearly with repeated measures (Study 3). The VCA provides a rigorous examination of where variance lies in data, can be implemented using mixed models with crossed random effects, and is general enough to be useful in any judgment domain in which agreement and disagreement are important to quantify and in which multiple raters independently rate multiple stimuli.

Item Type: Journal Article
Creators:
CreatorsEmailORCIDORCID Put Code
Martinez, Joel E.UNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Funk, FriederikeUNSPECIFIEDorcid.org/0000-0002-6514-3235UNSPECIFIED
Todorov, AlexanderUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
URN: urn:nbn:de:hbz:38-349368
DOI: 10.3758/s13428-019-01323-0
Journal or Publication Title: Behav. Res. Methods
Volume: 52
Number: 4
Page Range: S. 1428 - 1445
Date: 2020
Publisher: SPRINGER
Place of Publication: NEW YORK
ISSN: 1554-3528
Language: English
Faculty: Unspecified
Divisions: Unspecified
Subjects: no entry
Uncontrolled Keywords:
KeywordsLanguage
SENTENCING GUIDELINES; PARTICIPANTS RESPOND; STATISTICAL POWER; BEAUTY; IMPRESSIONS; PREFERENCES; CONSISTENCY; PSYCHOLOGY; SAMPLES; DESIGNMultiple languages
Psychology, Mathematical; Psychology, ExperimentalMultiple languages
URI: http://kups.ub.uni-koeln.de/id/eprint/34936

Downloads

Downloads per month over past year

Altmetric

Export

Actions (login required)

View Item View Item