Universität zu Köln

Dimitrova, Diana, Chu, Mingyuan, Wang, Lin ORCID: 0000-0001-6911-0660, Ozyurek, Asli and Hagoort, Peter (2016). Beat that Word: How Listeners Integrate Beat Gesture and Focus in Multimodal Speech Discourse. J. Cogn. Neurosci., 28 (9). S. 1255 - 1270. CAMBRIDGE: MIT PRESS. ISSN 1530-8898

Full text not available from this repository.

Abstract

Communication is facilitated when listeners allocate their attention to important information (focus) in the message, a process called information structure. Linguistic cues like the preceding context and pitch accent help listeners to identify focused information. In multimodal communication, relevant information can be emphasized by nonverbal cues like beat gestures, which represent rhythmic nonmeaningful hand movements. Recent studies have found that linguistic and nonverbal attention cues are integrated independently in single sentences. However, it is possible that these two cues interact when information is embedded in context, because context allows listeners to predict what information is important. In an ERP study, we tested this hypothesis and asked listeners to view videos capturing a dialogue. In the critical sentence, focused and nonfocused words were accompanied by beat gestures, grooming hand movements, or no gestures. ERP results showed that focused words are processed more attentively than nonfocused words as reflected in an N1 and P300 component. Hand movements also captured attention and elicited a P300 component. Importantly, beat gesture and focus interacted in a late time window of 600-900 msec relative to target word onset, giving rise to a late positivity when nonfocused words were accompanied by beat gestures. Our results show that listeners integrate beat gesture with the focus of the message and that integration costs arise when beat gesture falls on nonfocused information. This suggests that beat gestures fulfill a unique focusing function in multimodal discourse processing and that they have to be integrated with the information structure of the message.

Item Type:	Journal Article
Creators:	Creators Email ORCID ORCID Put Code Dimitrova, Diana UNSPECIFIED UNSPECIFIED UNSPECIFIED Chu, Mingyuan UNSPECIFIED UNSPECIFIED UNSPECIFIED Wang, Lin UNSPECIFIED orcid.org/0000-0001-6911-0660 UNSPECIFIED Ozyurek, Asli UNSPECIFIED UNSPECIFIED UNSPECIFIED Hagoort, Peter UNSPECIFIED UNSPECIFIED UNSPECIFIED
URN:	urn:nbn:de:hbz:38-265675
DOI:	10.1162/jocn_a_00963
Journal or Publication Title:	J. Cogn. Neurosci.
Volume:	28
Number:	9
Page Range:	S. 1255 - 1270
Date:	2016
Publisher:	MIT PRESS
Place of Publication:	CAMBRIDGE
ISSN:	1530-8898
Language:	English
Faculty:	Unspecified
Divisions:	Unspecified
Subjects:	no entry
Uncontrolled Keywords:	Keywords Language INFORMATION-STRUCTURE; LANGUAGE COMPREHENSION; ICONIC GESTURES; PITCH ACCENT; COMPONENT; RESPONSES; N400; HAND; WAVE Multiple languages Neurosciences; Psychology, Experimental Multiple languages
Refereed:	Yes
URI:	http://kups.ub.uni-koeln.de/id/eprint/26567