Document Type

Dissertation

Date of Degree

Spring 2015

Degree Name

PhD (Doctor of Philosophy)

Degree In

Speech and Hearing Science

First Advisor

Amanda J. Owen Van Horne

Abstract

The overall aim of this current research was to investigate effects of individual differences and task demand on co-speech gestures in communication. Specifically, we examined whether gesture use affected speakers' information content, and whether individual differences in working memory (WM) profiles and lexical retrieval, and task demand could account for variability in gesture use.

Forty-four speaker-listener pairs of Mandarin-speaking adults participated in a video description task. The speaker watched and described motion event videos to the listener, who had two options to choose from. The speaker's descriptions were transcribed and coded for motion element type (manner, path, source, goal, and trajectory), modality use (speech vs. gesture), gesture type (deictic vs. iconic), gestures' relation to speech (complementary vs. supplementary), and information type carried by gesture (spatial vs. semantic). A WM profile/discrepancy was measured by a difference between visuo-spatial and verbal working memory using Automated Working Memory Asessment (Alloway, 2007). Lexical retrieval was measured using a semantic fluency task (naming `animals' or `foods' in a one-minute interval). Task demand was manipulated by changing number of motion elements to be described in each video, ranging from two to four.

The results of an ANOVA showed that speakers did not include more information when they chose to gesture, although they sometimes used supplementary gestures that carried information absent from speech. However, a series of mixed model regression analyses showed that spatial complementary gestures decreased with task demand, whereas spatial supplementary gestures increased with task demand. Also, Individual differences in WM discrepancy and spatial WM capacity, not lexical retrieval, predicted production of semantic supplementary gestures. The interaction between task demand and WM discrepancy predicted spatial complementary gestures. Also, the interaction between task demand and WM discrepancy predicted semantic supplementary gestures.

Most importantly, we found that verbal dominant speakers produced fewer spatial complementary gestures when task demand was high, whereas spatial dominant speakers used these gestures similarly across task demands. Also, spatial dominant speakers tended to use more semantic supplementary gestures than verbal dominant speakers when task demand was low, but no such differences were found when task demand was high.

Taken together, our findings reveal that individuals' gesture production is a complex process, in which speaker-internal factors, such as WM, and speaker-external factors, such as task demand, and even interactions between the two factors could play a role. Given that communication is dynamic and complex, instead of restricting to one factor at a time, we may need to expand our scope to more influencing factors and their interactions to fully understand the underlying mechanism of multi-modal communication.

Public Abstract

This study was aimed to investigate whether gesture use affected speakers’ information content, and whether individual differences in working memory (WM; capacity of holding information in head), and lexical retrieval (ability to find words), and task demand could account for gesture use variability. Forty-four speaker-listener pairs of Mandarin speakers participated in a video description task. Speakers watched and described motion event videos to listeners. The descriptions were transcribed and coded for number of motion elements, modality use (speech vs. gesture), gestures’ relation to speech (complementary vs. supplementary), and information type gesture carried (spatial vs. semantic). A WM profile was measured by a discrepancy between visuo-spatial and verbal WM using Automated Working Memory Assessment (Allloway, 2007). Lexical retrieval was measured by naming ‘animals’ or ‘foods’ in a one-minute interval. Task demand was manipulated by changing number of items shown in each video. We found that speakers did not include more information when they chose to gesture. As task demand increased, spatial complementary gestures decreased but spatial supplementary gestures increased. Individual differences in WM discrepancy and spatial WM capacity, not lexical retrieval, predicted semantic supplementary gestures. Most importantly, verbal-dominant speakers produced fewer spatial complementary gestures when task demand was high, whereas spatial-dominant speakers used these gestures similarly across task demands. Also, spatial-dominant speakers used more semantic supplementary gestures than verbal-dominant speakers only in low task demand. We conclude that individuals’ gesture production is a complex process, in which speaker internal and external factors and interactions could play a role in multi-modal communication.

Keywords

publicabstract, communication, gesture, individual differences, task demand, working memory

Pages

ix, 94 pages

Bibliography

Includes bibliographical references (pages 86-94).

Copyright

Copyright 2015 Shan-ju Lin

Share

COinS