Gestures are now viewed as an integral part of spoken language. But little attention has been paid to the recipients’ cognitive processes of integrating both gesture and speech. How do people understand a speaker’s gestures when inserted into gaps in the flow of speech? What cognitive-semiotic mechanisms allow this integration to occur? And what linguistic and gestural properties do people draw on when construing multimodal meaning? This book offers answers by investigating multimodal utterances in which speech is replaced by gestures. Through fine-grained cognitive-linguistic and cognitive-semiotic analyses of multimodal utterances combined with naturalistic perception experiments, six chapters explore gestures’ potential to realize grammatical notions of nouns and verbs and to integrate with speech by merging into multimodal syntactic constructions. Analyses of speech-replacing gestures and a range of related phenomena compel us to consider gestures as well as spoken and signed language as manifestations of the same conceptual system. An overarching framework is proposed for studying these different modalities together – a multimodal cognitive grammar.

Gestures are ubiquitous and natural in our everyday life. They convey information about culture, discourse, thought, intentionality, emotion, intersubjectivity, cognition, and first and second language acquisition. Additionally, they are used by non-human primates to communicate with their peers and with humans. Consequently, the modern field of gesture studies has attracted researchers from a number of different disciplines such as anthropology, cognitive science, communication, neuroscience, psycholinguistics, primatology, psychology, robotics, sociology and semiotics. This volume presents an overview of the depth and breadth of current research in gesture. Its focus is on the interdisciplinary nature of gesture. The twenty-six chapters included in the volume are divided into six sections or themes: the nature and functions of gesture, first language development and gesture, second language effects on gesture, gesture in the classroom and in problem solving, gesture aspects of discourse and interaction, and gestural analysis of music and dance.

Intelligent Human Systems Integration 2024 Proceedings of the 7th International Conference on Intelligent Human Systems Integration: Integrating People and Intelligent Systems, Università degli Studi di Palermo, Palermo, Italy, February 22- 24, 2024

Repetitive sequences play a major role as a pattern-building device and are a basic syntagmatic linguistic means on all language levels in spoken and signed languages. Little attention has been paid to investigating them in multimodal language use. Do gestures exhibit different types of repetitive sequences? Do they build complex units based on these types and if so, how is the pattern building to be described? How is the interrelation of gestural and spoken units in such complex units? Is it possible to identify repetitive patterns that are comparable to spoken and signed languages and/or patterns specific to the gestural modality? Based on a corpus-analysis of multimodal usage-events, 7 chapters explore gestural repetitions with regard to their structure, semantic and syntactic relevance for multimodal utterances, and cognitive saliency. Fine-grained cognitive-linguistic analyses of multimodal usage events reveal that gestural repetitions are not only a basic principle of building patterns in spoken and signed languages, but also in gestures. By addressing questions of mediality and multimodality of language-in-use, the book contributes to the investigation of repetition as a fundamental means of sign and meaning construction (crosscutting modalities) and enhances the understanding of the multimodal character of language in use.

Co-speech gestures are ubiquitous: when people speak, they almost always produce gestures. Gestures reflect content in the mind of the speaker, often under the radar and frequently using rich mental images that complement speech. What are gestures doing? Why do we use them? This book is the first to systematically explore the functions of gesture in speaking, thinking, and communicating – focusing on the variety of purposes served for the gesturer as well as for the viewer of gestures. Chapters in this edited volume present a range of diverse perspectives (including neural, cognitive, social, developmental and educational), consider gestural behavior in multiple contexts (conversation, narration, persuasion, intervention, and instruction), and utilize an array of methodological approaches (including both naturalistic and experimental). The book demonstrates that gesture influences how humans develop ideas, express and share those ideas to create community, and engineer innovative solutions to problems.