The general focus of this book is on multimodal communication, which captures the temporal patterns of behavior in various dialogue settings. After an overview of current theoretical models of verbal and nonverbal communication cues, it presents studies on a range of related topics: paraverbal behavior patterns in the classroom setting; a proposed optimal methodology for conversational analysis; a study of time and mood at work; an experiment on the dynamics of multimodal interaction from the observer’s perspective; formal cues of uncertainty in conversation; how machines can know we understand them; and detecting topic changes using neural network techniques. A joint work bringing together psychologists, communication scientists, information scientists and linguists, the book will be of interest to those working on a wide range of applications from industry to home, and from health to security, with the main goals of revealing, embedding and implementing a rich spectrum of information on human behavior.

This book develops a new framework for describing the structure of multimodal documents: how language, image, layout and other modes of communication work together to convey meaning. Building on recent research in multimodal analysis, functional linguistics and information design, the book examines the textual, visual, and spatial aspects of page-based multimodal documents and employs an analytical model to describe and interpret their structure using the concepts of semiotic modes, medium and genre. To demonstrate and test this approach, the study performs a systematic, longitudinal analysis of a corpus of multimodal documents within a single genre: an extensively annotated corpus of tourist brochures produced between 1967-2008. The book provides multimodal discourse analysts with methodological tools to draw empirically-based conclusions about multimodal documents, and will be a valuable resource for researchers planning to develop and study multimodal corpora.

Gestures are now viewed as an integral part of spoken language. But little attention has been paid to the recipients’ cognitive processes of integrating both gesture and speech. How do people understand a speaker’s gestures when inserted into gaps in the flow of speech? What cognitive-semiotic mechanisms allow this integration to occur? And what linguistic and gestural properties do people draw on when construing multimodal meaning? This book offers answers by investigating multimodal utterances in which speech is replaced by gestures. Through fine-grained cognitive-linguistic and cognitive-semiotic analyses of multimodal utterances combined with naturalistic perception experiments, six chapters explore gestures’ potential to realize grammatical notions of nouns and verbs and to integrate with speech by merging into multimodal syntactic constructions. Analyses of speech-replacing gestures and a range of related phenomena compel us to consider gestures as well as spoken and signed language as manifestations of the same conceptual system. An overarching framework is proposed for studying these different modalities together – a multimodal cognitive grammar.

This book constitutes the refereed proceedings of the 6th International Workshop on Intelligent Virtual Agents, IVA 2006. The book presents 24 revised full papers and 11 revised short papers together with 3 invited talks and the abstracts of 19 poster papers. The papers are organized in topical sections on social impact of IVAs, IVAs recognizing human behavior, human interpretation of IVA behavior, embodied conversational agents, characteristics of nonverbal behavior and more.

Multimodal Composition gives instructors a starting point for rethinking the kinds of texts they teach and produce. Chapters take up fundamental questions, such as What is multimodal composition, and why should I care about it? How do I bring multimodal composition into the classroom? How do I use multiple modes in my scholarship? With practical discussions about assessing student work and incorporating multiple modes into composition scholarship, this book provides a firm foundation for graduate teaching assistants and established instructors alike.

This book constitutes revised selected papers from the 9th International Gesture Workshop, GW 2011, held in Athens, Greece, in May 2011. The 24 papers presented were carefully reviewed and selected from 35 submissions. They are ordered in five sections named: human computer interaction; cognitive processes; notation systems and animation; gestures and signs: linguistic analysis and tools; and gestures and speech.