TABLE OF CONTENTS Multimodal interface design depends on finding balance between the expressive power and naturalness of the language made available to users on the one hand and the requirements of processability on the other hand. The recognition technology that multimodal systems depend on is characterized by ambiguities and uncertainty, which derive directly from the expressiveness of communication that is offered. Current recognition technology is considerably limited when compared to human-level natural language processing. Particularly for na ve users, the fact that a system has some natural language capabilities may lead to expectations that are too high for a system to fulfill.
The challenge in designing multimodal interfaces consists therefore of finding ways to transparently guide user input in a way that agrees with the limited capabilities of current technology. Empirical evidence accumulated over the past decades has provided a wide variety of important insights into how users employ multimodal language to perform tasks. These findings are overviewed in Section 12.4.1, in which linguistic and cognitive factors are discussed. Section 12.4.2 then presents the principles of a human-centered design process that can guide interface development.
Human language production, on which multimodal systems depend, involves a highly automatized set of skills, many of which are not under full conscious control (Oviatt, 1996b). Multimodal language presents considerably different linguistic structure than unimodal language (Oviatt, 1997; Oviatt & Kuhn, 1998). Each modality is used in a markedly different fashion, as users self-select their unique capabilities...
