TABLE OF CONTENTS An essential distinguishing capability of multimodal systems is the use these systems make of two or more natural input modalities such as speech, handwriting, gestures, facial expressions, and other body movements (Oviatt, 2007). Using such systems, users may for example deal with a crisis management situation using speech and pen input over a map. An emergency response route can for example be established by sketching a line on a map along the desired route, using a digital pen or a Tablet PC stylus, while speaking "Create emergency route here" (Oviatt et al., 2003). In this example, a multimodal system analyzes the speech and sketching captured via sensors (a microphone and an instrumented writing surface), and interprets the user's intention by combining (or fusing) the complementary information provided via these two modalities. The result in this case is an interpretation that takes location information from the sketched line and the attributes of the object being created (that this is an "emergency route") from the speech.
Multimodal interfaces may take many forms and aspects (Figure 12.1). Systems may be hosted by small portable devices, such as on a PDA or cell phone that is taken to the field, or on a tablet computer used within offices or in cafeterias. Groups of people may interact multimodally via large interactive boards or sheets of digital paper. The common aspect in all these cases is the interface support for interaction via natural modes of communication involving...
