This chapter addresses the relationship between visual information and verbal or textual information. Most visualizations are not purely graphical; they are composites, combining images with text or spoken language. But why do we need words? And when will images and words each be most effective? How should labels be used in diagrams? How should visual and verbal material be integrated in multimedia presentations? A particularly thorny but interesting problem is whether or not we should be using visual languages to program computers. Although computers are rapidly becoming common in every household, very few householders are programmers. It has been suggested that visual programming languages may help make it easier for "non-programmers" to program computers.

We begin by considering the differences between visual and verbal means of communication, then move on to the application areas.

Coding Words and Images

Bertin, in his seminal work, The Semiology of Graphics, distinguishes two distinct sign systems. One cluster of sign systems is associated with auditory information processing and includes mathematical symbols, natural language, and music. The second cluster is based on visual information processing and includes graphics together with abstract and figurative imagery. More recently, the dual coding of Paivio (1987) proposes that there are fundamentally different types of information stored in working memory; he calls them imagens and logogens. Roughly speaking, imagens denote the mental representation of visual information, while logogens denote the mental representation of language information. Visual imagens consist of objects, natural groupings of objects, and whole parts...