1. Pictures and Words




When should we use a visual display?
What is a visual language?
Dual coding theory?
How to integrate images and words

2. Consider that hieroglyphs gave
way to more abstract symbols

Why turn back the clock?
∏λ
x ∝ ∫ϖ ∫
λ
∫Ψ
∞
1
-17K years
i

3. Architecture for visual thinking
10 billion neurons
Parallel, automatic
Features
Display
Proto-objects and
Patterns
Verbal
Working
Memory
Visual
Working
Memory
GIST
Visual
Query
Interac
ti
on Loo
p
Egocentric object and
Pattern map

4. Pavio’s dual coding theory
Associate structure
Visual
Visual
System
Image
Information
Te
Inf xt
orm
ati
on
Verbal Auditory Information
System From speech
Imagens
Non-verbal
responses
Logogens
Verbal
responses

5. Capacity of visual working
memory (Vogal, Woodman, Luck,
2001)
1 second



Task – change detection
Can see 3.3 objects
Each object can be complex
Orientation
Shape size, texture

6. Capacity of verbal working
memory


Used to be though of a 7 +/- 2
It is now thought of as more a duration of
proto-verbal codes.

7. Theory: Graphics and Words



Graphics for structural logic
Words for procedural logic: conditionals,
qualifiers, if-then else, while. + causality.
Not the formal logic of math but a kind of
concrete logic. (A vowel should have an
odd number on the back).

8. The nature of language





Chomsky, innate deep structures.
Common to computer languages
Critical period for language development
But being verbal is not essential to
language development
Sign languages for the deaf are the most
perfect examples of visual language

9. What is language




Description
Communication of intention
The ability to communicate procedures
and sequences of operations – including
logic – if, but, causes, do a then b then c
Thus far we have only dealt with
description

10. Can there be a true visual
language?
Production
Yes,
But not for most of us!!
Consider verbal language
A critical period
Abstraction, logic
(if, while, perhaps)
Based on speech
Sign languages are true
Visual languages
Understanding
Wernicke
Production
Broca
Auditory
Cortex
Visual
Cortex

11. Sign languages
ASL
Danish SL
Arose spontaneously
Are not related to verbal languages
Have syntax
Become more abstract
Chinese SL
NSL
Goldin-Meadow

12. To be fluent in visual language
we should be trained from early in
life

13. The visual system gives us
Rapid recognition and pattern finding
Visual processing and
visual recognition

14. Abstraction






Jane is Jim’s boss
Jim is Joe’s boss
Anne works for Jane
Mark works for Jim
Anne is Mary’s boss
Anne is Mike’s boss
Pattern
Jane
Jim
Joe
Mark
Anne
Mike
Mary

15. Visual and verbal pseudo-code

While letters in stack



Take a letter
Put a stamp on it
Put it in the ‘out tray’
get line of text
from input file
change characters
to upper case
write line to output
file
Visual programming
languages have a history
of failure
more input?
yes
no
Data flow diagrams are defunct

16. Attaching images and words






Diexis and the diectic gesture
Can be a glance or a nod
Pre-speech
Shown to disambiguate verbal
communications
Why the mouse is so powerful
The basis of shared environments

17. Attaching words to images
- use gestalt laws
A square
b
a
d
help
let me out!
hexagon
c
Some simple shapes

18. Integrated pictures and words more
Effective: Chandler and Sweller 1991

19. XPlane

21. Recap

Words for procedure, logic and abstract
meanings/ images for pattern and
structure

22. Linking images and words
Deixis

Pointing is an elementary speech act.
Pointing links images and words
Put that (points) there (points)

Subject verb predicate



23. Other kinds of gestures

Beat gestures for emphasis
Verb gestures showing how to do
something

McNeil Hand and mind


24. Issues in shared environments

Speech + Pointer + Visuals – most
important components

Subtle ways of directing attention also
important in meeting dynamics.

25. Examples of visual languages



Sanscrit
Petri-nets
Khoros

26. Sanscrit
Count from 1 to 3
for i = 1 to 3 do

27. Petri Net language
Petri nets are stochastic – timed attributed (tokens on nodes, transitions)

28. Khoros

29. Peter