Ref: Oulasvirta, Estlander, & Nurminen (2009) Personal and Ubiquitous Computing

1. Studying interaction with
3D mobile maps
Antti Oulasvirta

2. Acknowledgements
• Antti Nurminen
• Sara Estlander
• Tuomo Nyyssönen
interaction with 3D mobile maps 2

3. Literature
1. Oulasvirta, A., Estlander, S., & Nurminen, A. (in press).
Embodied interaction with a 3D versus 2D mobile map. Personal
and Ubiquitous Computing.
2. Nurminen, A., & Oulasvirta, A. (in press). Designing interactions
for navigation in 3D mobile maps. In L. Meng, A. Zipf, S. Winter
(Eds.), Map-based Mobile Services: Interactivity and Usability,
Springer.
3. Nurminen, A. (in press). IEEE Computer Graphics & Applications
4. Oulasvirta, A., & Nyyssönen, T. (submitted). ARMIv2: The swiss
army knife approach to mobile usability labs. Journal of Usability
Studies
interaction with 3D mobile maps 3

4. Mobile maps
• 2012: 42 million users in N America + Europe
• 2D dominates
– Raster maps (incl. satellite imaging)
– Vector maps
• At the present mainly for car navigation
• 3D?
• What about pedestrians?
interaction with 3D mobile maps 4

5. interaction with 3D mobile maps 5

6. The reference point problem
interaction with 3D mobile maps 6

7. Volumetric • Flat
Multiple views • One view
Directional view • Area view
Photorealism • Symbolic conventions and text
Six DOFs for movement • Scrolling, panning, zooming
interaction with 3D mobile maps 7

8. Research questions
1. Of what use is the 3rd dimension in solving the
reference point problem?
2. Do users’ strategies reflect the representational
and interactive differences between 3D and 2D?
3. Eventually, how should we design 3D mobile maps?
2D maps are centuries ahead in development!
interaction with 3D mobile maps 8

9. Contents
1. Design
1. Experimenting in the
field
2. Embodied interaction
interaction with 3D mobile maps 9

10. 1. Design
Designing 3D VE interaction
for mobile phones

11. m-LOMA: A 3D city model
• Modeles an area of 2x2 km2
• Wireframe models
• 3500 building facades textured
• 1px ~ 20 cm
• Currently running on N93
– 3D hardware + 240x320 resolution
• Average 30-60 fps
• Several maneuvering modes and assisting features
• All controls mapped to keys of N93
• No GPS due to ”urban canyons”
interaction with 3D mobile maps 11

12. interaction with 3D mobile maps 12

13. Video demonstration
• Show landmark
• Orbiting
• Animated view transition
• Free maneuvering
• Tracks mode
• (Marker arrow)
interaction with 3D mobile maps 13

14. Video
interaction with 3D mobile maps 14

15. Perspective change
• One button
• Animated change
interaction with 3D mobile maps 15

16. Tracks
• One button to turn on/off
• Shows
– tracks
– overlaid street names
– directions at crossings
• Guides movement
– rails
– pivoting
interaction with 3D mobile maps 16

17. 2D map design
pan / scroll
zoom in / out
240x320 px
raster format
official city map
interaction with 3D mobile maps 17

18. 2. A field experiment
16 pre-trained subjects, experienced 3D gamers
Pointing task paradigm
Proximate & remote targets, navigation
A pre-defined route in the modeled area
2D versus 3D, within-subjects
All maneuvering modes optional
Comprehensive data collection

19. Participants
• 19 subjects
recruited
• 3 re-run, 16 males,
M=23.3 yo
• Selection criteria
– Does not know the
Helsinki city center
very well
– Plays 3D first shooter
games frequently
• 3 cinema tickets
rewarded
interaction with 3D mobile maps 19

20. The pointing task paradigm
”Point with your index finger to the direction of the
building that is marked on the mobile map”
Facing direction in PE randomized
An orange
marker arrow
shows the
direction and
distance of
the target
interaction with 3D mobile maps 20

21. Task conditions
1. Proximate pointing
2. Remote pointing
3. Navigation
interaction with 3D mobile maps 21

22. Route
interaction with 3D mobile maps 22

23. Design
• Counterbalancing against two order effects:
– Which ”loop” is started with
– Which map type is started with
interaction with 3D mobile maps 23

24. Training
At home
• Training on desktop with fictitious
”Fakeham”
Before starting the trial:
• 15 practice tasks on N93
• the pointing task
• think aloud
• NASA TLX
+ Background questionnaires
+ 2 x working memory span test
interaction with 3D mobile maps 24

25. Data collection w/
”Mobile usability lab”

26. Data collection
Interaction
• Video observation (w/ ”mobile usability lab”)
• Verbal protocols (trained think aloud)
• Interaction with the map
Performance
• Task completion times & pointing errors
• Workload (NASA TLX)
interaction with 3D mobile maps 26

27. Wishlist for a mobile usability lab
• Mobility
• Captures embodied interaction
– hands and near bodyspace
– deployment of gaze to distant and proximate objects of interest
– walking
– abrupt attention-drawing events in the environment
• Unobtrusiveness
• Multi-method support
• Redundancy
• Quality control
interaction with 3D mobile maps 27

28. Previous systems
• Top row:
– Reichl et al. (2007)
– Schusteritsch et al. (2007),
• Bottom
– Lyons and Starner (2001),
– Applied Science
Laboratories (2008).
interaction with 3D mobile maps 28

29. ARMIv2
interaction with 3D mobile maps 29

30. Mic Mic
Surveillance Remote
camera camera
DC
Power 12V
Wireless adapter Wireless
sender sender
2.4 GHz
Wireless
video receiver Video hub Video
Recorder
”Quad”
Video
Lithium
Lithium DC 7.5
Audio
DC 7.5 DC 7.5
12V
12V
Video
Video ”Necklace”
cam
Face cam
UI cam
interaction with 3D mobile maps 30

31. Integrated video
• ~2000 kbps
MPEG-1
• 520x320
resolution
• 25 fps 25
• MPEG-1 Layer 3
64 kbps mono
sound track
interaction with 3D mobile maps 31

32. Analysis
• 381 tasks, 12.2 hours
• Coding of events from video (1 s accuracy)
– Facing
• Looks at the device, forward, left, right, straight up, up and to the left, or up
and to the right.
– Body turning
• The user turns his body using his feet: left or right.
– Device turning
• The user rotates the device in his hands: clockwise or counterclockwise.
– Head tilting
• The user brings one ear toward the shoulder: to the left, right, upright (back to
normal position).
– Walking
• The user moves his feet more than 50 cm
• Directional pointing errors
• Kappa .75 for inter-coder reliability
• Qualitative analysis for 75 randomly sampled verbal protocols
interaction with 3D mobile maps 32

33. Results

34. Example
• Participant #8, 50:00
interaction with 3D mobile maps 34

35. Task completion time
+ 2D and 3D equally accurate in pointing
interaction with 3D mobile maps 35

36. Movement in VE
interaction with 3D mobile maps 36

37. Use of functionalities
• Unit: times invoked per task
• 3D
– Switch perspective 1.8
– Tracks 1.3
– Show landmark 0.72
– Orbit 0.32
– Fly-to-target 0.19
• 2D
– move north/south 2.2
– move east/west 3.0
– zoom in 2.0
– zoom out 5.8 times.
interaction with 3D mobile maps 37

38. General form of solution
• Look at the target
– Scan around the target for cues
• Direct match
• Reference point search
– Cue scan
– Primed scan
– Ego-centric alignment
• Inference of target direction
• (Strategy change)
interaction with 3D mobile maps 38

39. Looking at the physical environment
interaction with 3D mobile maps 39

40. interaction with 3D mobile maps 40

41. interaction with 3D mobile maps 41

42. 3. Embodied interaction

43. interaction with 3D mobile maps 43

44. interaction with 3D mobile maps 44

45. Examples
• Aligning the two views, VE with PE
• Turning in PE to face in the direction of something
seen in VE
• Placing camera at current position in 3D
• Moving in PE before pointing
• Pointing at a location on the screen of the device
• Letting target arrow guide navigation
interaction with 3D mobile maps 45

46. Use of body
interaction with 3D mobile maps 46

47. Self-location
• Due to task, unavoidable if direct match is not possible
• ~¾ uses PE as primary source for cues, ¼ uses VE
• Deriving the target direction
– Directly if VE and PE aligned
– Indirectly if not
• Direct s-l reduces cognitive demands
interaction with 3D mobile maps 47

48. 3D
• 3D: Direct vs. indirect
(3D users often needed to double check their inference)
interaction with 3D mobile maps 48

49. 2D
• Turning the device in hands / tilting head
• Purpose: Align the roads on the map with roads seen
in PE
• More often direct ego-centric inference than indirect
interaction with 3D mobile maps 49

50. Practice effects
MAP*PRACTICE; Unweighted Means
Current effect: F(3, 45)=2,3837, p=,08178
Effective hypothesis decomposition
Vertical bars denote 0,99 confidence intervals
240
220
200
180
Task completion time
160
140
120
100
80
60
40
interaction with 3D mobile maps 1 2
Phase
3 4
50

51. Practice effects
• Users shifted to more 2D-like strategies:
– Orbiting fell by 19.8%
– Show landmark fell by 14.9%
– Fly-to-target fell by 8.1%.
– Tracks increased by 7.4%.
• 3D users started to
– look up less in the VE (6.0%)
– more in the PE (6.7%)
• 2D shows no changes in how its features are used
interaction with 3D mobile maps 51

52. WM span
• High Corsi span associated with
16% better performance in 3D
• High Corsi span group used
Tracks functionality 26% more
• No similar results for Mannequin
test (rotation)
• No significant effects for 2D
The problem of street-level 3D is
that you have to keep in mind
several invisible spatial locations
interaction with 3D mobile maps 52

53. Summary: Why 3D was slower?
• Slower movement time / time unit
– Spending lot of time wondering around in the street level view
• Wasting time in looking for salient&diagnostic visual cues in
facades
• Unpredictability in what is modeled
• Moving back and forth finding legible visual cues
• Losing one’s own position one’s found
• Inability to use preknowledge of areas
• Inability to update location on map while moving from one site to
another
• Self-location more difficult
interaction with 3D mobile maps 53

54. Summary
1. Positive potentials: Direct matching, ego-centric alignment,
perspective-taking
2. Task analytically strategies are the same
• Selection of cues differs
• Use of pre-knowledge differs
• Self-location tactics differs
Reflected in use of gaze & body
3. (Last slide)
interaction with 3D mobile maps 54

55. Conclusion: Improving 3D
• Design of 3D mobile maps ≠ design of virtual environments
We need to
• caricature facades, focusing on diagnostic and salient cues
• achieve predictability in modeling of cues
• emphasize salient details like logos & statues
• make roof-top view the default view (2.5D?)
• ”undoable” perspective changes, not only in vertical direction
• support self-location w/ GPS+e-compass, ”home button”
• guide maneuvering to tracks when on street level
• consider some map + 3D / focus+context views
interaction with 3D mobile maps 55

56. interaction with 3D mobile maps 56