Lecture 9 from a course on Mobile Based Augmented Reality Development taught by Mark Billinghurst and Zi Siang See on November 29th and 30th 2015 at Johor Bahru in Malaysia. This lecture describes principles for effective Interface Design for Mobile AR applications. Look for the other 9 lectures in the course.
1. LECTURE 9:
DESIGNING MOBILE AR
INTERFACES
Mark Billinghurst
mark.billinghurst@unisa.edu.au
Zi Siang See
zisiang@reina.com.my
November 29th-30th 2015
Mobile-Based Augmented Reality Development
3. Handheld HCI
â˘âŻConsider your user
â˘âŻFollow good HCI principles
â˘âŻAdapt HCI guidelines for handhelds
â˘âŻDesign to device constraints
â˘âŻMicro-Interactions
â˘âŻDesign Patterns
4. ConsiderYour User
â˘âŻ Consider context of user
â˘âŻ Physical, social, emotional, cognitive, etc
â˘âŻ Mobile Phone AR User
â˘âŻ Probably Mobile
â˘âŻ One hand interaction
â˘âŻ Short application use
â˘âŻ Need to be able to multitask
â˘âŻ Use in outdoor or indoor environment
â˘âŻ Want to enhance interaction with real world
5. Follow Good HCI Principles
â˘âŻ Provide good conceptual model/Metaphor
â˘âŻ customers want to understand how UI works
â˘âŻ Make things visible
â˘âŻ if object has function, interface should show it
â˘âŻ Map interface controls to customer s model
â˘âŻ infix -vs- postfix calculator -- whose model?
â˘âŻ Provide feedback
â˘âŻ what you see is what you get!
6. Adapting Existing Guidelines
â˘âŻ Mobile Phone AR
â˘âŻ Phone HCI Guidelines
â˘âŻ Mobile HCI Guidelines
â˘âŻ HMD Based AR
â˘âŻ 3D User Interface Guidelines
â˘âŻ VR Interface Guidelines
â˘âŻ Desktop AR
â˘âŻ Desktop UI Guidelines
7. iPhone Guidelines
â˘âŻMake it obvious how to use your content.
â˘âŻAvoid clutter, unused blank space, and busy
backgrounds.
â˘âŻMinimize required user input.
â˘âŻExpress essential information succinctly.
â˘âŻProvide a fingertip-sized target area for all links and
controls.
â˘âŻAvoid unnecessary interactivity.
â˘âŻProvide feedback when necessary
9. Applying Principles to MobileAR
â˘âŻ Clean
â˘âŻ LargeVideoView
â˘âŻ Large Icons
â˘âŻ Text Overlay
â˘âŻ Feedback
10. AR vs.NonAR Design
â˘âŻ Design Guidelines
â˘âŻ Design for 3D graphics + Interaction
â˘âŻ Consider elements of physical world
â˘âŻ Support implicit interaction
Characteristics Non-AR Interfaces AR Interfaces
Object Graphics Mainly 2D Mainly 3D
Object Types Mainly virtual objects Both virtual and physical objects
Object behaviors Mainly passive objects Both passive and active objects
Communication Mainly simple Mainly complex
HCI methods Mainly explicit Both explicit and implicit
12. Design to Device Constraints
â˘âŻUnderstand the platforms used and design for limitations
â˘âŻ Hardware, software platforms
â˘âŻEg Handheld AR game with visual tracking
â˘âŻ Use large screen icons
â˘âŻ Consider screen reflectivity
â˘âŻ Support one-hand interaction
â˘âŻ Consider the natural viewing angle
â˘âŻ Do not tire users out physically
â˘âŻ Do not encourage fast actions
â˘âŻ Keep at least one tracking surface in view
Art of Defense Game
13. HandheldAR Constraints/Affordances
â˘âŻ Camera and screen are linked
â˘âŻ Fast motions a problem when looking at screen
â˘âŻ Intuitive ânavigationâ
â˘âŻ Phone in hand
â˘âŻ Two handed activities: awkward or intuitive
â˘âŻ Extended periods of holding phone tiring
â˘âŻ Awareness of surrounding environment
â˘âŻ Small screen
â˘âŻ Extended periods of looking at screen tiring
â˘âŻ In general, small awkward platform
â˘âŻ Vibration, sound
â˘âŻ Can provide feedback when looking elsewhere
â˘âŻ Networking - Bluetooth, 802.11
â˘âŻ Collaboration possible
â˘âŻ Guaranteed minimum collection of buttons
â˘âŻ Sensors often available
â˘âŻ GPS, camera, accelerometer, compass, etc
15. Time Looking at Screen
Oulasvirta, A. (2005). The fragmentation of attention in mobile
interaction, and what to do with it. interactions, 12(6), 16-18.
17. Design for MicroInteractions
âŞâŻDesign interaction less than a few seconds
â˘âŻ Tiny bursts of interaction
â˘âŻ One task per interaction
â˘âŻ One input per interaction
âŞâŻBenefits
â˘âŻ Use limited input
â˘âŻ Minimize interruptions
â˘âŻ Reduce attention fragmentation
18. Design Patterns
âEach pattern describes a problem which occurs
over and over again in our environment, and then
describes the core of the solution to that problem in
such a way that you can use this solution a million
times over, without ever doing it the same way twice.â
â Christopher Alexander et al.
C.A. Alexander, A Pattern Language, Oxford Univ. Press, New York, 1977.
19. Handheld AR Design Patterns
Title Meaning Embodied Skills
Device Metaphors Using metaphor to suggest available player
actions
Body A&S NaĂŻve physics
Control Mapping Intuitive mapping between physical and
digital objects
Body A&S NaĂŻve physics
Seamful Design Making sense of and integrating the
technological seams through game design
Body A&S
World Consistency Whether the laws and rules in
physical world hold in digital world
NaĂŻve physics
Environmental A&S
Landmarks Reinforcing the connection between digital-
physical space through landmarks
Environmental A&S
Personal Presence The way that a player is represented in the
game decides how much they feel like living
in the digital game world
Environmental A&S
NaĂŻve physics
Living Creatures Game characters that are responsive to
physical, social events that mimic behaviours
of living beings
Social A&S Body A&S
Body constraints Movement of oneâs body position
constrains another playerâs action
Body A&S Social A&S
Hidden information The information that can be hidden and
revealed can foster emergent social play
Social A&S Body A&S
20. Example:Seamless Design
â˘âŻDesign to reduce seams in the user experience
â˘âŻEg:AR tracking failure, change in interaction mode
â˘âŻPaparazzi Game
â˘âŻChange between AR tracking to accelerometer input
Yan Xu , et.al. , Pre-patterns for designing embodied interactions in handheld augmented reality games,
Proceedings of the 2011 IEEE International Symposium on Mixed and Augmented Reality--Arts, Media,
and Humanities, p.19-28, October 26-29, 2011
21. Example:Living Creatures
â˘âŻVirtual creatures should respond to real world events
â˘âŻ eg. Player motion, wind, light, etc
â˘âŻ Creates illusion creatures are alive in the real world
â˘âŻSony EyePet
â˘âŻ Responds to player blowing on creature
23. AR Interaction
â˘âŻDesigning AR System = Interface Design
â˘âŻ Using different input and output technologies
â˘âŻObjective is a high quality of user experience
â˘âŻ Ease of use and learning
â˘âŻ Performance and satisfaction
25. Design of Objects
â˘âŻObjects
â˘âŻPurposely built â affordances
â˘âŻâFoundâ â repurposed
â˘âŻExisting â already at use in marketplace
â˘âŻMake affordances obvious (Norman)
â˘âŻObject affordances visible
â˘âŻGive feedback
â˘âŻProvide constraints
â˘âŻUse natural mapping
â˘âŻUse good cognitive model
26. Affordances:to give a clue
â˘âŻRefers to an attribute of an object that allows people to
know how to use it
â˘âŻ e.g. a button invites pushing, a door handle affords pulling
â˘âŻNorman (1988) used the term to discuss the design of
everyday objects
â˘âŻSince has been much popularised in interaction design to
discuss how to design interface objects
â˘âŻ e.g. scrollbars afford moving up and down, icons afford clicking
28. âAffordanceâ and Interface Design?
â˘âŻInterfaces are virtual and do not have affordances
like physical objects
â˘âŻNorman argues it does not make sense to talk
about interfaces in terms of ârealâ affordances
â˘âŻInstead interfaces are better conceptualized as
âperceivedâ affordances
â˘âŻ Learned conventions of arbitrary mappings between action
and effect at the interface
â˘âŻ Some mappings are better than others
30. â˘âŻAR is mixture of physical and virtual affordance
â˘âŻPhysical
â˘âŻTangible controllers and objects
â˘âŻVirtual
â˘âŻVirtual graphics and audio
31. Affordance Led Design
â˘âŻMake affordances perceivable
â˘âŻProvide visual, haptic, tactile, auditory cues
â˘âŻAffordance Led Usability
â˘âŻGive feedback
â˘âŻProvide constraints
â˘âŻUse natural mapping
â˘âŻUse good cognitive model
32. Example: AR Chemistry
â˘âŻ Tangible AR chemistry education (Fjeld)
Fjeld, M., Juchli, P., and Voegtli, B. M. 2003. Chemistry education: A tangible
interaction approach. Proceedings of INTERACT 2003, September 1st -5th
2003, Zurich, Switzerland.
35. Case Study 1:AR Lens
â˘âŻPhysical Components
â˘âŻLens handle
â˘âŻ Virtual lens attached to real object
â˘âŻDisplay Elements
â˘âŻLens view
â˘âŻ Reveal layers in dataset
â˘âŻInteraction Metaphor
â˘âŻPhysically holding lens
36. 3DAR Lenses:ModelViewer
!⯠Displays models made up of multiple parts
!⯠Each part can be shown or hidden through the lens
!⯠Allows the user to peer inside the model
!⯠Maintains focus + context
38. Case Study 2:LevelHead
â˘âŻPhysical Components
â˘âŻReal blocks
â˘âŻDisplay Elements
â˘âŻVirtual person and rooms
â˘âŻInteraction Metaphor
â˘âŻBlocks are rooms
39.
40. Handheld Interface Metaphors
â˘âŻ Tangible AR LensViewing
â˘âŻ Look through screen into AR scene
â˘âŻ Interact with screen to interact with
AR content
â˘âŻ Eg Invisible Train
â˘âŻ Tangible AR Lens Manipulation
â˘âŻ Select AR object and attach to device
â˘âŻ Use the motion of the device as input
â˘âŻ Eg AR Lego
41.
42. Space vs. Time - Multiplexed
â˘âŻSpace-multiplexed
â˘âŻMany devices each with one function
â˘âŻ Quicker to use, more intuitive, clutter
â˘âŻ Real Toolbox
â˘âŻTime-multiplexed
â˘âŻOne device with many functions
â˘âŻ Space efficient
â˘âŻ mouse
47. TangibleAR Design Principles
â˘âŻTangible AR Interfaces use TUI principles
â˘âŻPhysical controllers for moving virtual content
â˘âŻSupport for spatial 3D interaction techniques
â˘âŻTime and space multiplexed interaction
â˘âŻSupport for multi-handed interaction
â˘âŻMatch object affordances to task requirements
â˘âŻSupport parallel activity with multiple objects
â˘âŻAllow collaboration between multiple users
49. AR and Perception
"âŻCreating the illusion that virtual images are
seamlessly part of the real world
â˘âŻ Must match real and virtual cues
â˘âŻ Depth, occlusion, lighting, shadows..
50. AR as Perception Problem
â˘âŻGoal of AR to fool human senses â create
illusion that real and virtual are merged
â˘âŻDepth
â˘âŻSize
â˘âŻOcclusion
â˘âŻShadows
â˘âŻRelative motion
â˘âŻEtc..
53. Use the Following Depth Cues
â˘âŻMovement parallax.
â˘âŻIcon/Object size (for close objects)
â˘âŻLinear perspective
â˘âŻTo add side perspective bar.
â˘âŻOverlapping
â˘âŻWorks if the objects are big enough
â˘âŻShades and shadows.
â˘âŻDepends on the available computation
57. Information Presentation
"âŻInformation Presentation
â˘âŻ Amount of information
â˘âŻ Clutter, complexity
â˘âŻ Representation of information
â˘âŻ Navigation cues, POI representation
â˘âŻ Placement of information
â˘âŻ Head, body, world stabilized
â˘âŻ View combination
â˘âŻ Multiple views
64. Possible solutions
â˘âŻOverview + Detail
â˘âŻspatial separation; two views
â˘âŻFocus + Context
â˘âŻmerges both views into one view
â˘âŻZooming
â˘âŻtemporal separation