This is the COSC 426 Lecture 4 on Designing AR Interfaces. Taught by Mark Billinghurst from the HIT Lab NZ at the University of Canterbury. This is part of his graduate course on Augmented Reality. Taught on August 2nd 2013
3. Tracking Requirements
Augmented Reality Information Display
World Stabilized
Body Stabilized
Head Stabilized
Increasing Tracking
Requirements
Head Stabilized Body Stabilized World Stabilized
6. Markerless Tracking
Magnetic Tracker Inertial
Tracker
Ultrasonic
Tracker
Optical
Tracker
Marker-Based
Tracking
Markerless
Tracking
Specialized
Tracking
Edge-Based
Tracking
Template-Based
Tracking
Interest Point
Tracking
No more Markers! Markerless Tracking
7. Natural Feature Tracking
Use Natural Cues of Real Elements
Edges
Surface Texture
Interest Points
Model or Model-Free
++: no visual pollution
Contours
Features Points
Surfaces
8. Combining Sensors and Vision
Sensors
- Produce noisy output (= jittering augmentations)
- Are not sufficiently accurate (= wrongly placed augmentations)
- Gives us first information on where we are in the world,
and what we are looking at
Vision
- Is more accurate (= stable and correct augmentations)
- Requires choosing the correct keypoint database to track from
- Requires registering our local coordinate frame (online-
generated model) to the global one (world)
10. The Registration Problem
Virtual and Real must stay properly aligned
If not:
Breaks the illusion that the two coexist
Prevents acceptance of many serious applications
11. Sources of registration errors
Static errors
Optical distortions
Mechanical misalignments
Tracker errors
Incorrect viewing parameters
Dynamic errors
System delays (largest source of error)
- 1 ms delay = 1/3 mm registration error
12. Reducing static errors
Distortion compensation
Manual adjustments
View-based or direct measurements
Camera calibration (video)
13. Reducing dynamic errors (1)
Reduce system lag
Faster components/system modules
Reduce apparent lag
Image deflection
Image warping
14. Reducing System Lag
Tracking Calculate
Viewpoint
Simulation
Render
Scene
Draw to
Display
x,y,z
r,p,y
Application Loop
Faster Tracker Faster CPU Faster GPU Faster Display
20. Interaction Design is User Centered
A methodology for ensuring good user
experiences with products by getting feedback
from users to inform the design.
23. What, How and Why?
• What
1. Understand users, task, context
2. Produce a stable set of requirements
• How:
Data gathering activities
Data analysis activities
Expression as ‘requirements’
• Why:
Requirements definition: failure occurs most commonly
Getting requirements right is crucial
Iterate
24. What Are the Requirements?
Functional
What should the system do? What has it done?
Data
What type of data?
Environment or Context of use
Physical: dusty? noisy? light? heat? humidity? ….
Social: sharing of files, of displays, work alone ..
Organizational: IT support, hierarchy, training..
Users: Who are they?
Usability: learnability, throughput, flexibility..
25. Who REALLY are your Users/Stakeholders?
• Not as obvious as you think:
— those who interact directly with the product
— those who manage direct users
— those who receive output from the product
— those who make the purchasing decision
— those who use competitor’s products
• Three categories of user (Eason, 1987):
— primary: frequent hands-on
— secondary: occasional or via someone else
— tertiary: affected by its introduction, or will influence its purchase
26. Who are the Stakeholders?
Check-out operators
CustomersManagers and owners
• Suppliers
• Local shop
owners
29. Data Gathering Techniques (1)
Questionnaires
Looking for specific information
Qualitative and quantitative results
Good for getting data from a large, dispersed group
Interviews
Good for exploring issues, using props
Structured, unstructured or semi-structured
But are time consuming and difficult to visit everyone
30. Data Gathering Techniques (2)
Workshops or focus groups
Group interviews/activities
Good at gaining a consensus view and/or highlighting areas
of conflict
Observations
Spending time with users in day to day tasks
Good for understanding task context
requires time and can result in a huge amount of data
Documentation
Procedures and rules written down in manuals
31. A day in the Life of.. Cultural Probes.. Role Playing..
35. Elaboration and Reduction
Elaborate - generate solutions. These are the opportunities
Reduce - decide on the ones worth pursuing
Repeat - elaborate and reduce again on those solutions
Source: Laseau,P. (1980) Graphic Thinking for Architects & Designers. John Wiley and Sons
36. The Design Funnel
Alternate generation of ideas and convergence until resolution
Modified from Pugh, S. (1990) Total design: Integrated methods for successful products engineering.
Addison-Wesley. P. 75
38. Unmet Needs
Digital Cameras for Apple
Late 80s – What would happen if we married
a computer with a digital camera?
Many studies of current photography use.
Uncovering of unmet needs in traditional experience
- not enough photos
- not knowing whether photo good or not
- ability to add sound
- ability to organize
Features determined through research and designed
Iteratively.
41. Tools for Effective Design
Personas
Scenarios
Storyboards
Wireframes and Mock-ups
Prototypes
42. Persona Technique
• Personas are a design tool to help visualize who you are
designing for and imagine how person will use the product
• A persona is an archetype that represents the behavior and
goals of a group of users
• Based on insights and observations from customer research
• Not real people, but synthesised from real user characteristics
• Bring them to life with a name, characteristics, goals, background
• Develop multiple personas
43. Gunther the Ad Guy
Gunther is from Germany. He
Travels extensively for work and
As he is an advertising executive
he needs to present concepts to
clients quickly and easily. He is
a person very well-versed in new
technologies and wishes he had
easier portable solutions for his
presentations…..
44. Scenarios
Usage Scenarios are narrative descriptions of how the product
meets the needs of a persona
Short (2 pages max)
Focus on unmet needs of persona
Concrete story
Set of stories around essential tasks, problems...
Use to test ideas
45. A business woman travels from SF to Paris on a business trip. On
her way to the airport she narrowly misses a traffic delay. She
avoids the jam because her Smartphone beeps and send her a
warning text message on her route from the office to the airport.
Upon arrival the location-sensitive Smartphone notifies the airline
that she‘ll check in shortly and an airline employee finds her
immediately and takes her baggage. Her display shows that her
flight is on time and provides a map to her gate. On the way she
downloads tourist information (maps, events) for Paris.
46. Once found her seat, she begins to review the downloaded
information. She books a ticket for an opera she wants to see.
Her Smartphone makes the booking using her credit card number
stored in memory.The security software of the Smartphone
protects her against fraud.
The Smartphone stores the opera booking along with emails
written on the plane.As soon as she steps off the plane, it makes
the calls and sends the emails.As she leaves the airport, a map
appears on the display and guides her to her hotel
47. Storyboarding
Sequence of sketches showing use of system in
everyday use context
Concrete example
Easier (faster) to grasp than text based stories
Means of communication with users and system
developers
Sketches, not drawings...
Use to test interaction and make sure design works
52. Sketching is about design
Sketching is not about drawing
It is about design.
Sketching is a tool to help you:
- express
- develop, and
- communicate design ideas
Sketching is part of a process:
- idea generation,
- design elaboration
- design choices,
- engineering
53. Why is sketching useful?
Early ideation
Think through ideas
Force you to visualize how things come together
Communicate ideas to others to inspire new
designs
Active brainstorming
54. Figure 41: A Sketch of a Dialogue with a Sketch
The “conversation” between the sketch (right bubble) and the mind (left
bubble). A sketch is created from current knowledge (top arrow). Reading,
or interpreting the resulting representation (bottom arrow), creates new
knowledge. The creation results from what Goldschmidt calls “seeing
that” reasoning, and the extraction of new knowledge results from what
she calls “seeing as.”
sketch
representation
mind
(new) knowledge
Create
(seeing that)
Read
(seeing as)
54
55. Sketchbook examples
Industrial designer
ideas
variations
annotations
Source: sketch page from student Industrial designer Samnang Eav
Form factors for wearable computer watch
60. Technique: Annotations
Textual notes
Name and/or explain thinks
Add details
List of items
Questions/issues about design
Graphical marks
Connect notes to sketch elements
Relates sketch elements
Show dynamics of elements or interaction over time
61. Technique: Tracing
Basic Idea
- Copy/trace the fixed elements
of an existing interface/screen
Technology
- Camera, photograph, tracing
paper or..
- Photoshop or equivalent (trace
over image on a separate layer)
Drawing skill required
- Almost zero
62.
63. Technique Hybrid Sketches
Mixes full and low fidelity elements
High fidelity parts:
Fixed design elements
Low fidelity parts:
Speculative components
Contrast
Makes certain parts of the sketch stand out
64.
65.
66. Your Turn
Design a mobile AR user interface
Viewing virtual buildings in Christchurch
Views (Should be able to swap between)
Map View
AR View
List View
68. 68www.id-book.com
What is a prototype?
In interaction design it can be (among other things):
• a series of screen sketches
• a storyboard, i.e. a cartoon-like series of scenes
• a Powerpoint slide show
• a video simulating the use of a system
• a lump of wood (e.g. PalmPilot)
• a cardboard mock-up
• a piece of software with limited functionality
69. 69
www.id-book.com
Why Prototype?
• Evaluation and feedback are central to interaction design
• Stakeholders can see, hold, interact with a prototype
more easily than a document or a drawing
• Team members can communicate effectively
• You can test out ideas for yourself
• It encourages reflection: very important aspect of design
• Prototypes answer questions, and support designers in
choosing between alternatives
70. Sketch vs. Prototype
Sketch
Prototype
Invite
A)end
Suggest
Describe
Explore
Refine
Ques;on
Answer
Propose
Test
Provoke
Resolve
Tenta;ve,
non
commi)al
Specific
Depic;on
The primary differences are in the intent
71. Types of Prototypes
Low Fidelity – quick and dirty, easy access
materials like cardboard and paper.
High Fidelity – more involved electronic
versions similar in materials to final product.
72. RAPID Prototyping
Fast and inexpensive
Identifies problems before they’re coded
Elicits more and better feedback from users
Helps developers think creatively
Gets users and other stakeholders involved early
Fosters teamwork and communication
Avoids opinion wars
Helps decide design directions
73. Paper Prototyping (Low Fidelity)
Quick and simple means of sketching interfaces
Use office materials
Easier to criticize, quick to change
Creative process (develop in team)
Can also use for usability test (focus on flow of
interaction rather than visuals)
Used a lot to test out concepts before real design begins.
85. Technique: State Transition Diagrams
Create transition diagram
key interaction steps
branch points
For each transition
Sketch the screen
Include the transitional diagram
Label the transition with what triggered the
transition
(typically user input of set of system responses)
101. High-fidelity prototyping
• Uses materials that you would expect to be in the
final product.
• Prototype looks more like the final system than a
low-fidelity version.
• For a high-fidelity software prototype common
environments include Macromedia Director,Visual
Basic, and Smalltalk.
• Danger that users think they have a full
system…….see compromises
104. Rapid Prototyping
Speed development time with quick hardware mockups
handheld device connected to PC
LCD screen, USB phone keypad, Camera
Can use PC development tools for rapid development
Flash, Visual Basic, etc
105. ‘Wizard-of-Oz’ Prototyping
• The user thinks they are interacting with a
computer, but a developer is responding to output
rather than the system.
• Usually done early in design to understand users’
expectations
>Blurb blurb
>Do this
>Why?
User
106. Typical Iterations for HW/SW Product:
1. Paper prototype to evaluate conceptual model
2. Interactive computer-based prototype with rough
screens to evaluate feature placement
3. Tethered prototype to evaluate button + screen
interactions
4. Real device prototyped with major features
working
5. Real device prototyped with all features working
and graphic design implemented
111. BUNRATTY FOLK PARK
Irish visitor attraction run by Shannon Heritage
19th century life is recreated
Buildings from the mid-west have been relocated
to the 26-land surrounding Bunratty Castle
30 buildings are set in a rural or village setting
there.
112. AUGMENTED REALITY
In Bunratty Folk Park:
Allows the visitor to point a camera at an
exhibit, the device recognises it by it’s location
and layers digital information on to the display
3-dimensional virtual objects can be positioned
with real ones on display
Leads to dynamic combination of a live camera
view and information
113. NAVIGATIONAL AID
Smartphone Platform
Most people carry mobile phones and are
comfortable with them
Ideal Augmented Reality Technology
Global tracking tools
Wireless communication capabilities
Location based computing
Large display for interaction
115. HUMAN CENTRED DESIGN
Goal of the Navigational Aid
Easy to use, clear and understandable
Useful to visitors
Creating interaction between the visitor and the aid
through the user interface
Engage the visitor
It is necessary to understand the visitor to Bunratty
Identify visitor motives and goals while going through
the Folk Park.
116. HUMAN CENTRED DESIGN
Understanding the technology and Related Works
Literature
Similar Technologies
Electronic Tours in Museum Settings
Interactions design in Outdoor Museums
Understanding the User over time
Observations
Interviews
117. HUMAN CENTRED DESIGN
Findings
Most visitors do not use the map
Most visitors have mobile phones
Visitors want more information
View the Folk Park at their own pace
Information should be straight to the point
Large social interaction within groups
118. NEXT STEPS FROM RESEARCH
Define Visitors Goals
Define Functionalities of the Aid
Develop Personas – visitors who use the Aid
Develop Scenarios – how the persona uses the
navigational aid in the Folk Park
Draw up Storyboards on scenarios
119. FUNCTIONALITY
View Options
Camera View
Map View
List View
Sub-Options
Places
Events
Restaurants
AR Features
Text Information
3D Objects
3D Tour Guide
3D Placement of Buildings
124. ITERATIVE DESIGN PROCESS
Prototyping and User Testing
Low Fidelity Prototyping
Sketches
Paper Prototyping
Post-It Prototyping
PowerPoint Prototyping
High Fidelity Prototyping
Wikitude
125. INITIAL SKETCHES - Photoshop
Pros:
• Good for idea generation
• Cheap
• Concepts seem feasible
Cons:
• Not great feedback gained
• Photoshop not fast enough
for making changes
126. POST IT PROTOTYPING
First
DraJ
Camera
View
with
3D
Second
DraJ
Third
DraJ
•
Selec;on
highlighted
in
blue
•
Home
bu)on
added
for
easy
naviga;on
to
main
menu
127. POWERPOINT PROTOTYPING
Benefits
• Used for User Testing
• Interactive
• Functionalities work
• Quick
• Easy arrangement of slides
User Testing
• Participants found
• 15 minute sessions screen captured
• ‘Talk Allowed’ technique used
• Notes taken
• Post-Interview
131. FINAL DESIGN CONCEPT
Key Issues
Fix issues found in previous sessions
Design with guidelines in mind
Appealing to the Mental Model
Icon Design
Aesthetic Design
Colour/Font
Buttons
Look
132. VIDEO PROTOTYPE
Flexible
tool
for
capturing
the
use
of
an
interface
Elaborate
simula;on
of
how
the
naviga;onal
aid
will
work
Does
not
need
to
be
realis;c
in
every
detail
Gives
a
good
idea
of
how
the
finished
system
will
work
133. Project List
Mobile
Hybrid Tracking for Outdoor AR
City Scale AR Visualization
Outdoor AR Authoring Tool
Outdoor AR collaborative game
AR interaction for Google Glass
Non-Mobile
AR Face Painting
AR Authoring Tool
Tangible AR puppeteer studio
Gesture based interaction with AR content
134. BuildAR Assignment
Download and install Build AR (or HIT Lab NZ)
http://www.buildar.org/
Build two AR scenes
Marker based
- at least 3 markers, 3D models, same theme
Image based
- at least 2 images, 3D model, text, image/video
Due August 16th
135. Project Assignment
Design/Related work exercise
Individual
Each person find 2 relevant papers/videos/websites
Write two page literature review
As a team - prototype design
Sketch out the user interface of application
Design the interaction flow/Screen mockups
3 minute Presentation in class August 16th