A lecture on Mobile Augmented Reality. A lecture given by Mark Billinghurst at the University of Canterbury on Friday September 13th 2013. This is part of the COSC 426 graduate course on Augmented Reality.
5. Mobile Phone AR
Mobile Phones
camera
processor
display
AR on Mobile Phones
Simple graphics
Optimized computer vision
Collaborative Interaction
6. 2005: Collaborative AR
AR Tennis
Shared AR content
Two user game
Audio + haptic feedback
Bluetooth networking
9. Evolution of Mobile AR
Wearable AR
Handheld
AR Displays
Camera phone
1995 1997 2001 2003 2004
Camera phone
- Self contained AR
Wearable
Computers
PDAs
-Thin client AR
PDAs
-Self contained AR
Camera phone
- Thin client AR
11. Handheld AR Display - Tethered
1995, 1996 Handheld AR
ARPad, Cameleon
Rekimoto’s NaviCam, Transvision
Tethered LCD
PC Processing and Tracking
12. AR Pad (Mogilev 2002)
Handheld AR Display
LCD screen
Camera
SpaceOrb 3 DOF controller
Peripheral awareness
Viewpoint awareness
13. Mobile AR: Touring Machine (1997)
University of Columbia
Feiner, MacIntyre, Höllerer, Webster
Combines
See through head mounted display
GPS tracking
Orientation sensor
Backpack PC (custom)
Tablet input
14. MARS View
Virtual tags overlaid on the real world
“Information in place”
15. Backpack/Wearable AR
1997 Backpack AR
Feiner’s Touring Machine
AR Quake (Thomas)
Tinmith (Piekarski)
MCAR (Reitmayr)
Bulky, HMD based
16. PCI 3D Graphics Board
Hard Drive
Serial
Ports
CPU
PC104 Sound Card
PC104 PCMCIA
GPS
Antenna
Tracker
Controller
DC to DC
Converter
Battery
Wearable
Computer
GPS RTK
correction
Radio
Example self-built working
solution with PCI-based 3D graphics
Columbia Touring Machine
Mobile AR - Hardware
17. 1997 Philip Kahn invents camera phone
1999 First commercial camera phone
Sharp J-SH04
20. 2003 ARphone (Univ. of Sydney)
Transfer images via Bluetooth (slow – 30 sec/image)
Remote processing – AR Server
Mobile Phone AR – Thin Client
21. Early Phone Computer Vision Apps
2003 – Mozzies Game - Best mobile game
Optical motion flow detecting phone orientation
Siemens SX1 – Symbian, 120Mhz, VGA Camera
2005 – Marble Revolution (Bit-Side GmbH)
Winner of Nokia's Series 60 Challenge 2005
2005 – SymBall (VTT)
22. Computer Vision on Mobile Phone
Cameras and Phone CPU sufficient for computer vision
applications
Pattern Recognition (Static Processing)
QR Code
Shotcode (http://www.shotcode.com/)
Motion Flow (2D Image Processing)
GestureTek
- http://www.gesturetekmobile.com/
TinyMotion
3D Pose Calculation
Augmented Reality
23. Handheld AR – Self Contained
2003 PDA-based AR
ARToolKit port to PDA
Studierstube ported to PDA
AR Kanji Educational App.
Mr Virtuoso AR character
Wagner’s Invisible Train
- Collaborative AR
24. Mobile Phone AR – Self Contained
2004 Mobile Phone AR
Moehring, Bimber
Henrysson (ARToolKit)
Camera, processor, display together
25.
26. AR Advertising
Txt message to download AR application (200K)
See virtual content popping out of real paper advert
Tested May 2007 by Saatchi and Saatchi
28. Real World Information Overlay
Tag real world locations
GPS + Compass input
Overlay graphics data on live video
Applications
Travel guide, Advertising, etc
Eg: Mobilizy Wikitude (www.mobilizy.com)
Android based, Public API released
Other companies
Layar, AcrossAir, Tochnidot, RobotVision, etc
30. HIT Lab NZ Android AR Platform
Architectural Application
Loads 3D models
a OBJ/MTL format
Positions content in space
GPS, compass
Intuitive user interface
toolkit to modify the model
Connects to back end model database
32. 1995 Handheld Display: NaviCam, AR-PAD, Transvision
1997 Wearable AR: Touring Machine, AR Quake
2001 Handheld AR – Thin Client: AR-PDA, Bat Portal
2003 Handheld AR – Self contained: Invisible Train
2003 Mobile Phone – 2D Vision: Mozzies, Symball
2003 Mobile Phone – Thin Client: ARphone
2004 Mobile Phone – Self contained: Moehring, Symbian
History of Handheld and Mobile AR
33. Mobile AR by Weight
Backpack+HMD:
…5-8kg
Scale it down:
Vesp‘R [Kruijff ISMAR07]:
…Sony UMPC 1.1GHz
…1.5kg
…still >$5K
Scale it down more:
Smartphone…$500
…All-in-one
…0.1kg
…billions of units
1996
2003
2007
34. 2013 State of the Art
Handheld Hardware available
PDA, mobile phones, external cameras
Sensors: GPS, accelerometer, compass
Software Tools are Available
Tracking: ARToolKitPlus, stbTracker, Vuforia
Graphics: OpenGL ES
Authoring: Layar, Wikitude, Metaio Creator
What is needed:
High level authoring tools
Content development tools
Novel interaction techniques
User evaluation and usability
35. Mobile AR Companies
Mobile AR
GPS + compass
Many Companies
Layar
Wikitude
Acrossair
PressLite
Yelp
Robot vision
Etc..
42. iPhone 4
Apple iOS
Faster CPU (1.2GHz)
High screen resolution
3.5”, 960x640
camera API
Multi-touch
Hardware 3D
GPS, compass, accelerometer
and gyroscope
48. Computer Graphics on Mobile Phones
Small screen, limited input options
Limited support for accelerated graphics
Most phones have no GPU
Mobile Graphics Libraries
OpenGL ES (1.0, 2.0)
- Cross platform, subset of OpenGL
- C/C++ low level library
Java M3G
- Mobile 3D graphics API for J2ME platform
- Object importer, scene graph library
- Support from all major phone manufacturers
49.
50. OpenGL ES
Small-footprint subset of OpenGL
OpenGL is too large for embedded devices!
Powerful, low-level API, full functionality for 3D
games
Can do almost everything OpenGL can
Available on all key platforms
Software and hardware implementations available
Fully extensible
Extensions like in OpenGL
52. Versions
Two major tracks
Not compatible, parallel rather than competitive
OpenGL ES 1.x
Fixed function pipeline
Suitable for software implementations
All 1.x are backwards compatible
OpenGL ES 2.x
Vertex and pixel shaders using GLSL ES
All 2.x will be backwards compatible
60. Backpack-based
Höllerer et al. (1999), Piekarski & Thomas al. (2001), Reitmayr & Schmalstieg (2003)
Laptop, HMD
Enhanced GPS (DGPS / RTK) + inertial sensor for viewpoint tracking
Hand tracking w/ fiducial markers
61. Tracking for Handheld AR
SLIDE 61
Backpack-based 2.
Kalkusch et al., 2002
Video see-through HMD w/ camera
Viewpoint Tracking w/ inside-out computer vision using markers
ARToolKit markers on walls installed and surveyed manually
62. Tablet PC / UMPC-based
Schall et al., 2006
Hybrid tracking on UMPC
Camera fiducial marker tracking
When no marker in view inertial sensor + UWB tracking
63. PDA-based 1.
BatPortal (Newman et al., 2001)
PDA as thin client (rendering &
tracking on server + VNC)
Ultrasonic tracking
SHEEP (MacWilliams et al., 2003)
Tracking by ART (external IR
cameras + retroreflective target)
Projection-based AR environ.
64. non-AR Tracking on Phones
AR-PDA (2003)
Model-based tracking
PDA = thin client
tracking on server
Not real-time
Mosquito Hunt (2003)
Marble Revolution (2004)
Pingis (VTT, 2006)
Game control w/ optical
flow techniques
TinyMotion (2006)
GUI control & input
on cell phones
w/ image differencing
& block correlation
65. History of AR Tracking on Phones (1)
2003
ARToolKit on PDA
Wagner et at.
2004
3D Marker on Phone
Möhring et al.
2005
ARToolKit on Symbian
Henrysson et al.
66. Tracking for Handheld AR
SLIDE 66
Fiducial marker tracking on handhelds
Möhring et al., 2004 Henrysson et al., 2006Wagner et al., 2003
Rohs, 2006Bucolo et al., 2005
67. History of AR Tracking on Phones (2)
2005
Visual Codes
Rohs et at.
2008
Advanced Marker Tracking
Wagner et al.
2008
Natural Feature Tracking
Wagner et al.
68. What can we do on today‘s mobile phones?
Typical specs
600+ MHz
~5MB of available RAM
160x120 - 320x240 at 15-30 Hz camera
Possible to do
Marker tracking in 5-15ms
Natural feature tracking in 20-50ms
71. Handheld HCI
Consider your user
Follow good HCI principles
Adapt HCI guidelines for handhelds
Design to device constraints
Rapid prototyping
User evaluation
72. Sample Handheld AR Interfaces
Clean
Large Video View
Large Icons
Text Overlay
73. Handheld Display vs Fixed Display
Experiment comparing handheld moving, to handheld button input, small
fixed display, desktop display, large plasma
Users performed (1) navigation task, (2) selection task
Moving handheld display provided greater perceived FOV, higher degree of
presence, faster completion time
J. Hwang, J. Jung, G. Kim. Hand-held Virtual Reality: A Feasibility Study. In proceedings of VRST 2006
77. Handheld Interface Properties
Handheld interface vs. HMD interface
Display is handheld rather than headworn
Much greater peripheral view of real world
Display and input device connected
Can move device independent of view
Phone Keypad Touch Screen
One handed input
Keypad only
Bimanual interaction
Object based interaction
Two handed input
Stylus/touch screen
Screen based input/selection
Large screen
Limited number of buttons
78. Handheld Interface Metaphors
Tangible AR Lens Viewing
Look through screen into AR scene
Interact with screen to interact with AR
content
- Eg Invisible Train
Tangible AR Lens Manipulation
Select AR object and attach to device
Use the motion of the device as input
- Eg AR Lego
79.
80. Translation Study
Conditions
A: Object fixed to the phone (one handed)
B: Button and keypad input
C: Object fixed to the phone (bimanual)
- one hand for rotating tracking pattern
81. Results – Translation
• 9 subjects – within subject design
• Timing
• Tangible fastest
• twice as fast as keypad
• Survey
• Tangible easiest (Q1)
• Keypad most accurate (Q2)
• Tangible quickest (Q3)
• Tangible most enjoyable (Q4)
• Ranking
• Tangible favored
A
B C
Rank
1.44 2.56 2.0
82. Conditions
A: Arcball
B: Keypad input for rotation about
the object axis
C: Object fixed to the phone (one handed)
D: Object fixed to the phone (bimanual)
Rotation Study
83. • Timing
• Keypad(B) and Arcball(A) fastest
• No significant survey
differences
A B C D
Rank 3.0 2.3 2.4 2.2
Results – Rotation
84. Collaborative AR
AR Tennis
Virtual tennis court
Two user game
Audio + haptic feedback
Bluetooth messaging
88. Design Guidelines
Apply handheld HCI guidelines for on-screen content
- large buttons, little text input, etc
Design physical + virtual interface elements
Pick appropriate interface metaphor
- “handheld lens” approach using handheld motion
- Tangible AR for AR overlay
Build prototypes
Continuously evaluate application
92. AR Browsers
AR equivalent of web browser
Request and serve up content
Commercial outdoor AR applications
Junaio, Layar, Wikitude, etc
All have their own language specifications
Wikitude – ARML
Junaio – XML, AREL
93. AR Browsers
Commercial outdoor AR applications
Junaio, Layar, Wikitude, etc
All have their own language specifications
Wikitude – ARML
Junaio - XML
Need for common standard
Based on existing standards for geo-located content etc
Support for dynamic/interactive content
Easier to author mobile AR applications
Easy to render on AR browsers
102. Key Features
Content provided in information channels
Over 2,000 channels available
Two types of AR channels
GLUE channels – visual tracking
Location based channels – GPS, compass tracking
Simple to use interface with multiple views
List, map, AR (live) view
Point of Interest (POI) based
POIs are geo-located content
111. Search.php
<?xml version="1.0" encoding="UTF-8"?>
<results>
<poi id="1" interactionfeedback="none">
<name><![CDATA[[Hotel Hello World]]]></name>
<description><![CDATA[[This is a beautiful, family hotel and restaurant, just around the
corner. Special Dinner and Rooms available.]]]></description>
<l>37.776685,-122.422771,0</l>
<mime-type>text/plain</mime-type>
<icon>http://dev.junaio.com/publisherDownload/tutorial/icon_map.png</icon>
<thumbnail>http://dev.junaio.com/publisherDownload/tutorial/thumb.jpg</thumbnail>
<phone>555/1234567</phone>
<homepage> http://www.hotelaroundthecorner.com </homepage>
</poi>
</results>
113. Limitations of Plain XML
No interactivity
Only simple pop-ups
No user interface Customizations
Can only use Junaio GUI elements
No local interactivity
Always needs remote server connection
115. AREL
Augmented Reality Environment Language
Overcomes limitations of XML by itself
Based on web technologies; XML, HTML5, JavaScript
Core Components
1. AREL XML: Static file, specifies scene content
2. AREL JavaScript: Handles all interactions and animation. Any
user interaction send an event to AREL JS
3. AREL HTML5: GUI Elements. Buttons, icons, etc
Advantages
Scripting on device, more functionality, GUI customization
120. Basic Interactivity
Add a button on screen to move virtual
character
Use the following
HTML: button specification
Javascript: Interaction
PHP/XML: 3D model
Junaio Tutorial 5
http://www.junaio.com/develop/quickstart/
advanced-interactions-and-location-based-
model-3ds/
124. Logic_LBS5.js - JavaScript
Create an event listener
setEventListener();
Add functionality to model object
Load model from scene
Adding model behaviours
Add functionality to GUI objects
Define the event listener
Bind model behaviours to GUI objects