Lecture 5 of the COSC 426 graduate course on Augmented Reality. This lecture provides an overview of Mobile Augmented Reality. The Lecture is given by Mark Billinghurst of the HIT Lab NZ at the University of Canterbury

1. Lecture 5: Mobile AR
Mark Billinghurst
mark.billinghurst@hitlabnz.org
k billi h t@hitl b
HIT Lab NZ
University of Canterbury

2. AR on mobile phones
Low cost, widely spread platform
Billions of phones deployed
People know how to use them
Strong demand from commercial side
Huge chance for AR!
Target practical applications
Easy to use
Quality graphics
Robust tracking
15-30 Hz overall frame rate

3. Why would you use phones?
Robust and fool-proof
People know how to use their mobile phones…
Variety of supported devices
Self contained
Self-contained operation
Enough processing power to do everything natively
Ultra-mobile
Ultra mobile
Low cost devices
Even better: people already own the target hardware!
!
A very unique chance for bringing AR to the masses

4. Why would you not use phones?
Compared to PC-based setups
Less processing power and memory
p gp y
Harder to program, debug and deploy
Hardware difficult or impossible to extend
Small number of available libraries
Typically l little
T i ll only li l experience i research groups
i in h
So many different devices and operating systems

5. Other limitations of handheld AR
Usual limitations in mobile HCI
Small screen
- Less information possible to display
- Less immersion
Limited input
…
Other limitations
You see through the camera and not through the phone
Switch attention between background and phone
Strain factor of holding the phone up
Social issues with pointing the phone at people
p g p p p

6. Mobile AR History

7. Evolution of Mobile AR
E l fM bl
Camera phone
Camera phone
- Thin client AR
Wearable Wearable AR
Computers Camera phone
- Self contained AR
PDAs
Handheld -Thin client AR
AR Displays
p y
PDAs
-Self contained AR
1995 1997 2001 2003 2004

8. Handheld Displays
Tethered Applications
Fitzmaurice Chameleon (1994)
( )
Rekimoto’s Transvision (1995)
Tethered LCD
PC Processing and Tracking

9. Handheld AR Display - Tethered
1995, 1996 Handheld AR
ARPad,
ARPad Cameleon
Rekimoto’s NaviCam, Transvision
Tethered LCD
PC Processing and Tracking

10. Mobile AR: Touring Machine (1997)
Columbia University
Feiner, MacIntyre, Höllerer, Webster
Combines
See through head mounted display
GPS t ki
tracking
Orientation sensor
Backpack PC (
p (custom)
)
Tablet input

11. MARS View
Virtual tags overlaid on th real world
Vi t l t l id the l ld
“Information in place”

12. Backpack/Wearable AR
1997 Backpack AR
Feiner’s Touring Machine
AR Quake (Thomas)
Tinmith (Piekarski)
MCAR (Reitmayr)
Bulky, HMD based

13. Mobile AR - Hardware
RTK correction Antenna
GPS
Example self-built working
Antenna
solution with PCI-based 3D graphics
PCI based
HMD
Controller PCI 3D Graphics Board
Tracker
Controller
PC104 Sound Card
DC to DC
Wearable
Converter CPU
Computer
PC104 PCMCIA
Battery
GPS RTK Hard Drive
correction
Radio
Serial
Ports
Columbia Touring Machine

14. Sharp J-SH04
1997 Philip Kahn invents camera phone
1999 First commercial camera phone

15. Millions of Camera Phones
1200
1000
800
DSC
600
Phone
400
200
0
2002 2003 2004 2005 2006 2007 2008 2009 2010

16. Handheld AR – Thin Client
2001 BatPortal (AT&T Cambridge)
PDA used as I/O device
Wireless connection to workstation
Wi l i k i
Room-scale ultrasonic tracking (Bat)
2001 AR-PDA (C Lab)
PDA thin graphics client
Remote image processing
www.ar-pda.com

17. Mobile Phone AR – Thin Client
2003 ARphone (Univ. of Sydney)
Transfer images via Bluetooth (slow – 30 sec/image)
g ( g )
Remote processing – AR Server

18. Early Phone Computer Vision Apps
2003 – Mozzies Game - Best mobile game
Optical motion flow detecting phone orientation
Siemens SX1 – S bi 120Mh VGA Camera
Si Symbian, 120Mhz, C
2005 – Marble Revolution (Bit-Side GmbH)
Winner of Nokia's Series 60 Challenge 2005
Wi f N ki ' S i Ch ll
2005 – SymBall (VTT)

19. Computer Vision on Mobile Phone
Cameras and Phone CPU sufficient for computer vision
applications
Pattern Recognition (Static Processing)
g ( g)
QR Code
Shotcode (http://www.shotcode.com/)
Motion Flow (2D I
M i Fl Image P
Processing)
i )
GestureTek
- http://www.gesturetekmobile.com/
TinyMotion
3D Pose Calculation
Augmented Reality

20. Handheld AR – Self Contained
2003 PDA-based AR
ARToolKit port to PDA
Studierstube ported to PDA
AR Kanji Educational App.
Mr Virtuoso AR character
Wagner’s Invisible Train
- Collaborative AR

21. Mobile Phone AR – Self Contained
2004 M bil Phone AR
Mobile Ph
Moehring, Bimber
Henrysson (ARToolKit)
Camera, processor, display together

22. Location Aware Phones
Motorola Droid Nokia Navigator

23. Real World Information Overlay
Tag real world locations
GPS + Compass input
Overlay graphics data on live video
Applications
Travel guide, Advertising, etc
Eg: Mobilizy Wikitude (www.mobilizy.com)
Android based, Public API released
Other companies
Layar, AcrossAir, Tochnidot, RobotVision, etc

24. Layar – www.layar.com

25. HIT Lab NZ Android AR Platform
Architectural Application
Loads 3D models
a OBJ/MTL format
Positions content in space
p
GPS, compass
Intuitive user interface
toolkit to modify the model
Connects to back end model database

26. HIT Lab NZ Mobile Outdoor AR

28. History of Handheld and Mobile AR
1995 Handheld Display: NaviCam, AR-PAD, Transvision
1997 Wearable AR: Touring Machine, AR Quake
g
2001 Handheld AR – Thin Client: AR-PDA, Bat Portal
2003 Handheld AR – Self contained: Invisible Train
2003 Mobile Phone – 2D Vision: Mozzies, Symball
2003 Mobile Phone – Thin Client: ARphone
2004 Mobile Phone – Self contained: Moehring, Symbian

29. 1996 Mobile AR by Weight
2003
2007
Scale it down: Scale it down more:
Vesp R
Vesp‘R [Kruijff ISMAR07]: Smartphone…$500
S t h $500
…Sony UMPC 1.1GHz …All-in-one
Backpack+HMD: …1.5kg …0.1kg
…5-8kg
5 8k …still >$5K
till …billions of units

30. 2011 S
State of the Art
f h A
Handheld Hardware available
PDA, mobile phones, external cameras
Sensors: GPS, accelerometer, compass
Software Tools are Available
Tracking: ARToolKitPlus, QCAR
Graphics: OpenGL ES
Authoring: Studierstube, Layar, Wikitude, Unifye
What is needed:
High level authoring tools
Content development tools
Novel interaction techniques
User evaluation and usability

31. Mobile AR Companies
Mobile AR
GPS + compass
Many Companies
M C i
Layar
Wikitude
Acrossair
PressLite
Yelp
Robot vision
Etc..
E

32. $2 million USD in 2010
$732 million USD in 2014

33. Handset Manufacturers
Qualcomm
$100 million USD investment
Nokia
N ki
25+ people in NRC
Samsung
Exploring the space
Apple
586 AR Applications on App Store
Google
Google goggles/Android AR Applications
g g gg pp

34. Qualcomm
Acquired Imagination
q g
October 2010 - Releases free Android AR SDK
Computer vision tracking - marker, markerless
p g
Integrated with Unity 3D renderer
http://developer.qualcomm.com/ar
p p q

35. Rock-em Sock-em
Shared
Sh d AR D Demo
Markerless tracking

36. Apple
iPhone 4 SDK supports direct camera access
pp
Launches AR theme on App Store

37. > 500 AR apps on App Store

38. Developing AR applications

39. Mobile AR Technology
Involves
Tracking
g
Content Loading
Rendering/3D graphics
g g p
User Interface
Application Design
Evaluation

40. Scientific challenges
AR requires (unlike related disciplines)
Strict real time operation (30Hz)
- Unlike Ubicomp or mobile information systems
p y
High spatial precision (1cm, 1 degree)
- Unlike location-based services
Robustness for operation by human user
- Unlike many computer vision methods in automation etc.
Mobile h
M bil phone AR requires (in addition)
i (i ddi i )
No thin client!
Same l l of performance as desktop AR
S level f f d k
- New algorithms must be orders of magnitude more efficient
No unrealistic assumptions about HW

41. How does a basic AR application work?
Main loop
Get a video frame from the camera
Estimate the position and the orientation
- computer vision, sensor input (GPS, compass)
p p ( p )
Render the augmented scene (video + virtual
content)
)
Render GUI
Process user input
Update application status

42. Studierstube ES Framework
Typical AR application
Applications
framework
Developed at TU
Studierst
Graz
Platformtube Software Stack
a

43. End User
Application
pp
Programming
Libraries
Lib i
OS/Low Level API
Hardware

44. The Studierstube ES framework
Th S d b f k
Applications
o
User Interface - Application
Studie
Content
erstube Software S
Graphics
Stack
Tracking
Platform
Platform

45. Platform

46. What are the challenges?
Experiences with embedded platforms required
Many platforms (operating systems)
M l f ( i )
Slow CPUs (low clock rates, often no FPU)
Difficulties with tracking
Difficulties with visualizations that require a lot of data
processing
i
Slow memory access
No or weak hardware 3D acceleration
Difficulties with graphics

47. Processing power of mobile phones
Weak processing power
~1GHz,
~1GHz Single core
Often no floating point unit
- Floating po t co e ~40x s owe t a integer co e
oat g point code 0 slower than tege code
- Fixed point problematic for many algorithms
- Requires good math library
Code optimized for phones runs 5-10x slower on a
high-end phone than on an average PC
Not going to change dramatically due to battery power

48. So what are the common problems?
Bad camera quality under low lighting
Noise, motion blur
,
Strongly varies with different phones
Small memory
Keeping large databases in memory is problematic
Slow memory
Low clock rate
- Processing large memory areas is prohibitive
g g y p
- Typical CV building blocks (SVD, image processing) are too demanding
Slow data transfers between CPU and GPU

49. Platforms for handheld AR
Pl f f h dh ld
Pros Cons
Windows Easy to program and debug
Camera drivers are not always good
Mobile Lots of devices
Largest installed basis Hard to program and debug, SDK changes
Symbian
Good devices often, usually slow CPUs
Very nice hardware
Very nice hardware Camera API only with
Camera API only with OS4
iPhone
Hype factor Objective C as main language
Increasing number of devices
Android Java as main language
Java as main language
Hype factor
Linux (LiMo, Full Linux support, limited
Large set of different libraries
Large set of different libraries
Maemo)
M ) handsets for AR
h d f AR
RIM
Widely spread in the US Java only
Blackberry
Palm Only very few devices so far
Nice hardware
WebOS No native SDK so far

50. Worldwide smartphone market share
1Q09 1Q10
Symbian
RIM
iPhone
Android
Source: Gartner
http://www.gartner.com

51. 2011 US Market Share

52. What makes a device interesting for AR?
Open and easy to program
Good camera
Fast CPU, FPU is a plus
Good H/W 3D support
Large installed basis
g
Easy access to devices
GPS, accelerometer, compass
Enough memory/storage

53. Typical Smart Phone Hardware
CPU
300-800+ Mhz
GPU
None, or Power VR Chip (OpenGL ES1.0/2.0)
Input
p
Touch screen, keyboard, keypad
Sensors
GPU,
GPU compass, accelerometer, camera (1.3-5mb+)
l (1 3 5 b )
Networking
Bluetooth, Wifi,
Bluetooth Wifi 3G
Screen
320x240 up to 800x480

54. HTC HD2
Windows Mobile
Fast CPU (1GHz)
Big screen
4.3”, 800x480
GPS, compass and accelerometer
Good camera
Depends on lighting conditions
Hardware 3D
Slow texture upload:
slow video background rendering
video-background

55. HTC D
Desire
Android
Fast CPU (1GHz)
Smaller screen
3.7”, 800x480
Multi-touch
Hardware 3D
GPS, compass and
accelerometer

56. iPhone 4
Ph
Apple iOS 4 0
4.0
Faster CPU (1.2GHz)
High
Hi h screen resolution
l i
3.5”, 960x640
(Finally) camera API
Multi-touch
Hardware 3D
GPS, compass
GPS compass, accelerometer
and gyroscope

57. Hardware Sensors
Camera (resolution, fps)
C ( l i f )
Maker based/markerless tracking
Video
Vid overlap
l
GPS (resolution, update rate)
Outdoor location
Compass
Indoor/outdoor orientation
Accelerometer
Motion sensing, relative tilt
sensing

58. Programming Environments

59. Mobile Development Environments
Not like developing for desktops
Wide range of different OS
Limited CPU, low memory, poor graphics, no floating point
Popular Mobile OS
Symbian (S bi C++, C bid
S bi (Symbian C++ Carbide IDE)
iPhone (Objective C)
Android (Java, Native NDK wrapper)
(J pp )
Windows Mobile (C#, C++, Visual Studio)
Other
Palm OS, Blackberry, Linux

60. Programming Windows Mobile
Very similar to desktop Windows
Almost identical API
Unicode functions only
U d f l
Development tools
Embedded Visual C++ C#
C++,
- Deprecated, not suggested
Visual Studio 2005
Visual Studio 2008
- Required for FPU usage
For
F overview of camera bugs look at
i f b l k t
http://studierstube.org/handheld_ar/camera_phones.php

61. Programming Symbian
Development tools
Carbide.c++
Commercial version required for on-device debugging (important
since emulator is bad…)
SDK appropriate f your d
for device
Many quirks
Crippled
C i l d C++ support t
Writing to static variables not allowed/recommended
Cleanup Stack
p
API includes ~1500 classes
Moving to Qt
g

62. Programming iPhone
Harsh restrictions from Apple
Apps have to go through the apps store
Xcode
X d IDE for development
f d l
Nice development tools
Objective-C
Objective C
Required for application development
Can call into C/C++ code
C/C
Camera API support in iOS 4+
Can overlay on live video
y

63. Android
Hardware creators
HTC, LG, Samsung, Motorola
Widely
Wid l available phone
il bl h
Different form factors – tablets, phones, PC, etc
Multiple versions and fragmentation
Android 1.0, 1.6
Android 2.0, 2.1
Free Tools
Eclipse Development
App Integrator

64. Mobile Graphics

65. Computer Graphics on Mobile Phones
Small screen, limited input options
Limited support for accelerated graphics
Most phones have no GPU
Mobile Graphics Libraries
OpenGL ES ( , 2.0)
p (1.0, )
- Cross platform, subset of OpenGL
- C/C++ low level library
Java M3G
- Mobile 3D graphics API for J2ME platform
- Object importer, scene graph library
- Support from all major p
pp j phone manufacturers

67. OpenGL
O GL ES
Small-footprint
Small footprint subset of OpenGL
OpenGL is too large for embedded devices!
Powerful, l
P f l low-level API full functionality for 3D games
l l API, f ll f i li f
Can do almost everything OpenGL can (but only one way)
Available
A l bl on all key platforms
ll k l f
Software and hardware implementations available
Fully
F ll extensible
ibl
Extensions like in OpenGL
No redundancy!
Convenience functions removed

68. OpenGL ES
SLIDE 68
OpenGL ES vs. OpenGL (1.x)
OpenGL OpenGL ES
glBegin/glEnd 1
Primitive Types all no quads & polygons
Data Types float, double int etc
float double, int, etc… float,
float fixed
glDraw/Read Pixels glReadPixels only
Textures
T t 1D, 2D, 3D, cube 2D
Stencil optional
Window Bindings WGL, GLX, etc… EGL
1: Except for Security Critical profile

69. OpenGL ES
SLIDE 69
OpenGL ES on mobile devices
Java Applications C++ Applications
Scenegraph API
S h Game
G Middleware
Middl
M3G (JSR 184) Engine Engines

70. OpenGL ES Performance
Faster graphics (esp. hardware accelerated)
( )
Longer batter performance (> 10%)

71. Versions
Two major tracks
Not compatible, parallel rather than competitive
OpenGL ES 1 x
1.x
Fixed function pipeline
Suitable for software implementations
All 1.x are backwards compatible
OpenGL ES 2 x
2.x
Vertex and pixel shaders using GLSL ES
All 2.x will be backwards compatible
2x

72. Fixed Function (1 )
F dF (1.x)
http://www.khronos.org/opengles/2_X/

73. Programmable (2.x)
P bl (2 )
http://www.khronos.org/opengles/2_X/

74. OpenGL ES 1.x vs 2.0

75. Tracking

76. Mobile Augmented Reality’s goal
Create an affordable, massively multi-user, widespread platform
© Tinmith, U. of South Australia

77. Tracking is…
Estimating the device‘s pose (position and orientation)
Strictly in real time (30Hz)
With high spatial precision (1cm, 1 degree)
Robustly for operation by human user
No unrealistic assumptions about HW
Leaving enough power to other tasks (interaction graphics)
(interaction,

78. Tracking requirements for AR on phones
Fast and efficient
Form factor: light and robust
Track simultaneously
A large number of objects
By a large number of users
Requires little or no …
q
Device modification
Manual calibration (targeting non-technical users)
( g g )
Instrumentation of the physical environment
Low costs

79. Tracking on mobile phones
Vision-based tracking
Marker-based tracking
Model-based natural feature tracking
Natural feature tracking in unknown environments
Sensor tracking
GPS, inertial compass, gyroscope

80. Tracking for Handheld AR
SLIDE 80
Backpack-based 1.
Höllerer et al. (1999), Piekarski & Thomas al. (2001), Reitmayr & Schmalstieg (2003)
( ), ( ), y g( )
Laptop, HMD
Enhanced GPS (DGPS / RTK) + inertial sensor for viewpoint tracking
Hand tracking w/ fiducial markers

81. Tracking for Handheld AR
SLIDE 81
Backpack-based 2.
Kalkusch et al., 2002
Video see-through HMD w/ camera
g
Viewpoint Tracking w/ inside-out computer vision using markers
ARToolKit markers on walls installed and surveyed manually
y y

82. Tablet PC / UMPC-based 1.
Schall et al., 2006
Hybrid tracking on UMPC
Camera fiducial marker trackingg
When no marker in view inertial sensor + UWB tracking

83. Tablet PC / UMPC-based 2.
CAMERA
LEDs
Klein & Drummond, 2004
Combining outside in (LED tracking for low accuracy robust pose) &
outside-in accuracy,
inside-out (edge-based tracking for high accuracy) computer vision

84. PDA-based 1.
BatPortal (Newman et al., 2001)
( , ) SHEEP (MacWilliams et al., 2003)
PDA as thin client (rendering &
Tracking by ART (external IR
tracking on server + VNC) cameras + retroreflective target)
Ultrasonic tracking Projection-based AR environ.

85. PDA-based 2.
Signpost on PDA (Wagner & Schmalstieg, 2003)
First “truly” handheld AR platform: PDA + camera
Standalone, self-contained
Standalone self contained AR system
Optimized fiducial marker tracking library

86. History of non-AR Tracking on Phones (1)
AR-PDA (Gausemeier et al., 2003) Kick Real (Paelke et al., 2004)
Model-based
Model based tracking Edge detection of real foot + collision
PDA = thin client detection w/ virtual ball
tracking off-loaded to server 2D tracking and limited interaction
Not real-time (tailored to game)

87. History of non-AR Tracking on Phones (2)
PhoneGuide (Bruns et al., 2005) LightSense (Olwal, 2006)
Neural network for recognizing visual External camera tracks cell phone LED
features of museum artifacts Single user, only coarse position
Combined w/ BT “tracker” tracking, no orientation
Only object recognition
O Border-case
Border case of AR

88. History of non-AR Tracking on Phones (3)
Mosquito Hunt (Siemens, 2003)
Marble Revolution (BitSide, 2004)
Pingis (VTT,
Pi i (VTT 2006)
TinyMotion (Wang et al., 2006)
Game control w/ optical GUI control & input on cell phones
flow techniques w/ image differencing & block correlation

89. ARToolKit Tracking (Kato)
ARToolKit - Computer vision based marker tracking libraries
http://artoolkit.sourceforge.net/

90. History of AR Tracking on Phones (1)
2003
ARToolKit on PDA
Wagner et at.
2004
3D Marker on Phone
Möhring et al.
g
2005
ARToolKit on Symbian
Henrysson et al.

91. Tracking for Handheld AR
SLIDE 91
Fiducial marker tracking on handhelds
Wagner et al., 2003 Möhring et al., 2004 Henrysson et al., 2006
Bucolo et al., 2005 Rohs, 2006

92. History of AR Tracking on Phones (2)
2005
Visual Codes
Rohs et at
at.
2008
Advanced Marker Tracking
Wagner et al.
2008
Natural Feature Tracking
Wagner et al.

93. What can we do on today‘s mobile phones?
Typical specs
600+ MHz
~5MB of available RAM
160x120 - 320x240 at 15-30 Hz camera
Possible to do
Marker tracking in 5-15ms
Natural feature tracking in 20-50ms

94. Handheld AR Interfaces

95. Handheld HCI
Consider your user
Follow good HCI principles
Adapt HCI guidelines for handhelds
Design to device constraints
Rapid prototyping
User evaluation

96. Consider Your User
■ They are probably mobile
able to use the interface with one hand
■ They want quick access to application content
content.
Want enhanced interaction with the real world
Interaction with the real world is the main focus
■ They expect to be able to multitask
start phone call, use another application, etc
p pp

97. Norman’s Principles of Good Practice
Ensure a high degree of visibility
- allow the user to work out the current state of the system and the
range of actions possible.
f ti ibl
Provide feedback
- continuous, clear information about the results of actions.
Present a good conceptual model
- allow the user to build up a picture of the way the system holds
together, the relationships b
h h l i hi between its different parts and h
i diff d how to
move from one state to the next.
Offer good mappings
g pp g
- aim for clear, natural relationships between actions the user performs
and the results they achieve.

98. High L l D i Guidelines
Hi h Level Design G id li
From Shneiderman’s 8 desktop design g
p g guidelines:
Enable Frequent Users to Use Shortcuts
Offer Informative Feedback
Design Dialogs to Yield Closure
Gong and Tarasewich’s guidelines:
G dT i h’ id li
Design for Small Devices
Design for Limited and Split Attention
Design for speed and recovery
Allow for personalization
Design for Enjoyment

99. UI Device Constraints
Comparing Desktop to Handheld Interfaces
Screen Size Input Operation Multimedia Connectivity
Desktop > 1024 x 768 Mouse Two handed Millions of colours Wired
Keyboard Stationary Graphics accel. Always On
5.1 Audio
Handheld < 640 x 480 Stylus One handed 65K colours Wireless
Touch Mobile No graphics accel. Maybe On
Buttons Stereo audio

100. Example:
E m l O2 Active M
A ti Menu
Highly visual
Use PDA buttons for input
Large icons and easy to read text
Visually indicate which tabs are scrollable
Application UI looks different from device UI

101. iPhone Guidelines
Minimize required user input.
Avoid unnecessary interactivity.
y y
Provide feedback when necessary
Provide fingertip sized target areas
fingertip-sized areas.
Avoid clutter and busy backgrounds.
Express essential information succinctly.
Make it obvious how to use your content.
y

102. iPhone Interface

103. Designing for Device Constraints
Input Device
Touch, stylus, keyboard, buttons, keypad
y y yp
Screen
Size, resolution
,
Sensors
Camera – frame rate image resolution
rate,
GPS – resolution, coverage
Co pass accu acy
Compass - accuracy

104. Sample Handheld AR Interfaces
Clean
Large Video View
Large Icons
Text Overlay

105. Twitter 360
www.twitter-360.com
pp
iPhone application
See geo-located tweets in real world
Twitter.com
Twitter com supports geo tagging

106. Wikitude – www.mobilizy.com
Blah
Blah Blah Blah
Blah Blah Blah
Blah
Blah
Blah
Bl hh
Blah
Bl Blah
Bl h
Blah Blah Blah Blah
Blah Blah
Blah
Blah Blah Blah

107. Information Filtering

108. Information Filtering
Information Filtering (Julier et al. ’00)
al
• Remove clutter by goal- and distance based filtering
• User’s task is route finding: Sniper and relevant buildings are displayed;
objects, which are determined to be unnecessary, removed
bj t hi h d t i dt b d

109. HMD vs Handheld AR Interface
Wearable AR
W bl HandHeld AR
Output:
Display Input &
Output
Input

110. Handheld Interface Metaphors
Tangible AR Lens Viewing
Look through screen into AR scene
Interact with screen to interact with AR
content
- Eg Invisible Train
Tangible AR Lens Manipulation
Select AR object and attach to device
Use the motion of the device as input
- Eg AR Lego

112. Handheld Display vs Fixed Display
Experiment comparing handheld moving, to handheld button input, small
fixed display, desktop display, large plasma
Users performed (1) navigation task, (2) selection task
Moving handheld display provided greater perceived FOV, higher degree of
presence, faster completion time
J. Hwang, J. Jung, G. Kim. Hand-held Virtual Reality: A Feasibility Study. In proceedings of VRST 2006

113. Search Task Completion Time

114. FOV, Presence and Immersion
Perceived FOV and Actual FOV (deg. marked by subjects)
70 64
58 60
60 52
50 45
40 30 33 30 31 30 Perceived FOV
30 Actual FOV
20
10
0 7
Motion Button Small 17' screen 42' screen 5.7
6 5.4
based hh based hh screen
5 4.7 4.7 4.9
4.3 4.5 4.3 4.3
4.1
4 Presence
3 Immersion
2
1
0
Motion Button Small 17' screen 42' screen
based hh based hh screen

115. Rapid Prototyping
Speed development time by using quick hardware mockups
p p y gq p
handheld device connected to PC
LCD screen
USB phone keypad
Camera
Can use PC development tools
Flash, Visual Basic, etc

116. Mobile Physical Prototyping
Bug Labs
http://www.buglabs.net/
Open source hardware modules, each
p
producing one or more services.
g
Modules snap together physically and the
services connect together logically to
i h l i ll
enable users to easily build applications.

117. Software Prototyping
Python Symbian (HIT Lab NZ)
stbTracker wrapper
Access to SMS, Bluetooth, GPS
Rapid development

118. import e32
import appuifw
from gles import *
if e32.s60_version_info>=(3,0):
import imp
magnet=imp.load_dynamic('Magnet', '
t i l d d i ('M t' 'c:sysbinMagnet.pyd')
bi M t d')
else:
import Magnet
from Magnet import
#Define Model
def frameback(num_markers):
if (num markers > -1):
(num_markers 1):
glMatrixMode(GL_PROJECTION)
#Draw graphics
…
appuifw.app.orientation = 'landscape‘ # Use full frame
SetCameraCallback(frameback) # Register callback
createCamera() # Define camera
InitGLES() # Start Open GL
TrackerInit() # Start tracker
InitCamera() # Start camera

119. Design Guidelines
Apply handheld HCI guidelines for on screen content
on-screen
- large buttons, little text input, etc
Design physical + virtual interface elements
Pick appropriate interface metaphor
pp p p
- “handheld lens” approach using handheld motion
- Tangible AR for AR overlay
Build prototypes
Continuously evaluate application

120. AR Browsers

121. AR Browsers
Commercial outdoor AR applications
Junaio, Layar, Wikitude, etc
All have their own language specifications
Wikitude – ARML
Junaio - XML
Need for common standard
Based on existing standards for geo-located content etc
Support for dynamic/interactive content
Easier to author mobile AR applications
Easy to render on AR browsers

122. Layar

129. Junaio

133. Hello World Example
echo ""<?xml version="1.0" encoding="UTF-8"?>
" " " "
<results>
<poi id="1" interactionfeedback="none">
<name><![CDATA[Hello World POI]]></name>
<description><![CDATA[[This is my first POI.]]></description>
<l>48.1385,11.5750,0</l>
, ,
<o>0,0,0</o>
<mime-type>text/plain</mime-type>
<thumbnail>http://www.junaio.com/publisherDownload/tutorial/icon.jpg</
<thumbnail>http://www junaio com/publisherDownload/tutorial/icon jpg</
thumbnail>
<icon>http://www.junaio.com/publisherDownload/tutorial/icon_map.png</
icon>
</poi>
</results>"

145. KHARMA + Argon: A KML/HTML
Architecture and Browser for
AR Applications and Games
Blair MacIntyre and Al Hill
Bl i M I d Alex
School of Interactive Computing,
Georgia Institute of Technology 145

146. KHARMA:
KHARMA KML/HTML Augmented Reality Mobile Architecture
research tools { media hackers
our past focus
{ skilled computationalists
savvy technical designers
Ygrasil IDE
current focus { general public
breadth of adoption
Unity AR Toolkit
Designers AR Toolkit
KHARMA 146

147. KHARMA: KML/HTML Augmented Reality Mobile Architecture
Problem: limited authoring tools for mobile AR
limited expression (coord, name, desc, link) vs higher hurdle of 3D
no accepted standard and proprietary client protocol
limited client-side interactivity is more akin to Web 1 0
1.0
Solution: HTML with KML combines What? and How? with Where?
• allows extensive client side (albeit 2D) interactivity and expressivity
• two the most broadly adopted standards for presentation and geo location
• HTTP server distribution, CSS and Javascript allow for true Web 2.0 content
+
KML
HTML
147

148. KHARMA
Architecture with four components
Channel servers
- delivering individual channels of AR content,
Tracking servers
- providing content related to location
Tracking,
Tracking infrastructure servers
- delivering information about the physical environment,
Mobile client
- for generating the resulting augmentations

150. KML already supports HTML
• description tag accepts CDATA enclosed markup
- but no global styling of scripting support
• no control over balloon styling
• no control over balloon position and orientation
• no relative positioning <Placemark id="culc_center">
<name>CULC Visualization</name>
<description><![CDATA[
<div id="culc_image">Georgia T h>
<di id " l i ">G i Tech>
<img src="http://www.culc.net/culc.png"></div>
</description>
<Balloon>
<location>
<latitude>34.0</latitude>
<longitude>--84.5</longitude>
</location>
<orientation>
<heading>31</heading>
</orientation>
</Balloon>
</Placemark>

151. KARML extension to KML
<Placemark id="culc_center">
<name>CULC Visualization</name>
• added undecorated displayMode <description><![CDATA[
<div id="culc_image"><img src="http://www.culc.net/culc.png"></div>
</description>
<Style>
• added Balloon element <BalloonStyle>
<displayMode>undecorated</displayMode>
- similar in nature to Model element </BalloonStyle>
</Style>
<Balloon>
<locationMode targetId=”culc_g
g geospot”>relative</locationMode>
p
• relative locationMode <location>
- accepts “units” attribute <latitude units="meters">4.0</latitude>
<longitude units="meters">-2.5</longitude>
</location>
<orientation>
<heading>31</heading>
</orientation>
</Balloon>
</Placemark>
151

152. GPS Located Content
<Placemark>
Geospot <name>GeoSpot</name>
<description>a surveyed GeoSpot </description>
Surveyed location <Camera> <!-- camera element for GeoSpot -->
p
Moves camera to <longitude>-84.394135</longitude> <!- GPS coordinates -->
<latitude>33.76083</latitude>
fixed location <altitude>0</altitude>
<TimeStamp> <! when GeoSpot was surveyed -->
<!-- >
<when>1997-07-16T10:30:15+03:00</when>
</TimeStamp>
<Icon>
<href>http://geospot.imtc.gatech.edu/image/03_icon.png</href>
</Icon>
</Camera>
<Point> <! location displayed on map or within browser view -->
<!-- >
<coordinates>-84.394135,33.76083</coordinates>
</Point>
</Placemark>

153. For more information, please visit:
http://www.argon.gatech.edu/
http://www argon gatech edu/

154. Developing AR Experiences
Sony CSL © 2004

166. Game Case Study

180. Resources

181. Resources
Books
Mobile Interaction Design
Matt J
M Jones and Gary Marsden
dG M d
Designing for Small Screens
Studio 7.5
Handheld Usability
Scott Weiss
Designing the Mobile User Experience
Barbara Ballard

182. Developer Guidelines
Palm
http://www.access-company.com/developers/documents/docs/ui/UI_Design.html
Zen of Palm guidelines
http://www.access-company.com/developers/documents/docs/zenofpalm.pdf
Motorola
http://developer.motorola.com/docstools/developerguides/
iPhone Human Interface Guidelines
http://developer.apple.com/documentation/iPhone/Conceptual/iPhoneHIG/

183. Handheld HCI Design Websites
Do’s and Don’ts of PocketPC design
http://www.pocketpcmag.com/_archives/Nov04/Commandements.aspx
Usability
U bili special i
i l interest group – h dh ld usability
handheld bili
http://www.stcsig.org/usability/topics/handheld.html
Usable Mobile website
http://www.smartgroups.com/groups/usablemobile
Mobile Coders Website
http://www.mobilecoders.com/Articles/mc-01.asp
http://www mobilecoders com/Articles/mc-01 asp
Univ of Waikato Handheld Group
http://www.cs.waikato.ac.nz/hci/pdas.html
Mobile Interaction Website
http://www.cs.waikato.ac.nz/~mattj/mwshop.html

184. Platform – Recommended reading
Lots of low level information
on the complete ARM family
p y
Valuable tool for driver and
framework developers
Not that important for pure
application developers

185. Platform – Recommended reading
Very low level and targeted for PCs
Most information outdated on PC
Effective memory usage one
of the most important
optimization strategies
on mobile devices!

186. Tracking – Recommended reading
Lots of the basics on the
Computer Vision you will
p y
need for AR tracking
Several code and pseudo-code
pseudo code
snippets

187. Tracking – Recommended reading
All about the geometry
you will need for
a tracking system
Camera models
Projection
Epipolar geometry
Homographies
H hi
…

188. Graphics – Recommended reading
Mobile 3D Graphics
(all b t O GL
( ll about OpenGL ES 1.x)
1 )
OpenGL ES 2.0
Programming G Guide
OpenGL ES 2.0 Man Pages
http://www.khronos.org/opengles/sdk/docs/man/
ShaderX7
Chapter on “ Augmented Reality on Mobile Phones”

189. OpenGL ES Resources
Khronos Group OpenGL ES Page
http://www.khronos.org/opengles/
OpenGL ES 2.0 Book
http://www.opengles-book.com/
AMDs OpenGL ES 2.0 Emulator