This document provides an overview of mobile augmented reality (AR) and its history. It discusses the evolution of mobile AR hardware from early handheld displays to today's camera phones. It covers important milestones like the first camera phone and early AR applications. It also summarizes key technologies like computer vision, tracking, and graphics libraries that enable AR on mobile. Overall, the document traces the development of mobile AR from early prototypes to its growing commercialization and adoption in recent years.

1. COSC 426: Augmented Reality
Mark Billinghurst
mark.billinghurst@hitlabnz.org
Sept 12th 2012
Lecture 8: Mobile Augmented Reality

2. 1983 – Star Wars

3. 1999 - HIT Lab US

4. 1998: SGI O2 2008: Nokia N95
CPU: 300 Mhz CPU: 332 Mhz
HDD; 9GB HDD; 8GB
RAM: 512 mb RAM: 128 mb
Camera: VGA 30fps Camera: VGA 30 fps
Graphics: 500K poly/sec Graphics: 2m poly/sec

5. Mobile Phone AR
 Mobile Phones
 camera
 processor
 display
 AR on Mobile Phones
 Simple graphics
 Optimized computer vision
 Collaborative Interaction

6. 2005: Collaborative AR
 AR Tennis
 Shared AR content
 Two user game
 Audio + haptic feedback
 Bluetooth networking

8. Mobile AR History

9. Evolution of Mobile AR
Camera phone
Camera phone
- Thin client AR
Wearable Wearable AR
Computers Camera phone
- Self contained AR
PDAs
Handheld -Thin client AR
AR Displays
PDAs
-Self contained AR
1995 1997 2001 2003 2004

10. Handheld Displays
Tethered Applications
 Fitzmaurice Chameleon (1994)
 Rekimoto’s Transvision (1995)
 Tethered LCD
 PC Processing and Tracking

11. Handheld AR Display - Tethered
1995, 1996 Handheld AR
 ARPad, Cameleon
 Rekimoto’s NaviCam, Transvision
 Tethered LCD
 PC Processing and Tracking

12. NaviCam (Rekimoto, 1995)
Information is registered to
real-world context
 Hand held AR displays
Interaction
 Manipulation of a window
into information space
Applications
 Context-aware information displays

13. AR Pad (Mogilev 2002)
Handheld AR Display
 LCD screen
 Camera
 SpaceOrb 3 DOF controller
 Peripheral awareness
 Viewpoint awareness

14. Mobile AR: Touring Machine (1997)
 University of Columbia
 Feiner, MacIntyre, Höllerer, Webster
 Combines
 See through head mounted display
 GPS tracking
 Orientation sensor
 Backpack PC (custom)
 Tablet input

15. MARS View
 Virtual tags overlaid on the real world
 “Information in place”

16. Backpack/Wearable AR
1997 Backpack AR
 Feiner’s Touring Machine
 AR Quake (Thomas)
 Tinmith (Piekarski)
 MCAR (Reitmayr)
 Bulky, HMD based

17. Mobile AR - Hardware
GPS
Example self-built working
Antenna
solution with PCI-based 3D graphics
PCI 3D Graphics Board
Tracker
Controller
PC104 Sound Card
DC to DC
Wearable
Converter CPU
Computer
PC104 PCMCIA
Battery
GPS RTK Hard Drive
correction
Radio
Serial
Ports
Columbia Touring Machine

18. Sharp J-SH04
 1997 Philip Kahn invents camera phone
 1999 First commercial camera phone

19. Millions of Camera Phones

20. Handheld AR – Thin Client
2001 BatPortal (AT&T Cambridge)
 PDA used as I/O device
 Wireless connection to workstation
 Room-scale ultrasonic tracking (Bat)
2001 AR-PDA (C Lab)
 PDA thin graphics client
 Remote image processing
 www.ar-pda.com

21. Mobile Phone AR – Thin Client
2003 ARphone (Univ. of Sydney)
 Transfer images via Bluetooth (slow – 30 sec/image)
 Remote processing – AR Server



22. Early Phone Computer Vision Apps
2003 – Mozzies Game - Best mobile game
Optical motion flow detecting phone orientation
Siemens SX1 – Symbian, 120Mhz, VGA Camera
2005 – Marble Revolution (Bit-Side GmbH)
Winner of Nokia's Series 60 Challenge 2005
2005 – SymBall (VTT)

23. Computer Vision on Mobile Phone
 Cameras and Phone CPU sufficient for computer vision
applications
 Pattern Recognition (Static Processing)
 QR Code
 Shotcode (http://www.shotcode.com/)
 Motion Flow (2D Image Processing)
 GestureTek
- http://www.gesturetekmobile.com/
 TinyMotion
 3D Pose Calculation
 Augmented Reality

24. Handheld AR – Self Contained
2003 PDA-based AR
 ARToolKit port to PDA
 Studierstube ported to PDA
 AR Kanji Educational App.
 Mr Virtuoso AR character
 Wagner’s Invisible Train
- Collaborative AR

25. Mobile Phone AR – Self Contained
2004 Mobile Phone AR
 Moehring, Bimber
 Henrysson (ARToolKit)
 Camera, processor, display together

28. AR Advertising
 Txt message to download AR application (200K)
 See virtual content popping out of real paper advert
 Tested May 2007 by Saatchi and Saatchi

29. 2008 - Location Aware Phones
Motorola Droid Nokia Navigator

30. Real World Information Overlay
 Tag real world locations
 GPS + Compass input
 Overlay graphics data on live video
 Applications
 Travel guide, Advertising, etc
 Eg: Mobilizy Wikitude (www.mobilizy.com)
 Android based, Public API released
 Other companies
 Layar, AcrossAir, Tochnidot, RobotVision, etc

31. Layar – www.layar.com

32. HIT Lab NZ Android AR Platform
 Architectural Application
 Loads 3D models
 a OBJ/MTL format
 Positions content in space
 GPS, compass
 Intuitive user interface
 toolkit to modify the model
 Connects to back end model database

33. Architecture
Web Interface
Add models
Web application java
and php server
Android
application
Database server
Postgres

34. History of Handheld and Mobile AR
1995 Handheld Display: NaviCam, AR-PAD, Transvision
1997 Wearable AR: Touring Machine, AR Quake
2001 Handheld AR – Thin Client: AR-PDA, Bat Portal
2003 Handheld AR – Self contained: Invisible Train
2003 Mobile Phone – 2D Vision: Mozzies, Symball
2003 Mobile Phone – Thin Client: ARphone
2004 Mobile Phone – Self contained: Moehring, Symbian

35. 2012 State of the Art
Handheld Hardware available
PDA, mobile phones, external cameras
Sensors: GPS, accelerometer, compass
Software Tools are Available
Tracking: ARToolKitPlus, stbTracker, Vuforia
Graphics: OpenGL ES
Authoring: Layar, Wikitude, Metaio Creator
What is needed:
High level authoring tools
Content development tools
Novel interaction techniques
User evaluation and usability

36. Mobile AR Companies
 Mobile AR
 GPS + compass
 Many Companies
 Layar
 Wikitude
 Acrossair
 PressLite
 Yelp
 Robot vision
 Etc..

37. $2 million USD in 2010
$732 million USD in 2014

39. Handset Manufacturers
 Qualcomm
 $100 million USD investment
 Nokia
 25+ people in NRC
 Samsung
 Exploring the space
 Apple
 586 AR Applications on App Store
 Google
 Google goggles/Android AR Applications

40. Qualcomm
 Acquired Imagination
 October 2010 - Releases free Android AR SDK
 Computer vision tracking - marker, markerless
 Integrated with Unity 3D renderer
 http://developer.qualcomm.com/ar

41. Rock-em Sock-em
 Shared AR Demo
 Markerless tracking

42. Apple
 iPhone 4 SDK supports direct camera access
 Launches AR theme on App Store

43. Mobile AR Technology

44. Technology Components
 Software Platform
 Eg studierStube platform
 Developer Tools
 iOS, Android
 Tracking Technology
 Computer vision, sensor based
 Mobile Graphics
 OpenGL ES
 Interaction Methods
 Handheld Interaction

45. What are the challenges?
 Many platforms (operating systems)
 Slow CPUs (low clock rates, often no FPU)
 Difficulties with tracking
 Difficulties with visualizations that require a lot of data processing
 Small and slow memory access
 No or weak hardware 3D acceleration
 Difficulties with graphics
 Poor cameras
 Bad under low light, noise, etc

46. What makes a device interesting for AR?
 Open and easy to program
 Good camera
 Fast CPU, FPU is a plus
 Good H/W 3D support
 Large installed basis
 Easy access to devices
 GPS, accelerometer, compass
 Enough memory/storage

47. Typical Smart Phone Hardware
 CPU
 300-800+ Mhz
 GPU
 None, or Power VR Chip (OpenGL ES1.0/2.0)
 Input
 Touch screen, keyboard, keypad
 Sensors
 GPU, compass, accelerometer, camera (1.3-5mb+)
 Networking
 Bluetooth, Wifi, 3G
 Screen
 320x240 up to 800x480

48. HTC Desire
 Android
 Fast CPU (1GHz)
 Smaller screen
 3.7”, 800x480
 Multi-touch
 Hardware 3D
 GPS, compass and
accelerometer

49. iPhone 4
 Apple iOS 4.0
 Faster CPU (1.2GHz)
 High screen resolution
 3.5”, 960x640
 (Finally) camera API
 Multi-touch
 Hardware 3D
 GPS, compass, accelerometer
and gyroscope

50. Hardware Sensors
 Camera (resolution, fps)
 Maker based/markerless tracking
 Video overlap
 GPS (resolution, update rate)
 Outdoor location
 Compass
 Indoor/outdoor orientation
 Accelerometer
 Motion sensing, relative tilt

51. Studierstube ES Framework
 Typical AR application
framework
 Developed at TU Graz

52. End User
Application
Programming
Libraries
OS/Low Level API
Hardware

53. The Studierstube ES framework
User Interface - Application
Content
Graphics
Tracking
Platform

54. Mobile Development Environments
 Not like developing for desktops
 Wide range of different OS
 Limited CPU, low memory, poor graphics, no floating point
 Popular Mobile OS
 iPhone (Objective C)
 Android (Java, Native NDK wrapper)
 Windows Mobile (C#, C++, Visual Studio)
 Other
 Blackberry, Linux

55. Programming iPhone
 Harsh restrictions from Apple
 Apps have to go through the apps store
 Xcode IDE for development
 Nice development tools
 Objective-C
 Required for application development
 Can call into C/C++ code
 Limited camera API except in iOS 4
 Can overlay on live video
 No image processing

56. Android
 Hardware creators
 HTC, LG, Samsung, Motorola
 Widely available phone
 Different form factors – tablets, phones, PC, etc
 Multiple versions and fragmentation
 Android 1.0, 1.6
 Android 2.0, 2.1
 Free Tools
 Eclipse Development
 App Integrator

57. Mobile Graphics

58. Computer Graphics on Mobile Phones
 Small screen, limited input options
 Limited support for accelerated graphics
 Most phones have no GPU
 Mobile Graphics Libraries
 OpenGL ES (1.0, 2.0)
- Cross platform, subset of OpenGL
- C/C++ low level library
 Java M3G
- Mobile 3D graphics API for J2ME platform
- Object importer, scene graph library
- Support from all major phone manufacturers

60. OpenGL ES
 Small-footprint subset of OpenGL
 OpenGL is too large for embedded devices!
 Powerful, low-level API, full functionality for 3D games
 Can do almost everything OpenGL can
 Available on all key platforms
 Software and hardware implementations available
 Fully extensible
 Extensions like in OpenGL

61. OpenGL ES
SLIDE 61
OpenGL ES on mobile devices

62. OpenGL ES History
 Khronos Group – founded 2000
 Goal: create royalty free open multimedia standards
 Create OpenML specification
 Consortium of > 120 companies
 OpenGL ES working group – formed 2002
 20-30 companies
 Create API for 3D graphics on embedded devices
 Cleaned up and trimmed down version of OpenGL
 OpenGL ES 1.0 released 2004, 1.1 in 2005, 2.0 spec in 2007

65. Versions
 Two major tracks
 Not compatible, parallel rather than competitive
 OpenGL ES 1.x
 Fixed function pipeline
 Suitable for software implementations
 All 1.x are backwards compatible
 OpenGL ES 2.x
 Vertex and pixel shaders using GLSL ES
 All 2.x will be backwards compatible

66. Fixed Function (1.x)
h"p://www.khronos.org/opengles/2_X/

67. Programmable (2.x)
h"p://www.khronos.org/opengles/2_X/

68. OpenGL ES 1.x vs 2.0

69. Hardware supporting OpenGL ES 2.x
 Qualcomm SnapDragon
 Texas Instruments OMAP3
 NVIDIA Tegra (APX2500)
 Samsung S3C6410
 POWERVR SGX
 Now becoming available in phones!
 Typically come with OpenGL ES 1.x drivers too
 The emulation on GLES 2.0 hardware can be slow!

70. Tracking

71. Mobile Augmented Reality’s goal
Create an affordable, massively multi-user, widespread platform
© Tinmith, U. of South Australia

72. Tracking Requirements for AR on Phones
 Fast and efficient
 Form factor: light and robust
 Track simultaneously
 a large number of objects
 by a large number of users
 Requires little or no …
 Device modification
 Manual calibration (targeting non-technical users)
 Instrumentation of the physical environment
 Low costs

73. How to not do it…
Ka-Ping Yee: Peephole Displays, CHI’03

74. Tracking on mobile phones
 Vision-based tracking
 Marker-based tracking
 Model-based natural feature tracking
 Natural feature tracking in unknown environments
 Sensor tracking
 GPS, inertial compass, gyroscope

75. Backpack-based
Höllerer et al. (1999), Piekarski & Thomas al. (2001), Reitmayr & Schmalstieg (2003)
 Laptop, HMD
 Enhanced GPS (DGPS / RTK) + inertial sensor for viewpoint tracking
 Hand tracking w/ fiducial markers

76. Tablet PC / UMPC-based
Schall et al., 2006
 Hybrid tracking on UMPC
 Camera  fiducial marker tracking
 When no marker in view  inertial sensor + UWB tracking

77. PDA-based 1.
BatPortal (Newman et al., 2001) SHEEP (MacWilliams et al., 2003)
 PDA as thin client (rendering &
 Tracking by ART (external IR
tracking on server + VNC) cameras + retroreflective target)
 Ultrasonic tracking  Projection-based AR environ.

78. PDA-based 2.
Signpost on PDA (Wagner & Schmalstieg, 2003)
 First “truly” handheld AR platform: PDA + camera
 Standalone, self-contained AR system
 Optimized fiducial marker tracking library

79. History of non-AR Tracking on Phones (1)
AR-PDA (Gausemeier et al., 2003) Kick Real (Paelke et al., 2004)
 Model-based tracking  Edge detection of real foot + collision
 PDA = thin client detection w/ virtual ball
 tracking off-loaded to server  2D tracking and limited interaction
 Not real-time (tailored to game)

80. History of non-AR Tracking on Phones (2)
Mosquito Hunt (Siemens, 2003)
Marble Revolution (BitSide, 2004)
Pingis (VTT, 2006)
TinyMotion (Wang et al., 2006)
Game control w/ optical GUI control & input on cell phones
flow techniques w/ image differencing & block correlation

81. History of AR Tracking on Phones (1)
 2003
 ARToolKit on PDA
 Wagner et at.
 2004
 3D Marker on Phone
 Möhring et al.
 2005
 ARToolKit on Symbian
 Henrysson et al.

82. History of AR Tracking on Phones (2)
 2005
 Visual Codes
 Rohs et at.
 2008
 Advanced Marker Tracking
 Wagner et al.
 2008
 Natural Feature Tracking
 Wagner et al.

83. What can we do on today‘s mobile phones?
 Typical specs
 600+ MHz
 ~5MB of available RAM
 160x120 - 320x240 at 15-30 Hz camera
 Possible to do
 Marker tracking in 5-15ms
 Natural feature tracking in 20-50ms

84. Other Mobile Sensors
 Orientation
 Compass
 Relative movement/rotation
 Accelerometer, gyroscope
 Context
 Audio, light sensor, proximity
 Location
 GPS, A-GPS, Wifi positioning, Cell tower triangulation

85. Handheld AR Interfaces

86. Handheld HCI
 Consider your user
 Follow good HCI principles
 Adapt HCI guidelines for handhelds
 Design to device constraints
 Rapid prototyping
 User evaluation

87. Sample Handheld AR Interfaces
 Clean
 Large Video View
 Large Icons
 Text Overlay

88. Handheld Display vs Fixed Display
 Experiment comparing handheld moving, to handheld button input, small
fixed display, desktop display, large plasma
 Users performed (1) navigation task, (2) selection task
 Moving handheld display provided greater perceived FOV, higher degree of
presence, faster completion time
J. Hwang, J. Jung, G. Kim. Hand-held Virtual Reality: A Feasibility Study. In proceedings of VRST 2006

89. Search Task Completion Time

90. FOV, Presence and Immersion

91. Outdoor AR: Limited Field of View

92. Possible solutions
 Overview + Detail
 spatial separation; two views
 Focus + Context
 merges both views into one view
 Zooming
 temporal separation

93. Zooming Views
 TU Graz – HIT Lab NZ - collaboration
 Zooming panorama
 Zooming Map

94. Zooming AR interfaces
Context
Compass
Zooming
Panorama
Zooming
Map
 Interface Types
 Compass (C)
 Compass + Zooming Panorama (CP)
 Compass + Zooming Map (CM)
 Compass, Zooming Panorama, Zooming Map (CPM)

95. Twitter 360
 www.twitter-360.com
 iPhone application
 See geo-located tweets in real world
 Twitter.com supports geo tagging

96. Wikitude – www.mobilizy.com
Blah
Blah Blah Blah
Blah Blah Blah
Blah
Blah
Blah
Blah Blah
Blah Blah Blah Blah
Blah Blah
Blah
Blah Blah Blah

97. Information Filtering

98. Information Filtering
 Information Filtering (Julier et al. ’00)
• Remove clutter by goal- and distance based filtering
• User’s task is route finding: Sniper and relevant buildings are displayed;
objects, which are determined to be unnecessary, removed

99. HMD vs Handheld AR Interface
Wearable AR HandHeld AR
Output:
Display Input &
Output
Input

100. Handheld Interface Metaphors
 Tangible AR Lens Viewing
 Look through screen into AR scene
 Interact with screen to interact with AR
content
- Eg Invisible Train
 Tangible AR Lens Manipulation
 Select AR object and attach to device
 Use the motion of the device as input
- Eg AR Lego

101. Translation Study
Conditions
A: Object fixed to the phone (one handed)
B: Button and keypad input
C: Object fixed to the phone (bimanual)
- one hand for rotating tracking pattern

102. Results – Translation
• 9 subjects – within subject design
• Timing
• Tangible fastest
• twice as fast as keypad
• Survey
• Tangible easiest (Q1)
• Keypad most accurate (Q2)
• Tangible quickest (Q3)
• Tangible most enjoyable (Q4)
• Ranking B
A
C
• Tangible favored
Rank
1.44 2.56 2.0

103. Rotation Study
Conditions
A: Arcball
B: Keypad input for rotation about
the object axis
C: Object fixed to the phone (one handed)
D: Object fixed to the phone (bimanual)

104. Results – Rotation
• Timing
• Keypad(B) and Arcball(A) fastest
• No significant survey
differences
A B C D
Rank 3.0 2.3 2.4 2.2

105. TAT Augmented ID

108. Design Guidelines
Apply handheld HCI guidelines for on-screen content
- large buttons, little text input, etc
Design physical + virtual interface elements
Pick appropriate interface metaphor
- “handheld lens” approach using handheld motion
- Tangible AR for AR overlay
Build prototypes
Continuously evaluate application

109. Mobile AR Browsing

112. Junaio - www.junaio.com

114. QR Code Launch

115. Glue Tracking - Markerless
 Search for “instant tracker”

116. Junaio Interface

117. Interface
 List View, Map View, AR View

118. Architecture

120. Back-end Servers

121. Data Flow

122. Search.php
<?xml version="1.0" encoding="UTF-8"?>
<results>
<poi id="1" interactionfeedback="none">
<name><![CDATA[[Hotel Hello World]]]></name>
<description><![CDATA[[This is a beautiful, family hotel and restaurant, just around the
corner. Special Dinner and Rooms available.]]]></description>
<l>37.776685,-122.422771,0</l>
<mime-type>text/plain</mime-type>
<icon>http://dev.junaio.com/publisherDownload/tutorial/icon_map.png</icon>
<thumbnail>http://dev.junaio.com/publisherDownload/tutorial/thumb.jpg</thumbnail>
<phone>555/1234567</phone>
<homepage> http://www.hotelaroundthecorner.com </homepage>
</poi>
</results>

123. Creating a New Channel

124. AR Outcome

125. Mime Tag Types
 <mime-type>text/plain</mime-type>
 Changing mime tag changes media loaded

131. AR Browsers
 Commercial outdoor AR applications
 Junaio, Layar, Wikitude, etc
 All have their own language specifications
 Wikitude – ARML
 Junaio - XML
 Need for common standard
 Based on existing standards for geo-located content etc
 Support for dynamic/interactive content
 Easier to author mobile AR applications
 Easy to render on AR browsers

132. BirdsView - http://www.birdsview.de/
 Location Based CMS
 Add content, publish to Layar or Junaio

133. BirdsView on Junaio

134. BirdsView on Junaio

135. Limitations
 BirdsView Branding
 Their logo, not yours
 Limited POI Content
 Images, Text, URL
 Public Channel
 Your content + everyone else's

136. Hoppala Augmentation
 http://www.hoppala-agency.com/
 Rich media overlay

137. Hoppala in Junaio

138. Metaio Creator
 Drag and drop Junaio authoring

139. BuildAR – buildar.com

140. 3DOn
 Onsite Visualization Single Building
 GPS + Compass input
 Overlay graphics data on live video/Photos

141. 3D On Interface