3. Augmented Reality Definition
Defining Characteristics [Azuma 97]
Combines Real and Virtual Images
g
- Both can be seen at the same time
Interactive in real-time
- Virtual content can be interacted with
Registered in 3D
g
- Virtual objects appear fixed in space
4. What is not Augmented Reality?
Location-based services
Barcode detection (QR-codes)
B d d i (QR d )
Augmenting still images
g g g
Special effects in movies
…
… but they can be combined with AR!
7. AR History Summary
1960’s – 80’s: Early Experimentation
1980 s 90 s:
1980’s – 90’s: Basic Research
Tracking, displays
1995 – 2005: Tools/Applications
Interaction, usability, theory
y y
2005 - : Commercial Applications
Games, M di l Industry
G Medical, I d
16. AR Displays
AR
Visual Displays
Primarily Indoor Primarily Outdoor
Environments (Daylight) Environments
Not Head-Mounted Head-Mounted Head-Mounted Not Head Mounted
Display (HMD) Display (HMD) (e.g. vehicle mounted)
Virtual Images Projection CRT Display Liquid Crystal Cathode Ray Tube (CRT) Projection Display
or Virtual Retinal Display (VRD)
p y( ) Navigational Aids in Cars
g
seen off windows using beamsplitter Displays LCDs Many Military Applications Military Airborne Applications
& Assistive Technologies
e.g. window e.g. Reach-In e.g. Shared Space e.g. WLVA e.g. Head-Up
reflections Magic Book and IVRD Display (HUD)
18. Head Mounted Displays (HMD)
- Display and Optics mounted on Head
- May or may not fully occlude real world
- Provide full-color images
- Considerations
• Cumbersome to wear
• Brightness
• Low power consumption
• Resolution limited
• Cost is high?
g
19. Types of Head Mounted Displays
Occluded
See-thru
Multiplexed
20. Immersive VR Architecture
Virtual
World
head position/orientation
Head Non see-thru
Tracker Image source
& optics
Host Data Base Rendering
Frame
Processor
P Model
M d l Engine
E i
Buffer
virtual
to network object
Display
Driver
21. See-thru AR Architecture
head position/orientation
Head see-thru
Tracker combiner
real world
Host Data Base Rendering
Frame
Processor
P Model
M d l Engine
E i
Buffer
to network Virtual Image
Display superimposed
Driver over real world
object
Image source
g
25. DigiLens
Compact HOE
C
Solid state optics
Switchable Bragg Grating
Stacked SBG
Fast switching
Ultra compact
www.digilens.com
26. The Virtual Retinal Display
Image scanned onto retina
age sca e o to et a
Commercialized through Microvision
Nomad System - www.mvis.com
27. Strengths of optical AR
Simpler (cheaper)
Direct view of real world
Di i f l ld
Full resolution, no time delay (for real world)
Safety
Lower distortion
No eye displacement
but COASTAR video see-through avoids this
28. Video AR Architecture
Head-mounted
camera aligned to
head position/orientation display optics
Video image
Head
Tracker of real world
Video
Processor
Host Graphics Digital
Frame
Processor
P renderer
d Mixer
Mi
Buffer
to network
Display
Driver
Virtual image
inset into
Non see-thru
video of real
Image source
world
& optics
31. Video see-through HMD
MR Laboratory’s COAS A HMD
’ COASTAR
(Co-Optical Axis See-Through Augmented Reality)
Parallax-free
Parallax free video see through HMD
see-through
32. TriVisio
www.trivisio.com
p
Stereo video input
PAL resolution cameras
2 x SVGA displays
30 degree FOV
User adjustable convergence
$6,000 USD
34. Strengths of Video AR
True occlusion
Kiyokawa optical display that supports occlusion
y p p y pp
Digitized image of real world
Flexibility
Fl b l in composition
Matchable time delays
More registration, calibration strategies
Wide FOV is easier to support
35. Optical vs. Video AR Summary
Both have proponents
Video is more popular today?
Likely because lack of available optical products
y p p
Depends on application?
Manufacturing: optical i cheaper
M f i i l is h
Medical: video for calibration strategies
36. Eye multiplexed AR Architecture
head position/orientation
Head
Tracker real world
Host Data Base Rendering
Frame
Processor
P Model
M d l Engine
E i
Buffer
to network
Display Virtual Image
Driver inset into
real world scene
Opaque
Image source
41. Spatial Augmented Reality
Project onto irregular surfaces
Geometric Registration
Projector blending, High dynamic range
Book: Bimber, Rasker “Spatial Augmented Reality”
p g y
42. Projector-based AR
User (possibly
head-tracked)
Projector
Examples:
Real objects
Raskar,
Raskar MIT Media Lab
with retroreflective
Inami, Tachi Lab, U. Tokyo
covering
52. Video Monitor AR
Video Stereo
cameras Monitor g
glasses
Video
Graphics Combiner
53. Virtual Showcase
Mirrors on a projection table
Head
H d tracked stereo
k d
Up to 4 users
Merges graphic and real objects
M hi d l bj
Exhibit/museum applications
Fraunhofer Institute (2001)
Bimber, Frohlich
62. Tracking
Locating the users viewpoint
g p
Position (x,y,z)
Orientation (r p y)
(r,p,y)
Registration
Positioning virtual object wrt real world
63. Tracking Requirements
Head Stabilized Body Stabilized World Stabilized
Augmented Reality Information Display
World Stabilized
Body Stabilized Increasing Tracking
Requirements
Head Stabilized
64. Tracking Technologies
• Mechanical
• g
Electromagnetic
• Optical
• Acoustic
• Inertial d dead
I ti l and d d reckoning
k i
• GPS
• Hybrid
65. AR Tracking Taxonomy
AR
TRACKING
Indoor Outdoor
Environment Environment
Limited Range Extended Range Low Accuracy & High Accuracy
Not Robust & Robust
Low Accuracy High Accuracy Many Fiducials Not Hybridized Hybrid Tracking
at 15-60 Hz & High Speed in space/time GPS or GPS and
Hybrid
H b id but
b t Camera or
C Camera and
C d
Tracking no GPS Compass Compass
e.g.
e g AR Toolkit e.g.
e g IVRD e.g.
e g HiBall e.g.
e g WLVA e.g.
e g BARS
66. Tracking Types
Magnetic Inertial Ultrasonic Optical Mechanical
Tracker
T k Tracker
T k Tracker
T k Tracker
T k Tracker
Specialized Marker-Based
Marker Based Markerless
Tracking Tracking Tracking
Edge-Based Template-Based Interest Point
g
Tracking g
Tracking g
Tracking
67. Tracking Systems
Mechanical Tracker
Magnetic Tracker
Ultrasonic Tracker
Inertial Tracker
Vision (Optical Tracking)
Specialized (Infrared, Retro-Reflective)
Monocular (DVCam, Webcam)
M l (DVC W b )
69. Magnetic Tracker
Idea: difference between a magnetic transmitter
and a receiver
Flock of Birds (Ascension)
++: 6DOF robust
6DOF, b
-- : wired, sensible to metal, noisy, expensive
y p
70. Inertial Tracker
I lT k
Idea: measuring linear and angular orientation rates
(accelerometer/gyroscope)
IS300 (Intersense)
Wii Remote
++: no transmitter, cheap small high frequency wireless
transmitter cheap, small, frequency,
-- : drift, hysteris only 3DOF
71. Ultrasonics Tracker
Idea: Time of Flight or Phase Coherence Sound Waves
Phase-Coherence
Ultrasonic
Logitech IS600
++: Small, Cheap
-- : 3DOF, Line of Sight, Low resolution, Affected
Environment Conditon (pressure, temperature)
72. Global Positioning System (GPS)
Created by US in 1978
Currently 29 satellites
Satellites send position + time
GPS Receiver positioning
p g
4 satellites need to be visible
Differential time of arrival
Triangulation
Accuracy
y
5-30m+, blocked by weather, buildings etc
73. Problems with GPS
Takes time to get satellite fix
Satellites moving around
Earths atmosphere affects signal
p g
Assumes consistent speed (the speed of light).
Delay depends where you are on Earth
Weather effects
Signal reflection
Multi-path reflection off buildings
Signal blocking
Trees, buildings, mountains
T b ildi t i
Satellites send out bad data
Misreport their own position
74. Accurate to < 5cm close to base station (22m/100 km)
Expensive - $20-40,000 USD
76. Optical Tracker
Idea: Image Processing and Computer Vision
g g p
Specialized
Infrared, Retro-Reflective, Stereoscopic
f f S
ART Hi-Ball
Monocular Based Vision Tracking
g
78. Optical Tracking Technologies
Scalable active trackers
InterSense IS-900, 3rd Tech HiBall
3rd Tech, I
3 d T h Inc.
Passive optical computer vision
Line of sight, may require landmarks
sight
Can be brittle.
Computer vision i computationally-intensive
C i i is i ll i i
79. HiBall Tracking System (3rd Tech)
Inside-Out Tracker
O
$50K USD
Scalable over large area
Fast d t (2000H )
F t update (2000Hz)
Latency Less than 1 ms.
Accurate
Position 0.4mm RMS
0 4mm
Orientation 0.02° RMS
80.
81. Starting simple: Marker tracking
Has been done for more than 10 years
Several open source solutions exist
S l l i i
Fairly simple to implement
y p p
Standard computer vision methods
A rectangular marker provides 4 corner points
l k id i
Enough for pose estimation!
85. Marker Tracking – Fiducial Detection
Threshold the whole image to black and white
Search scanline by scanline for edges (white to black)
Follow edge until either
Back to starting pixel
Image border
Check for size
Reject fiducials early that are too small (or too large)
86. Marker Tracking – Rectangle Fitting
Start with an arbitrary point “x” on the contour
S ih bi i “ ” h
The point with maximum distance must be a corner c0
Create a diagonal through the center
C d l h h h
Find points c1 & c2 with maximum distance left and right of diag.
New diagonal from c1 to c2
Find point c3 right of diagonal with maximum distance
87. Marker Tracking – Pattern checking
Calculate homography using the 4 corner points
“Direct Linear Transform” algorithm
Maps normalized coordinates to marker coordinates
p
(simple perspective projection, no camera model)
Extract pattern by sampling
Check pattern
Id (implicit encoding)
Template (
T l (normalized cross correlation)
li d l i )
88. Marker Tracking – Corner refinement
Refine corner coordinates
Critical for high quality tracking
Remember: 4 points is the bare minimum!
So these 4 points should better be accurate…
Detect sub-pixel coordinates
E.g. Harris corner detector
g
- Specialized methods can be faster and more accurate
Strongly reduces jitter!
gy j
Undistort corner coordinates
Remove radial distortion from lens
R di l di t ti f l
89. Marker
M k tracking – P
k Pose estimation
Calculates marker position and rotation
C
relative to the camera
Initial estimation directly from homography
Very fast, but coarse
fast
Jitters a lot…
Refinement via G
R fi i Gauss-Newton iteration
N i i
6 parameters (3 for position, 3 for rotation) to refine
At each iteration we optimize on the reprojection error
91. Coordinates for Marker Tracking
•Camera Observed Screen
Marker Ideal Screen Screen
•Ideal ScreenCamera
•Marker Observed
•Perspective model (barrel shape)
•Goal
•Correspondence of
•Nonlinear function 4 vertices
•Obtained fromTranslation
•Rotation & Camera Calibration
•Obtained fromT processing
R t ti
•Real time image l ti
Camera Calibration
92. From Marker To Camera
F M k T C
Rotation & Translation
TCM : 4x4 transformation matrix
from marker coord. to camera coord.
93. Tracking challenges in ARToolKit
Occlusion Unfocused camera, Dark/unevenly lit Jittering
(image by M. Fiala) motion blur scene, vignetting (Photoshop illustration)
Image noise
False positives and inter-marker confusion (e.g. poor lens, block coding /
compression, neon tube)
(image by M. Fiala)
95. Other Marker Tracking Libraries
arTag
T
http://www.artag.net/
ARToolKitPlus [Discontinued]
http://studierstube.icg.tu-
graz.ac.at/handheld_ar/artoolkitplus.php
graz ac at/handheld ar/artoolkitplus php
stbTracker
http://studierstube.icg.tu-
htt // t di t b i t
graz.ac.at/handheld_ar/stbtracker.php
MXRToolKit
http://sourceforge.net/projects/mxrtoolkit/
98. Markerless Tracking
No more Markers! Markerless Tracking
Magnetic T k
M i Tracker Inertial
I i l Ultrasonic
Ul i Optical
O i l
Tracker Tracker Tracker
Specialized Marker-Based Markerless
Tracking Tracking Tracking
Edge-Based Template-Based Interest Point
Tracking Tracking Tracking
99. Natural feature tracking
Tracking from features of the surrounding
environment
Corners, edges, blobs, ...
Generally more diffi l than marker tracking
G ll difficult h k ki
Markers are designed for their purpose
g p p
The natural environment is not…
Less well established methods
well-established
Usually much slower than marker tracking
100. Natural Feature Tracking
Features Points
Use Natural Cues of Real Elements
Contours
Edges
Surface Texture
Interest Points
Model or Model-Free
++: no visual pollution
Surfaces
102. Tracking by d
T k b detection
This is what most trackers do
do… Camera Image
g
Targets are detected every frame
Keypoint detection
yp
Popular because
tracking and detection Descriptor creation
are solved simultaneously
and matching
d t hi
Outlier Removal
Pose estimation
and refinement
Pose
103. Natural feature tracking – What is a keypoint?
It depends on the detector you use!
For high performance use the FAST corner
detector
Apply
A l FAST t all pixels of your i
to ll i l f image
Obtain a set of keypoints for your image
- R d
Reduce the amount of corners using non-maximum suppression
h f
Describe the keypoints
E. Rosten and T. Drummond (May 2006). "Machine learning for high‐speed corner detection".
105. Natural feature tracking – Descriptors
Again depends on your choice of a descriptor!
Can use SIFT
Estimate the d i
E i h dominant keypoint
k i
orientation using gradients
Compensate for
C m ensate f r detected
orientation
Describe the keypoints in terms
of the gradients surrounding it
Wagner D., Reitmayr G., Mulloni A., Drummond T., Schmalstieg D.,
Real‐Time Detection and Tracking for Augmented Reality on Mobile Phones.
IEEE Transactions on Visualization and Computer Graphics, May/June, 2010
106. NFT – D b
Database creation
Offline step
p
Searching for corners in a static image
For robustness look at corners on multiple scales
Some corners are more descriptive at larger or smaller scales
We d ’t k
W don’t know how far users will be from our image
h f ill b f i
Build a database file with all descriptors and their
position on the original i
ii h i i l image
107. NFT – R l
Real-time tracking
k
Search for keypoints Camera Image
in the video image
Create the d
C t th descriptorsi t Keypoint detection
Match the descriptors from the Descriptor creation
p
live video against those and matching
in the database
Outlier Removal
O tli R l
Brute force is not an option
Need the speed-up of special Pose estimation
data structures and refinement
- E.g., we use multiple spill trees
Pose
108. NFT – O l removal
Outlier l
Cascade of removal techniques
Start with cheapest, finish with most
expensive…
First simple geometric tests
- E.g., line tests
• Select 2 points to form a line
• Check all other points being on correct side of line
Then, homography-based tests
109. NFT – P
Pose refinement
f
Pose from homography makes good starting point
Based on Gauss-Newton iteration
Try to minimize the re-projection error of the keypoints
Part of tracking pipeline that mostly benefits
from floating point usage
Can still be implemented effectively in fixed point
Typically 2-4 iterations are enough…
110. NFT – R l
Real-time tracking
k
Search for keypoints Camera Image
in the video image
p
Create the descriptors Keypoint detection
Match the descriptors from the
live video against those Descriptor creation
and matching
in the database
Remove the keypoints that Outlier Removal
are outliers Pose estimation
Use h
U the remaining k
i i keypoints
i
and refinement
to calculate the pose
Pose
of the camera
f h
111. NFT – R l
Results
Wagner D., Reitmayr G., Mulloni A., Drummond T., Schmalstieg D.,
Real‐Time Detection and Tracking for Augmented Reality on Mobile Phones.
IEEE Transactions on Visualization and Computer Graphics, May/June, 2010
112. Edge Based Tracking
RAPiD [Drummond et al. 02]
Initialization, Control Points, Pose Prediction (Global Method)
119. Marker vs. natural feature tracking
Marker tracking
Usually requires no database to be stored
Markers can be an eye-catcher
Tracking is less demanding
g g
The environment must be instrumented with markers
Markers usually work only when fully in view
y y y
Natural feature tracking
A database of keypoints must be stored/downloaded
Natural feature targets might catch the attention less
Natural f t
N t l feature targets are potentially everywhere
t t t ti ll h
Natural feature targets work also if partially in view
121. Example: Outdoor Hybrid Tracking
Combines
computer vision
- natural feature tracking
inertial gyroscope sensors
Both correct for each other
Inertial gyro - provides frame to frame
prediction of camera orientation
Computer vision - correct for gyro drift
126. The Registration Problem
Virtual and Real must stay properly aligned
If not:
t
Breaks the illusion that the two coexist
Prevents acceptance of many serious applications
127. Sources of registration errors
Static errors
S
Optical distortions
Mechanical misalignments
Tracker errors
Incorrect viewing parameters
Dynamic errors
System delays (largest source of error)
- 1 ms d l = 1/3 mm registration error
delay i t ti
128. Reducing static errors
Distortion compensation
Manual adjustments
View-based or direct measurements
[Azuma94] [Caudell92] [Janin93] etc.
Camera calibration (video)
[ARGOS94] [Bajura93] [Tuceryan95] etc.
130. Dynamic errors
Application Loop
x,y,z
Tracking r,p,y
Calculate Render Draw to
Viewpoint Scene Display
Simulation
20 Hz = 50ms 500 Hz = 2ms 30 Hz = 33ms 60 Hz = 17ms
Total Delay = 50 + 2 + 33 + 17 = 102 ms
1 ms delay = 1/3 mm = 33mm error
131. Reducing dynamic errors (1)
Reduce system lag
[Olano95] [Wloka95a] [Regan SIGGRAPH99]
Reduce apparent lag
Image deflection [Burbidge89] [Regan94] [So92]
[Kijima
[Kiji ISMR 2001]
Image warping [Mark 3DI 97]
132. Reducing System Lag
Application Loop
x,y,z
Tracking r,p,y
Calculate Render Draw to
Viewpoint Scene Display
Simulation
Faster Tracker Faster CPU Faster GPU Faster Display
133. Reducing Apparent Lag
Virtual Display Virtual Display
x,y,z
y
Physical
Ph i l r,p,y
Physical
Display Display
(640x480) (640x480)
Tracking
1280 x 960 Update 1280 x 960
Last known position Latest position
Application Loop
x,y,z
Tracking r,p,y
Calculate Render Draw to
Viewpoint
p Scene Display
p y
Simulation
134. Reducing dynamic errors (2)
Match input streams (video)
Delay video of real world to match system lag
Predictive Tracking
[Azuma94] [Emura94]
Inertial sensors helpful
Azuma / Bishop 1994
u a s op 99
137. Wrap-up
Tracking and Registration are key problems
Registration error
Measures against static error
Measures against dynamic error
M i d i
AR typically requires multiple tracking technologies
yp y q p g g
Research Areas: Hybrid Markerless Techniques,
Deformable Surface, Mobile, Outdoors
Surface Mobile
138. More Information
• M k Billi h t
Mark Billinghurst
– mark.billinghurst@hitlabnz.org
• Websites
– www.hitlabnz.org
hi l b