The poster was presented during the IEEE Virtual Reality conference (IEEE VR 2012) in Orange County, California, USA on March 4-8, 2012. IEEE VR is the premier conference for virtual reality, including work on augmented reality, mixed reality, and 3D user interfaces. The poster presenting an efficient natural feature tracking pipeline solely implemented in JavaScript. It is embedded in a web technologybased Augmented Reality system running plugin-free in web browsers. The evaluation shows that real-time framerates on desktop computers and while on smartphones interactive framerates are achieved. For more info about the poster, please follow this link http://icg.tugraz.at/publications/natural-feature-tracking-in-javascript/

1. Natural Feature Tracking in JavaScript
Christoph Oberhofer+, Jens Grubert+, Gerhard Reitmayr +
INTRODUCTION
We present an efficient natural feature tracking pipeline solely implemented in JavaScript. It is embedded in a web technology-based Augmented Reality system running
plugin-free in web browsers. An evaluation shows that real-time frame rates on a desktop computer and interactive frame rates on a smartphones are achieved.
MOTIVATION
Portability HTML5
Reliance on web technologies enables HTML5 provides the building blocks for realizing
platform independence. plugin-free Augmented Reality on the web.
As we employ standardized web technologies our system can
HTML5 elements and related frameworks (WebGL, WebRTC)
run on most platforms that provide an HTML5 enabled browser.
enable the creation of a complete Augmented Reality pipeline
including natural feature tracking.
IMPLEMENTATION Augmented Reality Pipeline in HTML5 Two-Stage Tracking Approach
Camera Access Detection
We employ the MediaStram API along
with the <video> and <canvas > elements <video> <canvas> pose
getUserMedia
to access the video stream of a camera. FAST [2] BRIEF [1] RANSAC
multi-level
keypoint detection
keypoint description +
matching
outlier removal estimation
A two-stage pipeline written entirely in
JavaScript efficiently computes the pose
of the camera (similar to [3]). When the pose
initial pose is detected it is incrementally
3D Rendering Computer Vision Tracking
updated during the tracking stage.
Image Level n Image Level 0
…
function initializationPhase(){
grabber.grab();
NCC-based NCC-based
imageDataContainer.updateData();
Finally, the pose is used to overlay 3D
fastDetector.detect(imageDataContainer, fastThreshold, fastPoints);
pose pose
fastDetector.nonmax_suppression(fastPoints, keyPoints);
brief.getDescriptorsForLevel(imageDataContainer, keyPoints);
keypoint keypoint
var correspondences = brief.match(template_keypointSet, keyPoints);
CVUtils.undistortPoints(correspondences.right, K, distCoeffs);
graphics with WebGL over the video. refinement refinement
var ransac = new RANSAC(correspondences, config.detection.ransac);
inlierSet = ransac.execute();
tracking tracking
if(inlierSet == null || inlierSet.length < config.detection.ransac.minNumOfInlier){
poseUpdated(TrackResult.FAILURE);
return;
}
var H = CHOLESKY.computeHomography(inlierSet);
inlierSet = new Array();
ransac.computeInlierSet(inlierSet, H.toMat());
H = CHOLESKY.computeHomography(inlierSet);
computePose(H, inlierSet);
// Switch to tracking phase
poseUpdated(TrackResult.GOOD);
currentPhase = Phase.TRACKING;
document.getElementById("numFeatures").innerHTML = inlierSet.length;
}
EVALUATION Detection Phase PC Detection Phase PC vs. Mobile
Chrome Canary 17
Tracking Phase PC Tracking Phase PC vs. Mobile
Chrome Canary 17
Firefox 8 (PC) Firefox 8 (PC)
Opera Next 12 Opera Next 12
Firefox Nightly 11.0a1
System Firefox 8
Firefox 8 (Android)
Firefox Nightly 11.0a1
Firefox 8 (Android)
PC: Core2Duo 3GHZ, 4GB RAM Google Native Client Firefox 8
Mobile: Samsung Galaxy SII Adobe Alchemy
Time [ms] 0 100 200 300 400 500 600 Time [ms] 0 5 10 15 20 25 30 35 40 45 50 Time [ms] 0 10 20 30 40 50 60 70 80 90 100
Time [ms] 0 20 40 60 80 100 120 140 160 keyframe track 2 pose 2 track 1 pose 1 track 0 pose 0 keyframe tracking 1 pose 1 tracking 0 pose 0
integral fast descriptor match ransac pose
integral fast descriptor match ransac pose
320x240px | FAST: 250 KP | BRIEF: 128 bit | RANSAC: min 30 inlier | max 300 iterations 320x240px | levels: 3 (120, 100, 70 KPs) | search radius: 2, 2, 4 px | patch size: 8x8 px 320x240px | levels: 2 (60, 40 KPs) | search radius: 2, 3 px | patch size: 8x8 px
In the detection phase the fastest web browser (Firefox 8) is 1.6 times faster than the slowest one (Opera 12). In the tracking phase the fastest web browser (Chrome Canary) is 2.5 times faster than the slowest one (Firefox 8).
An implementation in Google Native Client is 3.7 times faster and in Adobe Alchemy still 2 times faster. Compared to the PC the system runs 5.5 times slower in a web browser on a smartphone.
Compared to the PC the system runs 6 times slower in a web browser on a smartphone.
Overall the detection can be carried out with 11 frames per second (fps) on a PC but only with 2 fps on a smartphone. Once initialized the system can track with up to 50 fps on a PC and 9 fps on a smartphone.
REFERENCES [1] M. Calonder, V. Lepetit, C. Strecha, and P. Fua. BRIEF: Binary Robust Independent Elementary Features. In ECCV 2010, pages 778-792. +Graz
[2] E. Rosten and T. Drummond. Machine learning for high-speed corner detection. In ECCV 2006, pages 430-443. University of Technology, Austria |[ grubert| reitmayr]@icg.tugraz.at |ch.oberhofer@gmail.com
[3] D. Wagner, G. Reitmayr, A. Mulloni, T. Drummond, and D. Schmalstieg. Real-Time Detection and Tracking for
Augmented Reality on Mobile Phones. In IEEE TVCG, 99(1), pages 355-368, November 2010.
This work is made possible by the Austrian National Research Funding Agency FFG in the SmartReality project.