1. Creating Novel Augmented Reality
Applications for Mobile Users
www.diotasoft.com
Sebastian Knödel Ludovic Perrier
skn@diotasoft.com
lpe@diotasoft.com
sekont.tumblr.com
ludovic_perrier
sekond
Breizhcamp 1

2. Content
Brief History of Mobile AR
Mobile AR SDKs & APIs
iCyclope for IPhone
Qualcomm AR SDK (QCAR)
QCAR & Unity3D
Mobile AR Games
Future of mobile AR
Breizhcamp 2

3. What is mobile AR good for?
Services
- Information visualization
- Navigation
Games
Promotion, Advertising
Industrial Applications Courtesy Qualcomm
Why now?
Because todays Smart Phones are able to!
- Hardware (computing capacity, sensors,
cameras, screensize, …)
- Software, powerful SDKs (Android, Iphone,
QCAR, …)
Courtesy Layar
Breizhcamp 3

4. Mobile Augmented Reality
Milgram's Continuum
Mixed Reality
Real Augmented Augmented Virtual
Environment Environment Virtuality Environment
Breizhcamp 4

5. Whats the problem?
Where am I ?
Breizhcamp 5

6. Brief History of Mobile AR
Sensor based tracking
GPS position
Compass orientation
Courtesy Wikitude Wikitude 2009
Vision based tracking
Courtesy TU Graz
Special Marker 1999 ARToolkitPlus 2007 NyARToolkit 2009
Breizhcamp 6

7. Vision-based Tracking
Calculate camera position and orientation relative to
markers – image registration
1. Stage 2. Stage
Feature Detection Coordinate system
Breizhcamp 7

8. ARToolkit
Breizhcamp 8

9. Natural feature-based tracking
Breizhcamp 9

10. Brief History of Mobile AR
Vision based tracking
Courtesy Qualcomm
Qualcomm AR
Courtesy Metaio
Metaio/Juneio
Diotasoft – iCyclope – Werber 2010
Breizhcamp 10

11. iCylcope
Video & Sound
Capture Camera & Tracking
Rendering OpenGL ES
Breizhcamp 11

12. Video & Sound
Sound mp3 Video pvr
video = [Video alloc];
for (int i = 0; i < countFile; i++){
[imgDataArray addObject:[[NSData alloc]
initWithContentsOfFile:path]];
}
Breizhcamp 12

13. Camera Capture & Tracking
Capture camera image
/*We setup the input*/
AVCaptureDeviceInput *captureInput = [AVCaptureDeviceInput …];
/*We setup the output*/
AVCaptureVideoDataOutput *captureOutput =
[AVCaptureVideoDataOutput alloc] ..];
From image to position
bool mTracked =
[tracking imgPointer width:width height:height :position];
Breizhcamp 13

14. Rendering OpenGL ES
initTexture();
updateTexture();
updatePlaneForVideo();
updateTexture(){
NSData * data = [mVideo getCurrentImage];
glCompressedTexImage2D(GL_TEXTURE_2D, …, [data bytes])
}
Breizhcamp 14

15. Mobile AR SDKs & APIs
Proprietary Public
Breizhcamp 15

16. Qualcomm AR SDK (QCAR)
Different target types
Breizhcamp 16

17. Workflow QCAR
Target Images
Mobile application
App Logic &
Rendering Engine
Target Position
QDevNet
QCAR library
Target Resources
Breizhcamp 17

18. QCAR API
Virtual Multi Image Image
Button Target Target
Marker Developer Application
Tracker
Camera Rendering
Android
Breizhcamp 18

19. Developing with QCAR
myAR_App.java
/** The main activity for the myAR_App */
public class myAR_App extends Activity{…}
myAR_Renderer.java
/** The renderer class for the myAR_App activity. */
public class myAR_Renderer implements GLSurfaceView.Renderer{…}
myAR_GLView.java
/** Configuring an translucent OpenGL surface view */
public class myAR_GLView extends GLSurfaceView{…}
myAR_NativeRenderer.cpp
/** OpenGL Rendering */
myNativeRenderFrame(JNIEnv *, jobject){…}
Breizhcamp 19

20. Developing with QCAR
myAR_App
/** The main activity for the myAR_App */
public class myAR_App extends Activity
{
protected void onCreate(Bundle savedInstanceState)
{
loadTextures();
// init myAR_GLView, init myAR_Renderer, ScreenOrientation, SplashScreen
initApplication();
mInitQCARTask = new InitQCARTask(); // Initialize QCAR asynchronously.
mInitQCARTask.execute();
mLoadTrackerTask = new LoadTrackerTask(); // Load the tracking data
mInitQCARTask.execute();
onQCARInitializedNative();//tracking of multiple targets simultaneously
startCamera();
}
}
Breizhcamp 20

21. Developing with QCAR
myAR_GLView
/** Configuring an translucent OpenGL surface view */
public class myAR_GLView extends GLSurfaceView
{
public void init(int flags, boolean translucent, int depth, int stencil)
{
this.getHolder().setFormat(PixelFormat.TRANSLUCENT);
setOpenGLContext(1.1/2.0);
}
}
Breizhcamp 21

22. Developing with QCAR
myAR_Renderer
/** The renderer class for the myAR_App activity. */
public class myAR_Renderer implements GLSurfaceView.Renderer
{
public void onSurfaceCreated(GL10 gl, EGLConfig config)
{
initRendering(); // native fct
QCAR.onSurfaceCreated();
}
/** Called when the surface changed size. */
public void onSurfaceChanged(GL10 gl, int width, int height)
{
updateRendering(width, height); // native fct
QCAR.onSurfaceChanged(width, height);
}
/** Called to draw the current frame. */
public void onDrawFrame(GL10 gl)
{
// Call native function to render content
myRenderFrameNative();
}
}
Breizhcamp 22

23. Developing with QCAR
myAR_NativeRenderer.cpp
myRenderFrameNative(JNIEnv *, jobject){
glClear(…);
// Render video background:
QCAR::Renderer::getInstance().begin();
for(i … state.getNumActiveTrackables()) {
QCAR::Trackable* trackable = state.getActiveTrackable(i);
modelViewMatrix = convertPose2GLMatrix(trackable->getPose());
glMatrixMode(GL_MODELVIEW);
glLoadMatrixf(modelViewMatrix.data);
// Add Your OpenGLES 1.1/2.0 Render Code Here
drawTeapot();
}
// finish GL Rendering
QCAR::Renderer::getInstance().end();
}
Breizhcamp 23

24. Developing with QCAR
Breizhcamp 24

25. QCAR and Unity3D
Breizhcamp 25

26. QCAR and Unity3D
Workflow
• Create Target QDevNet
• Add target resources to project
• Add ARCamera Prefab
• Add myTarget Prefab
• Add 3D objects, physics, shaders,
Game Logic, etc.
• Deploy on Android device
Breizhcamp 26

27. Sound-plugin for Unity3D
JavaActivity.java
SDK
SoundManager.java
NDK NativeJavaBridge.CPP (JNI)
Unity3D AndroidSoundPlugin.CS (C#)
Breizhcamp 27

28. Sound-plugin for Unity3D
public class JavaClass{ JavaActivity.java SoundManager.java
private Activity mActivity;
public static void initSounds(Context theContext)
public JavaClass(Activity currentActivity) {
mActivity = currentActivity;
{
SoundManager.initSounds(mActivity); …
SDK } mAudioManager =
(AudioManager) mContext.getSystemService
public int ActivitySoundPlay(int id, (Context.AUDIO_SERVICE);
float volume) { }
return SoundManager.playSound(id, volume,…);
}
jint JNI_OnLoad(JavaVM* vm, void* reserved) {
NativeJavaBridge.CPP (JNI)
jclass cls_Activity = jni_env->FindClass("com/unity3d/player/UnityPlayer");
jclass cls_JavaClass = jni_env->FindClass("com/diotasoft/pluginunity/JavaClass");
activitySoundPlay = jni_env->GetMethodID(cls_JavaClass, "ActivitySoundPlay", "(IFFIF)I");
…
}
NDK int playSound(int index, float volume, int repeat, float pitch) {
jint currentStream = (jint)jni_env->CallObjectMethod(JavaClass, activitySoundPlay, index, volume,
repeat, pitch);
…
}
public class AndroidSoundPoolPlugin : MonoBehaviour {
AndroidSoundPlugin.CS (C#)
[DllImport("javabridge")]
Unity private static extern int playSound(int index, float volume, int repeat, float pitch);
[DllImport("javabridge")]
private static extern void stopSound(int currentStream);
};
Breizhcamp 28

29. Diotasoft Games
The Circus ARLabyrinth
Breizhcamp 29

30. Conclusions
QCAR Android QCAR Unity
SDK/NDK
• Closed system
• Open system (hands on)
• Fast content creation
• Existant code integration
• Plugins & profiling
• Features you implement
• Rich feature sets provided
• Developer community
Breizhcamp 30

31. Future is Markerless
Breizhcamp 31

32. Merci beaucoup
Breizhcamp 32