O slideshow foi denunciado.

Fusing AI With AR for Android - 9/28/2019

1

Compartilhar

1 de 30
1 de 30

Fusing AI With AR for Android - 9/28/2019

1

Compartilhar

Baixar para ler offline

This presentation describes techniques for taking an existing example of Tensorflow Lite from the Tensorflow repo, and combining it with the Sceneform codelab in order to make an app that combines machine learning / object detection with augmented reality. It also covers some pitfalls and gotchas, of building an AR application, plus how to build your own custom Tensorflow Lite object detection model for Android by using either your own Tensorflow code, or AutoML plus Firebase for a more hands-off approach.

This presentation describes techniques for taking an existing example of Tensorflow Lite from the Tensorflow repo, and combining it with the Sceneform codelab in order to make an app that combines machine learning / object detection with augmented reality. It also covers some pitfalls and gotchas, of building an AR application, plus how to build your own custom Tensorflow Lite object detection model for Android by using either your own Tensorflow code, or AutoML plus Firebase for a more hands-off approach.

Mais Conteúdo rRelacionado

Audiolivros relacionados

Gratuito durante 30 dias do Scribd

Ver tudo

Fusing AI With AR for Android - 9/28/2019

  1. 1. Stephen Wylie GDG Dallas @SWebCEO Fusing AI with AR NH GDG DevFest 9/28/2019
  2. 2. NH GDG DevFest ●GDE in Machine Learning ●Software Engineer @ rewardStyle ●Fix old arcade machines & computers, and pinballs of all eras ●Love fancy men’s clothing & vintage TV game shows ●Lots of patents in this area About Me
  3. 3. AI + AR: Does anyone do this themselves?
  4. 4. NH GDG DevFest Sports ● Homecourt- Practice basketball, get coaching ● MLB - ID players, show stats AI + AR: Does anyone do this themselves? Yes, of course! But with help... APIs & Frameworks Are Your Friends! Retail ● Burberry - Customize items, detect counterfeits ● Cup Noodles - Turn cups into musical instruments ● Wayfair, IKEA - Preview items in your own house Vehicle Recognition & Auto Loans ● Blippar - AR/VR company ● Capital One - AR feature of Auto Navigator ● USAA - Loans & insurance
  5. 5. Key Terms
  6. 6. Key Terms In AI ● MobileNet, SqueezeNet, Inception, ResNet Neural networks optimized for computer vision tasks ● Tensorflow Lite Inference engine optimized to run .tflite models on mobile devices ● MLKit APIs that enable specific Machine Learning use cases on devices or cloud In AR ● ARCore Cross-platform augmented reality SDK using OpenGL for motion tracking, scene building ● Sceneform AR SDK for Android saving you from learning OpenGL
  7. 7. Establish your Android app
  8. 8. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFra gment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
  9. 9. NH GDG DevFest ●Latest: ●Java JDK 10 (not 11, not 8) ●Building in Windows? You’ll need ~5GB worth of: ○Visual C++ Build Tools - visualcppbuildtools_full.exe ■Windows 10 SDK 10.0.10240 ■.NET Framework SDK ○A real shell like MSYS2 ●Pick your poison! Download Tensorflow source from GitHub or Docker Install Prerequisites ●Android Studio ●Android API level ●Android NDK ●Gradle ●Bazel
  10. 10. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFra gment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
  11. 11. NH GDG DevFest Code root: tensorflow/tensorflow/lite/examples/android/app ●Consider copying to a different folder ●Add Sceneform dependencies to Gradle build ●Update Gradle build to latest Tools, API >= 26 ●Add AR permissions to Android Manifest ●Make new Layout incorporating ArFragment ●Use this Layout in your CameraActivity ○Or make it a separate class ○Update Manifest with new Activity ○Ensure DetectorActivity extends it Gradle downloads ML model assets Start Modifying Code
  12. 12. Code Thus Far
  13. 13. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFra gment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
  14. 14. NH GDG DevFest ●Define androidsdk and androidndk path in WORKSPACE ●Omit API level & build tools version ●cd to tensorflow/ ●Run bazel build //tensorflow/lite/examples/android:tflite_demo ●Funky Windows business ●Pray hard! Build with Bazel
  15. 15. What Success Looks Like
  16. 16. Code Thus Far
  17. 17. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFra gment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
  18. 18. NH GDG DevFest ●Can’t use Bazel anymore ○Incorporate object detection model into src/main/jniLibs ○Set parameter for CPU architecture in Gradle ●Sceneform Plugin or Gradle Instruction ○Sceneform codelab calls for 1.4; Android Studio plugin is 1.6 ○Best to just use Gradle instructions to create SFB assets ●Add augmented reality code to MultiBoxTracker Add Augmented Reality Code
  19. 19. Final Product
  20. 20. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFra gment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
  21. 21. Pitfalls of AI + AR
  22. 22. Battery Life A Pitfall of AI + AR “Turn your phone into an incendiary device! Run AI & AR at the same time.” Throttling AI is key.
  23. 23. NH GDG DevFest Depth & size of object Another Pitfall of AI + AR Projecting 3D object in AR requires good understanding of 3D space: ●Size of object, in pixels, proportional to the camera ●Size of object, in real life, to position 3D object correctly ●Positioning 3D object visible to user without maneuvering ●Improvement: Correlate bounding box with hit detection
  24. 24. NH GDG DevFest Sensor drift Yet Another Pitfall of AI + AR Object out of the viewport? ●AR relies on point clouds, hardware sensors ●Hardware sensors likely to drift with lots of motion ●Objects anchored to world space likely to wander off Solution? ●Ensure scene has good contrast, enough corners to see every frame ●Attach objects to anchors ●Scan enough of the scene ahead of time before drawing AR
  25. 25. NH GDG DevFest Tracking Multiple Objects And Yet Another Pitfall of AI + AR ●Object detections could be missed, false, or switch IDs ●Discover correlations between objects moving on camera ○Pairwise dependencies ○Uniform or non-uniform hypergraph ●Multiple object tracking is easier with the tensorflowlite_demo library ●No library? Gotta code it yourself! Mitigating This ●Consumption issues? Minimize anchors as you instantiate objects ●Take the time to set up NDK and/or libraries
  26. 26. Building Your Model
  27. 27. NH GDG DevFest ●Garbage in, garbage out ● Need at least 100 samples per category ○ Kaggle ○ Record your own videos ● Keras imagedatagenerator ● Label your own bounding boxes Collecting Data
  28. 28. NH GDG DevFest Building Your Model Tensorflow Cloud AutoML + Firebase
  29. 29. NH GDG DevFest ●tflite_convert binary available in Github code ○ Convert a GraphDef ○ Convert a SavedModel ○ Convert a Keras model ● Or, use AutoML Vision Object Detection Beta ○ AutoML Vision Edge in ML Kit - Export to Android using Firebase ○ Models in this form are quantized - i.e. ints, not floats ○ Small, edge device compatible; may lose accuracy Converting Your Model
  30. 30. Thank You! ML Summit ‘19 #MLSummit @SWebCEO goshtastic.blogspot.com github.com/mrcity/mlworkshop Slideshare: StephenWylie3

×