O slideshow foi denunciado.

Fusing Artificial Intelligence with Augmented Reality on Android - 1 Feb. 2019

0

Compartilhar

Carregando em…3
×
1 de 19
1 de 19

Fusing Artificial Intelligence with Augmented Reality on Android - 1 Feb. 2019

0

Compartilhar

Baixar para ler offline

Descrição

Here is an overview of how to use open-source libraries and existing Google codelabs to synthesize augmented reality and artificial intelligence into a single application. The app performs object detection to guess a 2D bounding box around a detected object. The center of the box is provided to a hit test that, if successful, will be used as the coordinate to place a 3D model in the AR environment at that location.

The technologies involved are native Android in Java, Tensorflow Lite, Sceneform AR library, Bazel build, Gradle build, and contains extra instructions for Windows users.

Besides just wiring up the libraries to work together, there are other pitfalls and gotchas that must be considered, and those are touched upon near the end of the deck.

YouTube video from 1 Feb. 2019: TBD
GitHub repo of final project: TBD

Transcrição

  1. 1. Fusing AI with AR By Stephen Wylie Windy City DevFest - Chicago Feb. 1, 2019
  2. 2. About Me ● Google Developer Expert in Machine Learning ● Fix old arcade machines & computers, and pinballs of all eras ● Love fancy men’s clothing & vintage TV game shows ● Lots of patents in this area
  3. 3. AI + AR: Does anyone do this themselves? Yes, of course! But with help… Vehicle Recognition & Auto Loans ● Blippar - AR/VR company ● Capital One - AR feature of Auto Navigator ● USAA - Loans & insurance APIs & Frameworks Are Your Friends! Retail ● Burberry - Customize items, detect counterfeits ● Cup Noodles - Turn cups into musical instruments Sports ● Homecourt- Practice basketball, get coaching ● MLB - ID players, show stats
  4. 4. Key Terms ● MobileNet, SqueezeNet, Inception, ResNet Neural networks optimized for computer vision tasks ● Tensorflow Lite Inference engine optimized to run .tflite models on mobile devices ● MLKit APIs that enable specific Machine Learning use cases on devices or cloud ● ARCore Cross-platform augmented reality SDK using OpenGL for motion tracking, scene building ● Sceneform AR SDK for Android saving you from learning OpenGL
  5. 5. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFragment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
  6. 6. Install Prerequisites ● Latest: ● Java JDK 10 (not 11, not 8) ● Building in Windows? You’ll need ~5GB worth of: ○ Visual C++ Build Tools - visualcppbuildtools_full.exe ■ Windows 10 SDK 10.0.10240 ■ .NET Framework SDK ○ A real shell like MSYS2 ● Pick your poison! Download Tensorflow source from GitHub or Docker - Android Studio - Android API level - Android NDK - Gradle - Bazel
  7. 7. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFragment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
  8. 8. Start Modifying Code Code root: tensorflow/tensorflow/lite/examples/android/app ● Consider copying to a different folder ● Add Sceneform dependencies to Gradle build ● Update Gradle build to latest Tools, API >= 26 ● Add AR permissions to Android Manifest ● Make new Layout incorporating ArFragment ● Use this Layout in your CameraActivity ○ Or make it a separate class ○ Update Manifest with new Activity ○ Ensure DetectorActivity extends it Gradle downloads ML model assets
  9. 9. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFragment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
  10. 10. Build with Bazel ● Define androidsdk and androidndk path in WORKSPACE ● Omit API level & build tools version ● cd to tensorflow/ ● Run bazel build //tensorflow/lite/examples/android:tflite_demo ● Funky Windows business in “Windows Kits” directory ● Pray hard!
  11. 11. What Success Looks Like
  12. 12. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFragment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
  13. 13. Add Augmented Reality Code ● Can’t use Bazel anymore ○ Incorporate object detection model into src/main/jniLibs ○ Set parameter for CPU architecture in Gradle ● Sceneform Plugin or Gradle Instruction ○ Sceneform codelab calls for 1.4; Android Studio plugin is 1.6 ○ Best to just use Gradle instructions to create SFB assets ● Add augmented reality code to MultiBoxTracker
  14. 14. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFragment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
  15. 15. Pitfalls of AI + AR: Battery life “Turn your phone into an incendiary device! Run AI & AR at the same time.” Throttling AI is key.
  16. 16. Pitfalls of AI + AR: Depth & size of object Projecting 3D object in AR requires good understanding of 3D space: ● Size of object, in pixels, proportional to the camera ● Size of object, in real life, to position 3D object correctly ● Positioning 3D object visible to user without maneuvering ● Improvement: Correlate bounding box with hit detection
  17. 17. Pitfalls of AI + AR: Sensor drift Object out of the viewport? ● AR relies on point clouds, hardware sensors ● Hardware sensors likely to drift with lots of motion ● Objects anchored to world space likely to wander off Solution? ● Ensure scene has good contrast, enough corners to see every frame ● Attach objects to anchors ● Scan enough of the scene ahead of time before drawing AR
  18. 18. Pitfalls of AI + AR: Tracking multiple objects ● Object detections could be missed, false, or switch IDs ● Discover correlations between objects moving on camera ○ Pairwise dependencies ○ Uniform or non-uniform hypergraph ● Multiple object tracking is easier with the tensorflowlite_demo library ● No library? Gotta code it yourself! Mitigating This ● Consumption issues? Minimize anchors as you instantiate objects ● Take the time to set up NDK and/or libraries
  19. 19. Thank You @SWebCEO goshtastic.blogspot.com github.com/mrcity/mlworkshop Slideshare: StephenWylie3

Notas do Editor

  • Imagine recognizing a chair with your app. A normal-sized chair might be detected and you might understand it’s about 3 feet tall and 2 feet wide. This is useful to know when you’re sizing the billboard But then how far away from the camera is the chair? You’ll need to know that in order to find out where to place the billboard in space.

    Now imagine if your camera picks up on a tiny chair in a dollhouse. Your original billboard would be far too big to display with the chair if it were correctly placed above the chair. But, more likely, it would be placed way beyond the chair, and then the billboard would only appear correctly if you hold your phone at a specific angle. To capture the tiny dollhouse chair correctly, we need to consider our options:
  • Descrição

    Here is an overview of how to use open-source libraries and existing Google codelabs to synthesize augmented reality and artificial intelligence into a single application. The app performs object detection to guess a 2D bounding box around a detected object. The center of the box is provided to a hit test that, if successful, will be used as the coordinate to place a 3D model in the AR environment at that location.

    The technologies involved are native Android in Java, Tensorflow Lite, Sceneform AR library, Bazel build, Gradle build, and contains extra instructions for Windows users.

    Besides just wiring up the libraries to work together, there are other pitfalls and gotchas that must be considered, and those are touched upon near the end of the deck.

    YouTube video from 1 Feb. 2019: TBD
    GitHub repo of final project: TBD

    Transcrição

    1. 1. Fusing AI with AR By Stephen Wylie Windy City DevFest - Chicago Feb. 1, 2019
    2. 2. About Me ● Google Developer Expert in Machine Learning ● Fix old arcade machines & computers, and pinballs of all eras ● Love fancy men’s clothing & vintage TV game shows ● Lots of patents in this area
    3. 3. AI + AR: Does anyone do this themselves? Yes, of course! But with help… Vehicle Recognition & Auto Loans ● Blippar - AR/VR company ● Capital One - AR feature of Auto Navigator ● USAA - Loans & insurance APIs & Frameworks Are Your Friends! Retail ● Burberry - Customize items, detect counterfeits ● Cup Noodles - Turn cups into musical instruments Sports ● Homecourt- Practice basketball, get coaching ● MLB - ID players, show stats
    4. 4. Key Terms ● MobileNet, SqueezeNet, Inception, ResNet Neural networks optimized for computer vision tasks ● Tensorflow Lite Inference engine optimized to run .tflite models on mobile devices ● MLKit APIs that enable specific Machine Learning use cases on devices or cloud ● ARCore Cross-platform augmented reality SDK using OpenGL for motion tracking, scene building ● Sceneform AR SDK for Android saving you from learning OpenGL
    5. 5. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFragment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
    6. 6. Install Prerequisites ● Latest: ● Java JDK 10 (not 11, not 8) ● Building in Windows? You’ll need ~5GB worth of: ○ Visual C++ Build Tools - visualcppbuildtools_full.exe ■ Windows 10 SDK 10.0.10240 ■ .NET Framework SDK ○ A real shell like MSYS2 ● Pick your poison! Download Tensorflow source from GitHub or Docker - Android Studio - Android API level - Android NDK - Gradle - Bazel
    7. 7. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFragment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
    8. 8. Start Modifying Code Code root: tensorflow/tensorflow/lite/examples/android/app ● Consider copying to a different folder ● Add Sceneform dependencies to Gradle build ● Update Gradle build to latest Tools, API >= 26 ● Add AR permissions to Android Manifest ● Make new Layout incorporating ArFragment ● Use this Layout in your CameraActivity ○ Or make it a separate class ○ Update Manifest with new Activity ○ Ensure DetectorActivity extends it Gradle downloads ML model assets
    9. 9. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFragment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
    10. 10. Build with Bazel ● Define androidsdk and androidndk path in WORKSPACE ● Omit API level & build tools version ● cd to tensorflow/ ● Run bazel build //tensorflow/lite/examples/android:tflite_demo ● Funky Windows business in “Windows Kits” directory ● Pray hard!
    11. 11. What Success Looks Like
    12. 12. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFragment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
    13. 13. Add Augmented Reality Code ● Can’t use Bazel anymore ○ Incorporate object detection model into src/main/jniLibs ○ Set parameter for CPU architecture in Gradle ● Sceneform Plugin or Gradle Instruction ○ Sceneform codelab calls for 1.4; Android Studio plugin is 1.6 ○ Best to just use Gradle instructions to create SFB assets ● Add augmented reality code to MultiBoxTracker
    14. 14. Establish your Android app Preparation Install prereqs Install latest versions of APIs, SDKs, IDEs, and build tools Preparation Clone repo Get Tensorflow source from Docker, Github, etc. Understand & Build Open app code Edit app code to use Sceneform fragment rather than Camera- ConnectionFragment Bust your head Build with Bazel This smooths object detection, improves tracking on recognized entities Build More Add AR code Hit test the bounding box and draw the 3D model relative to that point Finish Profit! Or, fix problems and build your own TFLite model
    15. 15. Pitfalls of AI + AR: Battery life “Turn your phone into an incendiary device! Run AI & AR at the same time.” Throttling AI is key.
    16. 16. Pitfalls of AI + AR: Depth & size of object Projecting 3D object in AR requires good understanding of 3D space: ● Size of object, in pixels, proportional to the camera ● Size of object, in real life, to position 3D object correctly ● Positioning 3D object visible to user without maneuvering ● Improvement: Correlate bounding box with hit detection
    17. 17. Pitfalls of AI + AR: Sensor drift Object out of the viewport? ● AR relies on point clouds, hardware sensors ● Hardware sensors likely to drift with lots of motion ● Objects anchored to world space likely to wander off Solution? ● Ensure scene has good contrast, enough corners to see every frame ● Attach objects to anchors ● Scan enough of the scene ahead of time before drawing AR
    18. 18. Pitfalls of AI + AR: Tracking multiple objects ● Object detections could be missed, false, or switch IDs ● Discover correlations between objects moving on camera ○ Pairwise dependencies ○ Uniform or non-uniform hypergraph ● Multiple object tracking is easier with the tensorflowlite_demo library ● No library? Gotta code it yourself! Mitigating This ● Consumption issues? Minimize anchors as you instantiate objects ● Take the time to set up NDK and/or libraries
    19. 19. Thank You @SWebCEO goshtastic.blogspot.com github.com/mrcity/mlworkshop Slideshare: StephenWylie3

    Notas do Editor

  • Imagine recognizing a chair with your app. A normal-sized chair might be detected and you might understand it’s about 3 feet tall and 2 feet wide. This is useful to know when you’re sizing the billboard But then how far away from the camera is the chair? You’ll need to know that in order to find out where to place the billboard in space.

    Now imagine if your camera picks up on a tiny chair in a dollhouse. Your original billboard would be far too big to display with the chair if it were correctly placed above the chair. But, more likely, it would be placed way beyond the chair, and then the billboard would only appear correctly if you hold your phone at a specific angle. To capture the tiny dollhouse chair correctly, we need to consider our options:
  • Mais Conteúdo rRelacionado

    Livros relacionados

    Gratuito durante 30 dias do Scribd

    Ver tudo

    Audiolivros relacionados

    Gratuito durante 30 dias do Scribd

    Ver tudo

    ×