O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

[SP02] Developing autonomous vehicles with AirSim

AirSim is a simulator for autonomous vehicles built on Unreal Engine. AirSim provides a rich platform to develop autonomy by enabling use of AI technologies, such as deep learning, computer vision, reinforcement learning etc. It is open-source, cross platform and supports hardware-in-loop simulation, thus allowing rapid development and testing of the system. The simulation is developed as a plugin and can be simply be dropped into any Unreal environment. AirSim supports AI development capabilities by exposing APIs to enable data logging and controlling vehicles in a platform independent manner. We will give an overview of how to use AirSim for building realistic simulation environments and doing development for quadrotors that use popular flight controllers such as Pixhawk. It is developed as a plugin that can simply be dropped in to any Unreal environment you want. We will also showcase how the system can be used to incorporate machine learning components useful for building such autonomous systems.

製品/テクノロジ: AI (人工知能)/Deep Learning (深層学習)/Linux/Machine Learning (機械学習)/OSS/Windows

Ashish Kapoor
Microsoft Corporation
Microsoft Research
Principal Researcher

Shital Shah
Microsoft Corporation
Microsoft Research
Principal Research Software Development Engineer

  • Entre para ver os comentários

[SP02] Developing autonomous vehicles with AirSim

  1. 1. Part I: AI and Simulation Ashish Kapoor
  2. 2. Introducing Microsoft Research
  3. 3. Aerial Robotics at MSR • Dedicated Drones Lab • Hardware Lab • 333 Exemption • Indoor OptiTrack • Part 107 • Approved Test Sites
  4. 4. Aerial Robotics at MSR
  5. 5. Many Recent Successes of Machine Learning Peng et al. 2016 Mnih et al. 2013Silver et al. 2016 1. Close-world systems (also only in software) 2. Luxury to obtain near-infinite data via simulation etc.
  6. 6. Can we use such ML methods to build systems that operate in real-world?
  7. 7. Successes of Machine Intelligence Real World Flying Systems Peng et al. 2016 Mnih et al. 2013Silver et al. 2016
  8. 8. Fast Aerobatic Flights with Safety Constraints 11
  9. 9. Infinite Soaring Machine
  10. 10. Optimal Scanning: What Trajectory to Fly?
  11. 11. Racing FPV Quadrotors
  12. 12. Three Fundamental Challenges Lack of large amount of data • High Sample Complexity of ML Methods Computational Constraints • Real-time Performance, Limited Memory and Compute Power Operating in the Open World • Safety, Uncertainty, Mixed- Initiative Autonomy
  13. 13. Part II: Getting to Know AirSim Shital Shah
  14. 14. A Day at Office Ultrasonic sensors Stereo vision Companion computer
  15. 15. Current State of UAV Simulators
  16. 16. AirSim: The Next Generation Simulator
  17. 17. What Does Simulator Enables? • Wide variety of environment, day of time, weather patterns • No legal hassle, much cheaper and safe Generate lots of training data • Run exact same code that would be run onboard • Slow down or accelerate simulated time Develop autonomy algorithms • If it doesn’t work in simulator then it won’t work in real world • Use sensors with varies parameters Test perception algorithms • Can fail thousands of time to learn patterns • Run in cloud for distributed learning Reinforcement Learning techniques
  18. 18. Why Use Unreal Engine? Actual footage captured from simulated drone in AirSim with Open World Environment
  19. 19. The Heart of the Vehicle Sensor Input Motor Output Sensors generate information about the world around Motor voltages drives the propeller Humans provide desired state Sensor Input Desired State
  20. 20. How Simulator Works? Sensor Input Motor Output Desired State Humans provide desired state Generate all the sensor data using ground truth Run physics engine to get Lin & Ang • Position • Velocity • Acceleration
  21. 21. What Does Physics Engine Do? Physics Engine Force Torque Position Velocity Acceleration Linear & Angular Flavors Total of 3 + 3 = 6 vectors
  22. 22. The Physics Loop Initialize Get all the forces and torques Find 𝑥𝑥, ̇𝑥𝑥, ̇̇𝑥𝑥 𝛼𝛼, ̇𝛼𝛼, ̇̇𝛼𝛼 Time elapsed dt
  24. 24. Physics - Linear Dynamics 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 = 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚 𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑦𝑦𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 += 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 ∗ 𝑑𝑑𝑑𝑑 𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 += 𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑣𝑦𝑦𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 ∗ 𝑑𝑑𝑑𝑑 *In real physics engine you would use better integrator than Euler
  25. 25. Physics - Angular Dynamics 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 = 𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 − 𝜔𝜔 × (𝐼𝐼 𝜔𝜔) 𝐼𝐼 𝜔𝜔𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 += 𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 ∗ 𝑑𝑑𝑑𝑑 𝑞𝑞𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 = 𝑞𝑞𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 ∗ 𝑄𝑄(𝜔𝜔 ∗ 𝑑𝑑𝑑𝑑) *In real physics engine you would use better integrator than Euler *Calculations are in body frame
  26. 26. Simulating IMU Physics engine tells us, • Angular velocity • Linear acceleration So just add noise, bias and drift! 𝐼𝐼 𝐼𝐼 𝐼𝐼 = 𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺𝐺 + 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴
  27. 27. Simulating Barometer
  28. 28. Simulating Magnetometer
  29. 29. AirSim is Open Source Unreal Engine 4 • Designed as UE4 plugin • Just drop in to 100s of realistic environments APIs for Dozens of Languages • Get camera images • Send commands to the vehicle C++ Header- Only Library • Eigen library as only dependency • Cross-platform Highly Extensible • Add new vehicles models • Add new modes
  30. 30. Experiment: Simulation vs. Reality Simulated trajectory in purple, real trajectory in red
  31. 31. AirSim Has APIs from PythonClient import * import cv2 import sys client = AirSimClient('') # get depth image result = client.setImageTypeForCamera(0, AirSimImageType.Scene) # show image in opencv rawImage = np.fromstring(result, np.int8) png = cv2.imdecode(rawImage, cv2.IMREAD_UNCHANGED) cv2.imshow("Camera Image", png) Few lines of Python code can get you FPV image from drone! Scene from Unreal Boy with a Kite environment
  32. 32. Make Drone Move in AirSim Using APIs from PythonClient import * import sys client = AirSimClient('') # Stay 5 meters above ground z = -5 # Fly! client.moveOnPath([(0,-253,z),(125,-253,z),(125,0,z),(0,0,z)], 15, 0, DrivetrainType.ForwardOnly, YawMode(False,0), 20, 1) Same code can be ran from offboard computer on real drone! Other languages available: C++, C#, Java and many more!
  33. 33. Make Drone Move in AirSim Using APIs
  34. 34. AirSim Extensibility New Vehicle Types Car Apollo Moon Lander New Sensor Types Ultrasonics LIDAR New Physics Engines PhysX Bullet New Sim Modes Computer Vision Just Physics You can contribute on GitHub: https://github.com/microsoft/airsim
  35. 35. Part III: Applications & Future Ashish Kapoor
  36. 36. AgIoT: Precision Agriculture Scan farm using drone to capture low level details on daily basis, analyze differences each day and fuse with sensor information to identify areas that needs specific work
  37. 37. AgIoT: Vision 40 Spatio-temporal view of the farm Sensors & UAVs Yield estimation Precision Irrigation Pest Infection Fertilizer application …Ag Services
  38. 38. AgIoT:System Architecture 41 Azure ML Analytics • Monsanto • Cargill • DuPont • Syngenta • Sakata • Fertilizer • … Cameras, Drones, sensors Weather Data (Rain, Wind, Pollen) Recommendations (daily best practices) Farm Aggregator (IOT Edge) Farm Aggregator (IOT Edge)
  39. 39. Autonomous 3D Scanning of Large Structures 3D reconstruction in simulator using a simulated drone flight 3D reconstruction in real world using actual drone
  40. 40. Autonomous 3D Scanning of Large Structures
  41. 41. Thank You! In Collaboration With: Debadeepta Dey, Neel Joshi, Andrey Kolobov, Chris Lovett, Jim Piavis, Gireeja Ranade, Shital Shah, Sudipta Sinha Interns: Sanjiban Choudhury, Mike Roberts, Niteesh Sood, Wen Sun
  42. 42. セッションアンケートにご協力ください  専用アプリからご回答いただけます。 decode 2017  スケジュールビルダーで受講セッションを 登録後、アンケート画面からご回答ください。  アンケートの回答時間はたったの 15 秒です!
  43. 43. Ask the Speaker のご案内 本セッションの詳細は『Ask the Speaker Room』各コーナーカウンタにて ご説明させていただきます。是非、お立ち寄りください。
  44. 44. Appendix
  45. 45. Architecture
  46. 46. Evaluation: Simulation Vs. Reality
  47. 47. Optimal Scanning: What Trajectory to Fly? • Safety: How risky are the flight maneuvers? • Efficiency: How long does the process take? • Performance: How good is the 3D model?
  48. 48. © 2017 Microsoft Corporation. All rights reserved. 本情報の内容(添付文書、リンク先などを含む)は、作成日時点でのものであり、予告なく変更される場合があります。