MIPI DevCon Bangalore 2017: Imaging Systems Design for Mixed Reality Scenarios
•
0 gostou•212 visualizações
This talk will tie the different usage opportunities to imaging requirements for a Mixed Reality system for both consumer and enterprise markets in the coming years and dive into system design aspects like placement, location, types of image sensors, bandwidth requirements for tethered and wireless HMD scenarios and the processing pipeline architecture with the critical technology building blocks from multiple camera sources distributed between a HMD and a host system. Example end usages like Obstacle Avoidance and Avatar Navigation with Mixed Reality headsets will be used to provide a use case decomposition view from capture to application.
Recomendados
Recomendados
Mais conteúdo relacionado
Destaque
Destaque (10)
Semelhante a MIPI DevCon Bangalore 2017: Imaging Systems Design for Mixed Reality Scenarios
Semelhante a MIPI DevCon Bangalore 2017: Imaging Systems Design for Mixed Reality Scenarios (20)
Mais de MIPI Alliance
Mais de MIPI Alliance (20)
Último
Último (6)
MIPI DevCon Bangalore 2017: Imaging Systems Design for Mixed Reality Scenarios
- 1. Prasanna Krishnaswamy Intel Platform Architect Imaging Systems Design for Mixed Reality Scenarios
- 2. © 2017 MIPI Alliance, Inc. Agenda • Mixed Reality (MR) – Introduction • Key Use Cases • MR System Design Overview • Imaging Sensors for MR • Use Case Decomposition • Opportunities 2 Intel
- 3. © 2017 MIPI Alliance, Inc. Mixed Reality - Introduction 3 Intel • Experiences that overlay graphics on video streams of physical world are Augmented Reality • Experiences that occlude your view to present a digital experience are Virtual Reality • Experiences enabled between these two extremes are Mixed Reality, which is blending of the physical world and digital world Diagrams from Microsoft Mixed Reality Academy https://developer.microsoft.com/en- us/windows/mixed-reality/academy
- 5. © 2017 MIPI Alliance, Inc. System Design Overview 5 Intel 2MP-13MP FE Global Shutter FE Global Shutter Active/Pa ssive Stereo Data Coalescing and/or Compression 6DOF and Dense SLAM Decode + RGB Dewarp + Depth Stitching GPU / ISP CVA GPU/ CVA Gesture Detection Face/Person/Obj ect Detection and Tracking Activity Semantic Segmentation Material Understanding CPU/ CVA GPU/ CVA GPU/ CVA 3D Reconstruction OS Services or SDK Plugins Holo API Unity Plugins GPU MR Application Compression GPU/ISP/CVA Acceleration Active/Pa ssive Stereo MIPI/USB Monocular vs Binocular SLAM Workload Partitioning GPU/CVA Acceleration Metadata Containers IMU I2C MR Camera System Design Image Pre Processing Pipe Inside Out 6DOF Real time reconstruction MR Workload Distribuiton and Representation VideoSee ThroughCamera Tracking Cameras Depth Cameras MIPI/USB MIPI/USB MIPI/USB MIPI/USB
- 6. © 2017 MIPI Alliance, Inc. Image Sensors for MR 6 Intel Type of Sensor Application Typical Specification for MIPI CSI-2℠ Imaging Sensors System Design Considerations Tracking Sensor Inside out 6DOF WFOV, Fish Eye, Global Shutter, Monochrome, >120fps, >720p Wider FOV than display to track in periphery Depth Sensor 3D Reconstruction, Semantic, Gestures Active vs Passive, Upto 1080p/30fps,0.3m-5m, few mm error Independent of tracking sensor RGB Sensor Texture, See Through Mode Upto 13MP/30fps for high texture Match Display Resolution & fps for See Through Mode (trending 4k/120fps) Configurable ISP processing for simultaneous high quality texture and video see through Event Sensor Inside out Tracking, Object Tracking QVGA @ 2k fps, FOV matching depth camera Augmented with RGB and Depth camera Hyper Spectral or Thermal Sensors Fast Classification RGB-NIR/LWIR Sensor, QVGA @ 30fps Low power ROI or event generation
- 7. © 2017 MIPI Alliance, Inc. MIPI CSI-2SM Advantages in MR 7 Intel • Allows flexible ISP, Vision Accelerator and Host SOC combinations in MR enabling segmentation and mobility VPU -> Vision Processing Unit, typically consists of CV and Neural Network accelerators All in One MR HMD with vision processing distributed between a host and a vision processing unit ISP & VPU SOC MIPI Rx MIPI Tx MIPI Rx SOC with ISP MIPI Rx All in One MR HMD with all vision processing on host MR HMD tethered to a host PC MIPI To Host Converter SOC MIPI Rx HMD Host USB3 or TBT CSI-2 CSI-2 CSI-2 CSI-2 CSI-2 CSI-2 CSI-2 CSI-2 CSI-2 CSI-2
- 8. © 2017 MIPI Alliance, Inc. MIPI CSI-2 Advantages in MR 8 Intel • Total power consumption on HMD, especially for optical see through or AIO HMD with MIPI CSI-2 ISP & VPU SOC MIPI Rx MIPI Tx MIPI Rx CSI-2 Camera Type Typical Power Depth Camera 300mW - 1.5W RGB Camera 300mw - 1W Tracking Camera 300 - 500 mw Total power dissipated on a AIO HMD can be anywhere from 10-25W. Camera power starts to play a significant portion of the same CSI-2 CSI-2 CSI-2
- 10. © 2017 MIPI Alliance, Inc. MIPI CSI-2 Future Opportunities for MR 10 Intel • Custom camera modules for MR – Different combinations that can be integrated into MR HMD’s • Metadata Containers – Standardized Metadata for MR vision analytics from on board HMD for consumption by host • MIPI to USB/TBT/Wifi Reference Designs for MR Headsets – Enables users to buy off market lower cost HMD’s to be used with PC’s increasing adoption • Low power event sensing for MR – Standardized event triggers from event sensors to feed into other sensors for better power management • Adoption of Beyond Visible Cameras – Adoption of MIPI based Thermal, Hyperspectral sensors for MR use cases
- 11. © 2017 MIPI Alliance, Inc. NOTICES AND DISCLAIMERS • Intel, the Intel logo, Intel® are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. • *Other names and brands may be claimed as the property of others. • Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com. • Intel technologies may require enabled hardware, specific software, or services activation. Check with your system manufacturer or retailer. • No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. 11 Intel®