O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
Matteo Valoriani
Nice to Meet You
Matteo Valoriani
PhD at Politecnico of Milano
CEO of Fifth Element
Speaker and Consultant
Microsoft MVP f...
Usage Scenarios
TRAININGRETAIL HEALTHCARETHERAPY EDUCATION
You have to be a
magician…
… or at least a
good illusionist
“Any sufficiently
advanced
technology is
indistinguishable
from magic”
(Arthur C. Clarke)
Sensor Anatomy
Kinect 2 - Specs
3D DEPTH SENSOR
RGB CAMERA
MULTI-ARRAY MIC
Hardware:
Depth resolution:
512×424
RGB resolution:
1920×1080 ...
Kinect Adapter
Kinect 2 vs Kinect 1
Version 1 Version 2
Depth range 0.4m → 4.0m 0.4m → 4.5m
Color stream 640×480 1920×1080
Depth stream 320×240 512×424
Infrar...
System / Software Requirements
OS Windows 8, 8.1, Embedded 8, Embedded 8.1 (x64)
CPU Intel Core i7 3.1GHz (or higher)
RAM ...
DEMO
Kinect Evolution
Accessing Kinect Data Sources
High-level architecture
Basic Flow of Programming
Sensor Stream Frame Data
Sensor Source Reader Frame Data
Kinect for Windows SDK v1
Kinect for Wi...
• In “New Project” create a new Windows Store app
• Enable Microphone and Webcam capabilities
• Add a reference to Microso...
Represents a single physical sensor
Always valid: when device is disconnected no more frame are
generated.
Use IsAviable P...
Sources
Give access to frames
– Events
– Polling
Multiple readers may be
created on a single source
Readers can be paused
Readers
...
Frame references
void InfraredFrameReader
InfraredFrameArrivedEventArgs
using (InfraredFrame frame =
args.FrameReference.A...
• Gives access to the frame data
– Make a local copy or access the underlying buffer directly
• Contains metadata for the ...
• Allows the app to get a matched set of frames from multiple
sources on a single event
• Delivers frames at the lowest FP...
System
Kinect
Driver
Kinect SDK
Application
Kinect
Driver
Kinect SDK
Application
Kinect
Service
Kinect SDK
Application
Kin...
Connection
Kinect for Windows v1 Kinect for Windows v2
PCPC
Example Multiple Connection
PCPC PC
Hub
Server
DEMO
Display frame in a Windows Store App
Data Source Details
Physical image sensors, ColorFrameSource,
InfraredFrameSource, DepthFrameSource,
BodyIndexFrameSource,...
Kinect Data Sources
• 30 or 15 fps, based on lighting
conditions
ColorFrameSource
InfraredFrameSource
– 16-bit distance in millimeters from the
sensor’s focal plane
DepthFrameSource
– 0 to 5: Index of the corresponding body,
as tracked by the body source
– > 5: No tracked body at that pixel
BodyIndexFra...
• Range is 0.5-4.5 meters
• Frame data is a collection of Body objects each
with 25 joints
– Each joint has position in 3D...
Skeleton vs Body
Coordinate System
 ColorSpace (Coordinate System of the Color
Image)
– … Color
 DepthSpace (Coordinate System of the
Dep...
• Three coordinate systems
• Coordinate mapper provides conversions between each system
• Convert single or multiple point...
DEMO
Handle body frames and coordinate mapping
Kinect Studio
Recording and Playback
Recordable Data Sources
Infrared
13 MB/s
Depth
13 MB/s
BodyFrame
BodyIndex
Color
120 MB/s
Audio
32 KB/s
Legend
Record/Play...
DEMO
Kinect Studio 2.0
Recording | Playback
Gesture Recognition
Heuristic + Machine Learning Approaches
DEMO
Programmatic detection of a gesture/pose
Gesture Builder
• New tool, shipping with v2 SDK
• Organize data using projects and solutions
• Give meaning to data by ta...
Visual Gesture Builder
Your Application
DEMO
Machine Learning Demo
Heuristic
• Gesture is a coding problem
• Quick to do simple
gestures/poses (hand over head)
• ML can also be useful to fi...
Hand Pointer Gestures
Hand pointer gestures
Engagement Targeting
Press
Panning/Zoom
DEMO
Controls Basics – XAML
UserViewer | Engagement | Targeting | Press
Demo
Building ControlsBasics-XAML from Scratch
• File New Project > Grid App
• Add Reference to Microsoft.Kinect.Xaml.Cont...
Window
Frame
Page
Window
Frame
Page
Grid
KinectRegion
Kinect
UserViewer
Adding Hand Cursor Support
System Engagement Manual Engagement
• Hands over head
• In certain location of room
• Etc…
“Engagement” for hand cursor
Ki...
DEMO
PHiZ (Physical Interaction Zone) & KinectPointerPoint
PHIZ – Physical Interaction Zone
X / Y - For Cursor Rendering
KinectPointerPoint->Position->X
– Between 0.0 and 1.0
Kinect...
Press
Recommended Minimum Size:
• 208 x 208 (in 1080p resolution)
• Press attraction towards center
• Larger buttons will ...
Cursors
Demo
Controls Basics – XAML
Scroll
Zoom
Interacting with Controls (in Preview release)
• UI Frameworks
– XAML (Store)
– WPF
• DirectX (Store)
ControlsBasics-
XAML...
Interacting with Content
• Navigating a camera through a 3d space
• Rotating a 3d object
• “Scrubbing” a video
Kinect Resources
• General Info & Blog -> http://kinectforwindows.com
• Purchase Sensor -> http://aka.ms/k4wv2purchase
• v...
Q&A
Tutto il nateriale di questa sessione sul sito
http://www.dotnetcampus.it
@MatteoValoriani
©2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Office, Azure, System Center, Dynamics and other pro...
Próximos SlideShares
Carregando em…5
×

Programming with kinect v2

8.933 visualizações

Publicada em

Programming with kinect v2

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Programming with kinect v2

  1. 1. Matteo Valoriani
  2. 2. Nice to Meet You Matteo Valoriani PhD at Politecnico of Milano CEO of Fifth Element Speaker and Consultant Microsoft MVP for Kinect Intel Software Innovator: RealSense email: mvaloriani@gmail.com twitter: @MatteoValoriani linkedin: https://it.linkedin.com/in/matteovaloriani 2
  3. 3. Usage Scenarios TRAININGRETAIL HEALTHCARETHERAPY EDUCATION
  4. 4. You have to be a magician… … or at least a good illusionist
  5. 5. “Any sufficiently advanced technology is indistinguishable from magic” (Arthur C. Clarke)
  6. 6. Sensor Anatomy
  7. 7. Kinect 2 - Specs 3D DEPTH SENSOR RGB CAMERA MULTI-ARRAY MIC Hardware: Depth resolution: 512×424 RGB resolution: 1920×1080 (16:9) FrameRate: 60 FPS Latency: 60 ms
  8. 8. Kinect Adapter
  9. 9. Kinect 2 vs Kinect 1
  10. 10. Version 1 Version 2 Depth range 0.4m → 4.0m 0.4m → 4.5m Color stream 640×480 1920×1080 Depth stream 320×240 512×424 Infrared stream None 512×424 Type of Light Light coding ToF Audio stream 4-mic array 16 kHz 4-mic array 48 kHz USB 2.0 3.0 # Bodies Traked 2 (+4) 6 # Joints 20 25 Hand Traking External tools Yes Face Traking Yes Yes+Expressions FOV 57° H 43° V 70° H 60° V Tilt Motorized Manual
  11. 11. System / Software Requirements OS Windows 8, 8.1, Embedded 8, Embedded 8.1 (x64) CPU Intel Core i7 3.1GHz (or higher) RAM 4GB (or more) GPU DirectX 11 supported USB USB 3.0 Compiler Visual Studio 2012, 2013 (Supported Express) Language Native (C++), Managed (C#,VB.NET), WinRT (C#,HTML) Other Unity Pro (Add-in), Cinder, openFrameworks (wrapper)
  12. 12. DEMO Kinect Evolution
  13. 13. Accessing Kinect Data Sources
  14. 14. High-level architecture
  15. 15. Basic Flow of Programming Sensor Stream Frame Data Sensor Source Reader Frame Data Kinect for Windows SDK v1 Kinect for Windows SDK v2  Source independent to each Data (e.g. ColorSource, DepthSource, InfraredSource, BodyIndexSource, BodySource, …)  Doesn’t depend on each other Source (e.g. Doesn't need to Depth Source when retrieve Body Data)
  16. 16. • In “New Project” create a new Windows Store app • Enable Microphone and Webcam capabilities • Add a reference to Microsoft.Kinect • Use the Microsoft.Kinect namespace in your code Creating a new store app using Kinect
  17. 17. Represents a single physical sensor Always valid: when device is disconnected no more frame are generated. Use IsAviable Property to verify if the device is connected The KinectSensor class this KinectSensor this // Make the world a better place with Kinect this
  18. 18. Sources
  19. 19. Give access to frames – Events – Polling Multiple readers may be created on a single source Readers can be paused Readers InfraredFrameReader reader = sensor.InfraredFrameSource.OpenReader(); reader.FrameArrived += InfraredReaderFrameArrived; ...
  20. 20. Frame references void InfraredFrameReader InfraredFrameArrivedEventArgs using (InfraredFrame frame = args.FrameReference.AcquireFrame()) { if (frame != null) { // Get what you need from the frame } } } Sent in frame event args AcquireFrame gives access to the actual frame
  21. 21. • Gives access to the frame data – Make a local copy or access the underlying buffer directly • Contains metadata for the frame – e.g. Color: format, width, height, etc. • Important: Minimize how long you hold onto the frame – Not Disposing frames will cause you to not receive more frames Frames
  22. 22. • Allows the app to get a matched set of frames from multiple sources on a single event • Delivers frames at the lowest FPS of the selected sources MultiSourceFrameReader MultiSourceFrameReader MultiReader = Sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color | FrameSourceTypes.BodyIndex | FrameSourceTypes.Body); var frame = args.FrameReference.AcquireFrame(); if (frame != null) { using (colorFrame = frame.ColorFrameReference.AcquireFrame()) using (bodyFrame = frame.BodyFrameReference.AcquireFrame()) using (bodyIndexFrame = frame.BodyIndexFrameReference.AcquireFrame()){ // } }
  23. 23. System Kinect Driver Kinect SDK Application Kinect Driver Kinect SDK Application Kinect Service Kinect SDK Application Kinect SDK Application Kinect for Windows v1 Kinect for Windows v2
  24. 24. Connection Kinect for Windows v1 Kinect for Windows v2 PCPC
  25. 25. Example Multiple Connection PCPC PC Hub Server
  26. 26. DEMO Display frame in a Windows Store App
  27. 27. Data Source Details Physical image sensors, ColorFrameSource, InfraredFrameSource, DepthFrameSource, BodyIndexFrameSource, BodyFrameSource, AudioFrameSource
  28. 28. Kinect Data Sources
  29. 29. • 30 or 15 fps, based on lighting conditions ColorFrameSource
  30. 30. InfraredFrameSource
  31. 31. – 16-bit distance in millimeters from the sensor’s focal plane DepthFrameSource
  32. 32. – 0 to 5: Index of the corresponding body, as tracked by the body source – > 5: No tracked body at that pixel BodyIndexFrameSource 255 0 1
  33. 33. • Range is 0.5-4.5 meters • Frame data is a collection of Body objects each with 25 joints – Each joint has position in 3D space and an orientation • Up to six simultaneous bodies • 30fps • Hand State on 2 bodies • Lean BodyFrameSource
  34. 34. Skeleton vs Body
  35. 35. Coordinate System  ColorSpace (Coordinate System of the Color Image) – … Color  DepthSpace (Coordinate System of the Depth Data) – … Depth, Infrared, BodyIndex  CameraSpace (Coordinate System with the origin located the Depth Sensor) – … Body (Joint)
  36. 36. • Three coordinate systems • Coordinate mapper provides conversions between each system • Convert single or multiple points Coordinate mapping Name Applies to Dimensions Units Range Origin ColorSpacePoint Color 2 pixels 1920x1080 Top left corner DepthSpacePoint Depth, Infrared, Body index 2 pixels 512x424 Top left corner CameraSpacePoint Body 3 meters – Infrared/depth camera
  37. 37. DEMO Handle body frames and coordinate mapping
  38. 38. Kinect Studio Recording and Playback
  39. 39. Recordable Data Sources Infrared 13 MB/s Depth 13 MB/s BodyFrame BodyIndex Color 120 MB/s Audio 32 KB/s Legend Record/Play Record Only
  40. 40. DEMO Kinect Studio 2.0 Recording | Playback
  41. 41. Gesture Recognition Heuristic + Machine Learning Approaches
  42. 42. DEMO Programmatic detection of a gesture/pose
  43. 43. Gesture Builder • New tool, shipping with v2 SDK • Organize data using projects and solutions • Give meaning to data by tagging gestures • Build gestures using machine learning technology – Adaptive Boosting (AdaBoost) Trigger • Determines if player is performing gesture – Random Forest Regression (RFR) Progress • Determines the progress of the gesture performed by player • Analyze / test the results of gesture detection • Live preview of results
  44. 44. Visual Gesture Builder Your Application
  45. 45. DEMO Machine Learning Demo
  46. 46. Heuristic • Gesture is a coding problem • Quick to do simple gestures/poses (hand over head) • ML can also be useful to find good signals for Heuristic approach Machine Learning (ML) with G.B. • Gesture is a data problem • Signals which may not be easily human understandable (progress in a baseball swing) • Large investment for production • Danger of over-fitting, causes you to be too specific – eliminating recognition of generic cases Gesture Recognition
  47. 47. Hand Pointer Gestures
  48. 48. Hand pointer gestures Engagement Targeting Press Panning/Zoom
  49. 49. DEMO Controls Basics – XAML UserViewer | Engagement | Targeting | Press
  50. 50. Demo Building ControlsBasics-XAML from Scratch • File New Project > Grid App • Add Reference to Microsoft.Kinect.Xaml.Controls • Enable Microphone + Camera capabilities for app • Add KinectRegion as container for rootFrame in App.xaml.cs • Run it!
  51. 51. Window Frame Page Window Frame Page Grid KinectRegion Kinect UserViewer Adding Hand Cursor Support
  52. 52. System Engagement Manual Engagement • Hands over head • In certain location of room • Etc… “Engagement” for hand cursor KinectRegion.SetOnePersonSystemEngagement() KinectRegion.SetTwoPersonSystemEngagement() KinectRegion.SetOnePersonManualEngagement(…) KinectRegion.SetTwoPersonManualEngagement(…)
  53. 53. DEMO PHiZ (Physical Interaction Zone) & KinectPointerPoint
  54. 54. PHIZ – Physical Interaction Zone X / Y - For Cursor Rendering KinectPointerPoint->Position->X – Between 0.0 and 1.0 KinectPointerPoint->Position->Y – Between 0.0 and 1.0 X / Y – Allows offscreen info KinectPointerPoint->Properties->UnclampedPosition->X KinectPointerPoint->Properties->UnclampedPosition->Y Z KinectPointerPoint->Properties->HandReachExtent (raw) KinectPointerPoint->Properties->PressExtent (affected by user/UI) Correlating PointerPoint to NUI Data KinectPointerPoint->Properties->BodyTrackingId KinectPointerPoint->Properties->HandType KinectPointerPoint->Properties->BodyTimeCounter
  55. 55. Press Recommended Minimum Size: • 208 x 208 (in 1080p resolution) • Press attraction towards center • Larger buttons will just attract away from the edge Adapting to smaller visual sizes: • Make large-size hit testable • Set KinectRegion.KinectPressInset (Thickness) to non-visible part of button • Attraction region is smaller
  56. 56. Cursors
  57. 57. Demo Controls Basics – XAML Scroll Zoom
  58. 58. Interacting with Controls (in Preview release) • UI Frameworks – XAML (Store) – WPF • DirectX (Store) ControlsBasics- XAML Microsoft.Kinect. Xaml.Controls Microsoft.Kinect.Toolkit.Input WindowsPreview.Kinect ControlsBasics- WPF Microsoft.Kinect. Wpf.Controls Microsoft.Kinect ControlsBasics- DX KinectCoreWindow.PointerMoved, KinectPointerPoint, KinectGestureRecognizer InputPointerManager, PressableModel, ManipulatableModel Controls Layer – Cursor rendering, Hit Test, Route PointerPoint via InputPointerManager App (host KinectRegion, etc…)
  59. 59. Interacting with Content • Navigating a camera through a 3d space • Rotating a 3d object • “Scrubbing” a video
  60. 60. Kinect Resources • General Info & Blog -> http://kinectforwindows.com • Purchase Sensor -> http://aka.ms/k4wv2purchase • v2 Preview SDK -> http://aka.ms/k4wv2previewsdk • Developer Forums -> http://aka.ms/k4wv2forum
  61. 61. Q&A Tutto il nateriale di questa sessione sul sito http://www.dotnetcampus.it @MatteoValoriani
  62. 62. ©2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Office, Azure, System Center, Dynamics and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

×