2. Nice to Meet You
Matteo Valoriani
PhD at Politecnico of Milano
CEO of Fifth Element
Speaker and Consultant
Microsoft MVP for Kinect
Intel Software Innovator: RealSense
email: mvaloriani@gmail.com
twitter: @MatteoValoriani
linkedin: https://it.linkedin.com/in/matteovaloriani
2
15. Basic Flow of Programming
Sensor Stream Frame Data
Sensor Source Reader Frame Data
Kinect for Windows SDK v1
Kinect for Windows SDK v2
Source independent to each Data
(e.g. ColorSource, DepthSource, InfraredSource, BodyIndexSource, BodySource, …)
Doesn’t depend on each other Source
(e.g. Doesn't need to Depth Source when retrieve Body Data)
16. • In “New Project” create a new Windows Store app
• Enable Microphone and Webcam capabilities
• Add a reference to Microsoft.Kinect
• Use the Microsoft.Kinect namespace in your code
Creating a new store app using Kinect
17. Represents a single physical sensor
Always valid: when device is disconnected no more frame are
generated.
Use IsAviable Property to verify if the device is connected
The KinectSensor class
this KinectSensor
this
// Make the world a better place with Kinect
this
19. Give access to frames
– Events
– Polling
Multiple readers may be
created on a single source
Readers can be paused
Readers
InfraredFrameReader reader =
sensor.InfraredFrameSource.OpenReader();
reader.FrameArrived +=
InfraredReaderFrameArrived;
...
21. • Gives access to the frame data
– Make a local copy or access the underlying buffer directly
• Contains metadata for the frame
– e.g. Color: format, width, height, etc.
• Important: Minimize how long you hold onto the frame
– Not Disposing frames will cause you to not receive more frames
Frames
22. • Allows the app to get a matched set of frames from multiple
sources on a single event
• Delivers frames at the lowest FPS of the selected sources
MultiSourceFrameReader
MultiSourceFrameReader MultiReader =
Sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color |
FrameSourceTypes.BodyIndex |
FrameSourceTypes.Body);
var frame = args.FrameReference.AcquireFrame();
if (frame != null) {
using (colorFrame = frame.ColorFrameReference.AcquireFrame())
using (bodyFrame = frame.BodyFrameReference.AcquireFrame())
using (bodyIndexFrame = frame.BodyIndexFrameReference.AcquireFrame()){
//
}
}
31. – 16-bit distance in millimeters from the
sensor’s focal plane
DepthFrameSource
32. – 0 to 5: Index of the corresponding body,
as tracked by the body source
– > 5: No tracked body at that pixel
BodyIndexFrameSource
255
0 1
33. • Range is 0.5-4.5 meters
• Frame data is a collection of Body objects each
with 25 joints
– Each joint has position in 3D space and an
orientation
• Up to six simultaneous bodies
• 30fps
• Hand State on 2 bodies
• Lean
BodyFrameSource
35. Coordinate System
ColorSpace (Coordinate System of the Color
Image)
– … Color
DepthSpace (Coordinate System of the
Depth Data)
– … Depth, Infrared, BodyIndex
CameraSpace (Coordinate System with the
origin located the Depth Sensor)
– … Body (Joint)
36. • Three coordinate systems
• Coordinate mapper provides conversions between each system
• Convert single or multiple points
Coordinate mapping
Name Applies to Dimensions Units Range Origin
ColorSpacePoint Color 2 pixels 1920x1080 Top left corner
DepthSpacePoint Depth,
Infrared,
Body index
2 pixels 512x424 Top left corner
CameraSpacePoint Body 3 meters – Infrared/depth
camera
43. Gesture Builder
• New tool, shipping with v2 SDK
• Organize data using projects and solutions
• Give meaning to data by tagging gestures
• Build gestures using machine learning technology
– Adaptive Boosting (AdaBoost) Trigger
• Determines if player is performing gesture
– Random Forest Regression (RFR) Progress
• Determines the progress of the gesture performed by player
• Analyze / test the results of gesture detection
• Live preview of results
46. Heuristic
• Gesture is a coding problem
• Quick to do simple
gestures/poses (hand over head)
• ML can also be useful to find
good signals for Heuristic
approach
Machine Learning (ML) with G.B.
• Gesture is a data problem
• Signals which may not be easily
human understandable (progress
in a baseball swing)
• Large investment for production
• Danger of over-fitting, causes you
to be too specific – eliminating
recognition of generic cases
Gesture Recognition
50. Demo
Building ControlsBasics-XAML from Scratch
• File New Project > Grid App
• Add Reference to Microsoft.Kinect.Xaml.Controls
• Enable Microphone + Camera capabilities for app
• Add KinectRegion as container for rootFrame in
App.xaml.cs
• Run it!
52. System Engagement Manual Engagement
• Hands over head
• In certain location of room
• Etc…
“Engagement” for hand cursor
KinectRegion.SetOnePersonSystemEngagement()
KinectRegion.SetTwoPersonSystemEngagement()
KinectRegion.SetOnePersonManualEngagement(…)
KinectRegion.SetTwoPersonManualEngagement(…)
54. PHIZ – Physical Interaction Zone
X / Y - For Cursor Rendering
KinectPointerPoint->Position->X
– Between 0.0 and 1.0
KinectPointerPoint->Position->Y
– Between 0.0 and 1.0
X / Y – Allows offscreen info
KinectPointerPoint->Properties->UnclampedPosition->X
KinectPointerPoint->Properties->UnclampedPosition->Y
Z
KinectPointerPoint->Properties->HandReachExtent (raw)
KinectPointerPoint->Properties->PressExtent (affected by user/UI)
Correlating PointerPoint to NUI Data
KinectPointerPoint->Properties->BodyTrackingId
KinectPointerPoint->Properties->HandType
KinectPointerPoint->Properties->BodyTimeCounter
55. Press
Recommended Minimum Size:
• 208 x 208 (in 1080p resolution)
• Press attraction towards center
• Larger buttons will just
attract away from the edge
Adapting to smaller visual sizes:
• Make large-size hit testable
• Set KinectRegion.KinectPressInset
(Thickness) to non-visible part of button
• Attraction region is smaller