How to use c# to develop application based on RealSense

1. Template designed by
Programming with RealSense
Matteo Valoriani
mvaloriani@gmail.com
@matteovaloriani

2. Matteo Valoriani
PhD at Politecnico of Milano
CEO of Fifth Element
Speaker and Consultant
Microsoft MVP
Intel Software Innovator
email: mvaloriani@gmail.com
twitter: @MatteoValoriani
linkedin: https://it.linkedin.com/in/matteovaloriani
Nice to Meet You
2

3. RealSense Introduction
Capture Hands Data
Detect HandsGestures
Understands Hardwsare Limits
RealSense Cameras
Agenda

4. The Intel® RealSense™ 3D Camera
4* Other names and brands may be claimed as the property of others

5. Intel Confidential - for use 5
Dell* Inspiron 23 7000
Available

6. Enthusiast-grade performance
- Intel 4th-gen Core i7/i5 processors
- NV GTX850M/860M
Silent cooling system
- Efficient heat removal with
minimum fan noise
Specially-designed Game Keys
- Specially marked WASD keys
- Chiclet Keyboard with Red backlighting
Majestic Sound
- SonicMaster Premium
- ROG Audio Wizard
- Headphone amp to enhance
gaming headset experience
Gaming-centric design
- Matte black finish with
red diamond cut
- Full HD IPS display with
178-degree wide-view
- Aluminum Design with
illuminated ROG logo
Intel® RealSense3D Camera
- Enhance experience with natural interaction.
- Immersive Collaboration/ Creation
- Capture faces/ objects in full 3D
Asus* G771
Intel Confidential - for use under NDA only

7. What Kinds of Applications can you develop?
7
Capture
and Share
Immersive
Collaboration/Creation
Interact
Naturally
Gaming
and Play
Learning and
Edutainment

8. demo

9. When we roll out the public beta SDK later in 2014, we’re
exploring a developer kit to help you get started
Kit would include:
• The Intel® RealSense™ SDK for Windows Beta
• Peripheral depth camera
• Full VGA depth resolution
• 1080p RGB camera
• 20cm – 120cm range
• USB 3.0 interface
• Requires 4th generation Intel® Core™ processor
(or later) & Windows* 8.1
The Developer Kit
9

10. Competitive
technologies focus
on a living-room
experience or a
sub-set of Intel
RealSense
technology
features
Designed for close-range interactions
120 cm
Intel®RealSense™
3D camera
56°(v) x 72° (v)
20 cm

11. Leap, RealSense, Kinect
2,5 cm 60 cm 2 m 4 m

12. 12
What OS and hardware works with RealSense technologies?
1Specific algorithms may have different range and accuracy.
**Roadmap Notice: All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice.
*Other names and brands may be claimed as the property of others.
Intel® RealSense™ SDK
for Windows*, Version 2014
supports Windows 8.x (64-bit),
Windows 10
Download the SDK
intel.com/realsense/sdk
Intel® RealSense™ Developer Kit 3D
camera
Get started developing quickly :
• Full VGA depth resolution
• 1080p RGB camera
• 0.2 – 1.2 meter range1
• USB 3.0 interface
• Requires Intel® Core™ processor
Reserve Yours Now

13. 13
System Requirements
Required Hardware • A dedicated USB 3 port in the client system (for peripheral dev kit camera), do not use a hub
• 4th generation (or later) Intel® Core™ i5 or i7 processor with Intel® Iris™ Graphics and Intel®
HD Graphics
• 8GB free hard disk space
• Intel® RealSense™ 3D Camera (front-facing, F200 – integrated or peripheral)
Required OS • Microsoft Windows* 8.1 64-bit Desktop Mode, 10
• Microsoft Windows 8.1 New UI (Metro) Mode (coming soon)**
• Support for Android* coming soon**
Supported
Programming
Languages
• C++
• C# (Microsoft .Net* 4.0 Framework is required)
• Java* (JDK 1.7.0_11 or later)
• JavaScript*
Supported IDE • Microsoft* Visual Studio 2010-2013 with service pack 1 or newer
• Unity* PRO 4. 1. 0 or later for Unity game development
Supported
Development Tools
• Microsoft* .NET 4.0 Framework for C# development
• Processing* 2.1.2 or higher for Processing framework development
• Java* JDK 1.7.0_11 or higher for Java development
Supported Browsers for
JavaScript
Development
• Microsoft Internet Explorer* 10.0.13
• Google Chrome* 33.0.1750.146
• Mozilla Firefox* 27.0.1

14. Understands 4 basic types of input
Categories of
Input
Capabilities Features
Hands • Hand and Finger
Tracking
• Gesture Recognition
• 22-point Hand and Finger Tracking
• 9 static and dynamic mid-air gestures
Face • Face Detection and
Tracking
• Multiple Face Detection and tracking
• 78-point Landmark Detection (facial features)
• Emotion Recognition (7 emotions, coming post-Beta)
• Pulse Estimation
• Face Recognition (Coming post-beta)
Speech • Speech Recognition • Command and Control
• Dictation
• Text to Speech
Environment • Segmentation
• 3D Scanning
• Augmented Reality
• Background Removal
• 3D Object / Face / Room Scanning (Coming post-beta)
• 2D/3D Object Tracking
• Scene Perception (coming post-beta)

15. Coordinate Systems
World coordinates Image coordinates

16. Mass Center: Image and world coordinates of the calculated hand image mass
center.
Extremity Points: Special tracking points such as the closest, left-most, top-most,
right-most, bottom-most and center points which form the boundaries of the
mask of the hand silhouette.
Body Side: Whether it is a left or right hand.
Palm Orientation: Estimation of where the hand is facing.
Tracked Joints: Positions and rotations of the user’s hand in world and image
coordinates.
Hands Data

17. Normalized Joints: Posture of user’s hand without changing dimensions of
the hand, i.e., the distances between each joint (bone-length) are always the
same.
Finger Data: Degree of foldedness and the radius of a particular fingertip.
Segmentation Image: A mask of the hand silhouette allowing to separate a
hand from the background image.
Hand Openness: (0-100 value) indicating if all fingers are completely folded
to all fingers fully spread.
Hands Data

18. /* Initialize a PXCMSenseManager instance */
manager = PXCMSenseManager.CreateInstance();
if (manager == null){
Debug.Writeline("SenseManager Initialization Failed");
}
Creating a Session for Hand Tracking
PXCMSenseManager: organizes a
pipeline by starting, stopping, and
pausing the operations of its various
modalities.

19. 1. Initialize a PXCMSenseManager instance
2. Enable hand tracking using EnableHand.
3. Retrieve an instance of hand module using QueryHand on
the instance of PXCMSenseManager.
4. Initialize the pipeline using Init
Initializing the Pipeline

20. /* Enable the hand tracking module*/
sts = manager.EnableHand();
if (sts != pxcmStatus.PXCM_STATUS_NO_ERROR)
Debug.WriteLine("PXCSenseManager.EnableHand: " + sts);
/* Retrieve an instance of hand to configure */
handAnalyzer = manager.QueryHand();
if (handAnalyzer == null)
Debug.WriteLine("PXCSenseManager.QueryHand");
/* Initialize the execution pipeline */
sts = manager.Init();
if (sts != pxcmStatus.PXCM_STATUS_NO_ERROR){
Debug.WriteLine("PXCMSenseManager.Init Failed");
}
handData = handAnalyzer.CreateOutput();
Initializing the Pipeline
EnableHand(): Enable the hand
tracking module.
QueryHand(): Retrieve an instance of
hand to configure
Init(): Initialize the execution
pipeline

21. config = handAnalyzer.CreateActiveConfiguration();
config.EnableAllGestures();
//config.EnableGesture("v_sign");
config.EnableAllAlerts();
config.ApplyChanges();
Enable Gestures
Enable All gestures or specific gestures
to recognize

22. private CancellationTokenSource streamCancellationTokenSource;
private CancellationToken streamCancellationToken;
Task streamTask = new Task(new Action(AcquireFrames),
streamCancellationToken);
streamTask.Start();
Acquire Data using Task

23. private void AcquireFrames() {
while (manager.AcquireFrame(true) >= pxcmStatus.PXCM_STATUS_NO_ERROR) {
this.outputData.Update();
PXCMHandData.IHand handData;
PXCMHandData.GestureData gestureData;
//AcquireGestures
// AcquireHands
manager.ReleaseFrame();
}
}
AcquireFrames
Acquire new data end refresh the
elaborated output
Release the frame or you can not
elaborate the next !!!

24. Gesture Tracking

25. //Gestures
for (int i = 0; i < outputData.QueryFiredGesturesNumber(); i++){
if (outputData.QueryFiredGestureData(i, out gestureData) ==
pxcmStatus.PXCM_STATUS_NO_ERROR){
outputData.QueryHandDataById(gestureData.handId, out handData);
var bodySide = handData.QueryBodySide();
//Do somethings with data
sb.Clear();
sb.AppendLine("Timestamp: " + gestureData.timeStamp);
sb.AppendLine("Gestrue identified: " + gestureData.name);
sb.ppendLine("Gestrue state: " + gestureData.state);
sb.AppendLine("Gestrue handId: " + gestureData.handId);
sb.AppendLine("Gestrue bodyside: " + bodySide.GetType());
sb.AppendLine();
}
}
Analyze Gestures

26. Hand Tracking Modes

27. Dictionary<PXCMHandData.JointType, PXCMHandData.JointData>[] hands = new Dictionary<PXCMHandData.JointType,
PXCMHandData.JointData>[4];
PXCMHandData.IHand[] totalHands = new PXCMHandData.IHand[4];
int numberOfHands = outputData.QueryNumberOfHands();
if (numberOfHands > 0) {
for (int i = 0; i < numberOfHands; i++) {
Dictionary<PXCMHandData.JointType, PXCMHandData.JointData> handDictionary =
new
Dictionary<PXCMHandData.JointType, PXCMHandData.JointData>();
PXCMHandData.JointData temp;
if (outputData.QueryHandData(PXCMHandData.AccessOrderType.ACCESS_ORDER_BY_ID, i, out handData) ==
pxcmStatus.PXCM_STATUS_NO_ERROR) {
// iterate through Joints
for (int j = 0; j < PXCMHandData.NUMBER_OF_JOINTS; j++) {
handData.QueryTrackedJoint((PXCMHandData.JointType)j, out temp);
handDictionary.Add((PXCMHandData.JointType)j, temp);
}
hands[i] = handDictionary;
totalHands[i] = handData;
// Do somethings with data
Analyze Hands Joints

28. demo

29. Capture Volumes
The user is performing a hand gesture outside of the capture volume.
The camera will not see this gesture

30. Understands Hardware Limits

31. Vertical Rages
60cm
58cm
120cm
56°
20cm
17cm
70cm 35cm
73cm
Effective Range
Gestures Range
Effective Range
3D Facial Traking Range
2D Facial Traking Range
115cm

32. Vertical Misalignment
56°

33. Horizontal Rages
60cm
87cm
120cm
72°
20cm
24cm
Effective Range
Gestures Range
Effective Range
3D Facial Traking Range
2D Facial Traking Range
170cm
70cm 35cm
108cm
50cm

34. RealSense Cameras

35. FRONT-FACING
Intel RealSense 3D Camera F200
Availability starting Q4’14
REAR-FACING
Intel RealSense 3D Camera R200
In-Market Target 2H’15
Intel® RealSense™ Cameras

36. Primary: 8 MP, 3264 x
2448 pixels, autofocus
Secondary: 2 MP
Intel RealSense Snapshot,
2 x 720p cameras
RealSense SnapShot

37. Display: 8.4" OLED infinity (2560 x
1600)
Dimensions: 215.9 x 124.2 x 6.1
mm
Weight: 305 g
OS: Android OS, v4.4.2 (KitKat),
CPU: Intel Atom Z3580, Quad-
core 2.3 GHz
Memory: 16 GB (+microSD), 2 GB
RAM
Dell Venue 8 7000

38. Target Platform:
Tablets, 2-in-1s, and more
The camera typically faces away from the user. It was designed to sense the environment around it in 3 dimensions.
Specifications
- Longer range** (up to 3-4 meters indoors, longer range outdoors)
- Depth/IR: 640x480 resolution at 60fps
- RGB: 1080p at 30fps
- USB 3.0 required
- Developer Kit Dimensions: 130mm x 20mm x 7mm
- Supports Microsoft Windows* 8.1 (64-bit mode), Android* support coming soon.
Targeted Usages
- 3D Capture: faces, people, and environments
- Depth-enhanced Augmented Reality
- Depth-enhanced photography and video
- Measurement
- Face detection and tracking
RealSense R200

39. Free Intel® RealSense™ SDK for Windows Beta in 2nd Half of 2014
Intel® RealSense™ Developer Kit being explored
Huge opportunity to reach customers with integrated 3D camera
Works with languages/frameworks/engines you already use
High-level APIs for NUI beginners. Low-level APIs for NUI experts
Getting Started
39All products, software, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice.
software.intel.com/RealSense

40. Q&A

41. Grazie a tutti per la partecipazione
Riceverete il link per il download a slide e demo via email nei
prossimi giorni
Per contattarmi
mvaloriani@gmail.com
Grazie