Enhancing the measurement of clinical outcomes using Microsoft Kinect choices (Philip Breedon, Bill Byrom, Luke Siena and Willie Muehlhausen) Interactive Technologies and Games (ITAG) Conference 2016 Health, Disability and EducationDates: Wednesday 26 October 2016 - Thursday 27 October 2016 Location: The Council House, NG1 2DT

1. Enhancing the Measurement of Clinical
Outcomes Using Microsoft Kinect
Philip Breedon and Francesco Luke Siena
Design for Health and Wellbeing
Research Group
Nottingham Trent University
Bill Byrom and Willie Muehlhausen
Product Innovation
ICON Clinical Research

2. 2
Presentation Overview
Overview of Clinical Trials
Motion Capture Platforms in Healthcare
Review of Kinect Applications for Outcomes Measurement
Example Measurement System

3. Overview of Clinical Trials
Bill Byrom, ICON

4. 29%
Do not advance
55%
Do not advance
40%
Do not advance

5. • Clinical trials rely upon robust and validated methodologies to measure health status and to
detect treatment-related changes in health status over time
• In some cases outcomes measures used rely on subjective ratings by the investigators at
each study research site.
– performance, balance, movement or mobility based on observation of the patient conducting a
specified movement or activity.
• Subjective ratings are not very sensitive to detecting small improvements
– Inter-rater reliability
• Objective measures preferred
– More sensitive
– Less prone to rater variability
– Able to measure detailed or subtle aspects of movement and mobility.
5
Objective measurement

6. Motion Capture Platforms in Healthcare
Luke Siena, Nottingham Trent University

7. 1. 3D Camera Systems and Sensors
2. Benefits of Motion Capture Platforms
3. Comparison Of Key Hardware & Utilities
4. Understanding The Progress Within The Motion Capture Platform Market
7
Motion Capture Platforms in Healthcare

8. 8
3D Camera Systems & Sensors

9. 9
Motion Capture Platforms in Healthcare
• 3D camera systems and sensors have great potential to continue
having a positive impact on the market in a variety of industries,
especially within health care and clinical platforms.
• Hardware specification improvements may still be required when
considering accurate tracking of fine or rapid movements, and
therefore the sampling rates associated with the capture of this
data may need to improve.
• The application of motion capture camera systems and technology
in clinical and home health care applications, especially within the
rehabilitation sector is constantly evolving.
• Platforms such as Neuroforma, JINTRONIX, Stroke Recovery with
Kinect and Face To Face have recently been developed, amongst
others.

10. • There is a growing body of applications utilising motion capture
technology that study or encourage movement in wellness, healthcare
and clinical research.
• The area of rehabilitation is constantly exploring ways of providing
engaging environments through regular exercise regimes to enable
patient feedback and correction.
• Ensuring exercises are being performed correctly for optimal benefit.
• Enabling remote assessment and adjustment of exercise regimes
between clinic visits ensures regular patient contact and reviews which
can be monitored.
10
Benefits Of Motion Capture Platforms In Healthcare

11. 11
Comparison of Microsoft Kinect 1.0 & 2.0 For
HealthCare Utility Applications
Function Kinect 1.0 Kinect 2.0
RGB Camera (Pixel) 1280 × 1024 or 640 × 480 1920 × 1080
Depth Camera (Pixel) 640 × 480 512 × 424
Sampling Rate (FPS / Hz) 30 FPS 30 FPS
SDK 1.8 Compatibility Yes No
SDK 2.0 Compatibility No Yes
Face Tracking Yes Yes
Expression Recognition No (Possible With Additional Algorithms) Yes
Bone Orientations No Yes
Body Joint Forces No Yes
Hand Tracking No (Possible With Additional Tools) Yes
Muscle Simulation No Yes
Heart Rate Measurement No Yes
* Price & Specifications as of May 2016

12. Capability / Function Intel RealSense SR300 Kinect 2.0
RGB Camera (Pixel) 1080p at 30 FPS, 720p at 60 FPS 1920 × 1080 at 30 FPS
Depth Camera (Pixel) Up to 640 x 480 at 60 FPS (Fast
VGA, VGA), HVGA at 110 FPS
512 × 424 at 30 FPS
Skeletal Joint Definition Points 22 26
Face Tracking & Recognition Yes Yes
Expression Recognition Yes Yes
Gesture Recognition Yes Yes
Hand Tracking Yes Yes
Audio Stream Dual Array Microphones 4-Mic-Array
Connectivity (USB) 3.0 3.0
Approx. price (USD)* 130 190
12
Comparison of Intel® RealSense TM SR300 and
Microsoft Kinect 2.0 For HealthCare Systems
* Price & Specifications as of May 2016

13. 13
Why Champion The Intel® RealSense™ ?
• The Intel camera offers greater resolution and sampling rate
in comparison to Kinect 2.0, which may offer advantages
when tracking fine or fast movements.
• One of the novelties of the Intel RealSense 3D camera range
is its versatility for integration into a variety of platforms,
yet at the same time it remains affordable.
• Intel have developed a number of Intel RealSense camera
systems which can be integrated into a variety of platforms
whether this be Desktop PC’s, All-In-One PC’s, 2 In 1 PC’s,
external camera systems, smartphones and tablet kits and
even a robotics.

14. Review of Kinect Applications for Outcomes
Measurement
Bill Byrom, ICON

15. 1. Gait and balance
2. Upper extremity movement
3. Chest wall motion analysis
4. Facial analysis
15
Four main areas of measurement

16. • Various performance tests
proposed
– Short walking tests
– Treadmill walking tests
– Balance tests
• Spline interpolation to estimate
100 Hz sampling frequency
• Custom error correction
technique to improve data
artefact identification
16
Gait and balance
Pfister A. et al. (2014)

17. 17
Gait and balance
Ref Performance Measure Indication Comparator n Validation evidence
[1] Treadmill
walking tests
Hip/knee flexion/extension
Stride timing
Healthy
volunteers
(HV)
VICON motion
capture
28 Kinect underestimated flexion,
overestimated extension. Stride timing
often well correlated.
[2] Short walk Velocity, stride length,
hip/knee ROM
MS + HV PRO (MSWS)
ClinRO (EDSS)
20 Able to distinguish MS form controls
Reliability good except step width and hip
ROM
[3] 6 m walk Step length, foot swing
velocity, mean and peak
gait velocity, asymmetry
Stroke 10mWT, TUG,
Step test
30 Kinect parameters reliable: ICCs > 0.8
Feasible to instrument gait analysis
[4] Standing,
stepping, walk
on spot, UPDSS
Various PD + HV VICON motion
capture
19 Good for gross movements
Poor for fine movement
Good correlation with VICON (r > 0.8)
[5] Short max
speed walk
Speed; L/R, Up/Down and
3D deviation; speed
deviation
MS + HV 25 foot walk
test
44 Able to differentiate MS and controls
Good concordance with 25-foot walk test

18. 18
Upper extremity movement
Lin J-L. et al. (2014)
• Range of motion and
reaching volume
estimated from various
performance tests
– Standard range of motion
movements
– Movement task

19. Upper extremity movement
Ref Performance Measure Indication Comparator n Validation evidence
[6] Shoulder
movement
Shoulder flexion,
abduction, rotation
Adhesive
capsulitis + HV
Goniometer 27 ICCs: 0.864-0.942
[7] FMA / ARAT Shoulder/elbow/wrist
flexion, abduction, rotation
Stroke Impulse motion cap.
+ clinician ass.
9 MC: R2 = 0.64, p < 0.001
Clin. Ass: R2 = 0.86, p < 0.001
[8] Arm
movement
Shoulder flexion,
abduction, rotation,
extension
Healthy
volunteers (HV)
Goniometer 10 r = 0.86 to 0.99
[9] Arm
movement
3D workable reaching
space
HV Impulse motion
capture
10 R2 = 0.79
[10] Pediatric
Functional
Assessment
Index finger and thumb,
wrist, elbow, shoulder
ROM
HV Clinician assessment 12 “Technically sound approach”
[11] Movement
task
Involuntary movements /
dyskinesia
HV Clinician assessment 4 Cohen’s kappa 0.85, p < 0.05
[12] Fugl-Meyer,
WMFT, ARAT
Shoulder, elbow and wrist
position
HV Optitrack motion
capture
10 “Kinect is sufficiently accurate
and responsive”
[13] Arm/hand
movements
Machine learning
identification
MS Differentiate MS
from HV
1041 “Automated MS assessment
possible”

20. • Four Kinect cameras used to generate a 3D image
of the chest
• Performance test:
– Quiet breathing for 20 s, followed by a relaxed vital
capacity (VC) manoeuver (maximum inspiration and
expiration) and followed by 20 s of quiet breathing.
• Tidal volume, Respiratory Rate, and minute
ventilation compared to spirometry
– Good concordance for
• Cystic Fibrosis patients: r>0.8656
• Healthy volunteers r> 0.922
20
Chest wall analysis

21. 21
Facial analysis
Face to Face solution
• Rehabilitation system for facial paralysis in
stroke patients.
• Recognizes facial expressions
• Facial exercise performance is assessed by
the system and scored according to how
well the user can undertake each of the
defined set of expressions.
• Potential to apply to providing longitudinal
objective measures of change to assess
treatment effects.

22. 22
Summary findings
• May be less able to measure fine or rapid
movements
– Sampling rate of camera
– Resolution and depth of vision
• Joint detection accuracy with conventional SDK may
limit some applications
• May provide a low cost alternative to specialist labs
or subjective endpoints in large scale trials

23. Example Measurement System
Bill Byrom, ICON

24. • Objectives
• Understand how to develop
applications using the Kinect
Windows SDK
• Demonstrate the concept of health
outcomes measurement using Kinect
• Input into definition of future
requirements
24
Proof of concept: shoulder ROM

25. 25
Proof of concept: shoulder ROM

26. Thank you
Any questions?
@billbyrom
https://uk.linkedin.com/in/bill-byrom-7136975

27. References
[1] Pfister A. et al. (2014). Comparative abilities of Microsoft Kinect and Vicon
3D motion capture for gait analysis . J Med Eng Technol; 38: 274-280.
[8] Lin J-L. et al. (2014). Assessment of range of shoulder motion using Kinect.
Gerontechnology ; 13:249
[2] Gholami F. et al. (2015). https://arxiv.org/pdf/1508.02405v1.pdf [9] Kurillo G. et al. (2013). Evaluation of upper extremity reachable workspace
using Kinect camera. Technology and Health Care ; 21:641–656
[3] Clarke R.A. et al. (2015). Instrumenting gait assessment using the Kinect in
people living with stroke: reliability and association with balance tests. J
NeuroEngineering and Rehab; 12:15-23.
[10] Rammer J.R. et al. (2014). Evaluation of Upper Extremity Movement
Characteristics during Standardized Pediatric Functional Assessment with a
Kinect-Based Markerless Motion Analysis System. Conf Proc IEEE Eng Med Biol
Soc. 2014: 2525–2528.
[4] Galna B. et al. (2014). Accuracy of the Microsoft Kinect sensor for
measuring movement in people with Parkinson’s disease. Gait & Posture;
39: 1062–1068
[11] Li S.et al. (2015). Quantitative Assessment of ADL: A Pilot Study of Upper
Extremity Reaching Tasks. J Sensors; Article ID 236474.
[5] Behrens J. et al. (2014). Using perceptive computing in multiple sclerosis -
the Short Maximum Speed Walk test. J NeuroEngineering and Rehab;
11:89-98.
[12] Webster D. et al. (2014). Experimental Evaluation of Microsoft Kinect’s
Accuracy and Capture Rate for Stroke Rehabilitation Applications. IEEE Haptics
Symposium 2014.
[6] Lee S.H. et al. (2015). Measurement of Shoulder Range of Motion
in Patients with Adhesive Capsulitis Using a Kinect. PLOS ONE10(6):
e0129398.
[13] Kontschieder P. et al. (). Quantifying Progression of Multiple Sclerosis via
Classification of Depth Videos. , in (Golland, P. et al. eds. Medical Image
Computing and Computer-Assisted Intervention – MICCAI 2014, Volume 8674
of the series Lecture Notes in Computer Science); pp 429-437.
[7] Olesh E.V. (2014). Automated Assessment of Upper Extremity Movement
Impairment due to Stroke. PLoS ONE 9(8): e104487
[14] Harte J.M. et al. (2015). Chest wall motion analysis in healthy volunteers and
adults with cystic fibrosis using a novel Kinect-based motion tracking system.
Med. Biol. Eng. Comput.; DOI 10.1007/s11517-015-1433-1.
[15] Breedon P. et al. (2014). First for Stroke: using the Microsoft' Kinect' as a facial
paralysis stroke rehabilitation tool. Int J Integrated Care (IJIC), 14.