Multimodal Analysis of Speech, Handwriting and Gait for the Assessment of Patients with Parkinson’s Disease

1. Multimodal Analysis of Speech, Handwriting
and Gait for the Assessment of Patients with
Parkinson’s Disease
Student: Juan Camilo V´asquez Correa
Advisors:
Prof. Juan Rafael Orozco Arroyave1, Prof. Elmar N¨oth2
1GITA research group, University of Antioquia UdeA.
2Pattern recognition Lab. Friedrich Alexander Universit¨at. Erlangen-N¨urnberg.
jcamilo.vasquez@udea.edu.co
1 / 33

2. Outline
Introduction
Methods
Speech Analysis
Gait Analysis
Handwriting Analysis
Multimodal Analysis
Results
Multi-view learning using GCCA
Convolutional neural networks for speech analysis
Ongoing work
Conclusion
2 / 33

3. Introduction: Parkinson’s Disease
Second most prevalent
neurological disorder
worldwide.
Patients develop sev-
eral motor and non-
motor impairments. (O.
Hornykiewicz 1998).
3 / 33

4. Introduction: Parkinson’s Disease II
Motor impairments
Bradykinesia
Rigidity
Resting tremor
Micrographia
Dysartrhia
Non–Motor impairments
Depression
Sleep disorders
Cognitive impairments
Sensory system deficits
Evaluated by neurologist experts according to the
MDS-UPDRS-III scale (C. G. Goetz et al. 2008).
4 / 33

5. Introduction: Parkinson’s Disease II
Motor impairments
Bradykinesia
Rigidity
Resting tremor
Micrographia
Dysartrhia
Non–Motor impairments
Depression
Sleep disorders
Cognitive impairments
Sensory system
deficits
Evaluated by psychologist experts.
5 / 33

6. Introduction: Research Problem
Motor evaluation is expensive and time–consuming.
Neurologists evaluate perceptually the motor deficits of the
patients.
The assessment of the motor capabilities provides suitable
information to update the treatment and the medication.
6 / 33

7. Introduction: Research Problem
Motor evaluation is expensive and time–consuming.
Neurologists evaluate perceptually the motor deficits of the
patients.
The assessment of the motor capabilities provides suitable
information to update the treatment and the medication.
7 / 33

8. Introduction: Research Problem
Motor evaluation is expensive and time–consuming.
Neurologists evaluate perceptually the motor deficits of the
patients.
The assessment of the motor capabilities provides suitable
information to update the treatment and the medication.
8 / 33

9. Introduction: Justification
if the motor evaluation is performed with bio–signals such
as speech, handwriting and gait, the treatment could be fol-
lowed in a more objective way
9 / 33

10. Introduction: Proposal
These three bio-signals constitute a reliable source of
information to describe several symptoms of PD patients.
The combination of such sources of information allows to
perform an accurate quantification of the neurological state
of the patients.
The multimodal analysis that includes information
from different kind of sensors for the analysis of PD
has not been enough studied (Q. W. Oung, et al. 2015)
10 / 33

11. Introduction: Proposal
These three bio-signals constitute a reliable source of
information to describe several symptoms of PD patients.
The combination of such sources of information allows to
perform an accurate quantification of the neurological state
of the patients.
The multimodal analysis that includes information
from different kind of sensors for the analysis of PD
has not been enough studied (Q. W. Oung, et al. 2015)
11 / 33

12. Introduction: Proposal
These three bio-signals constitute a reliable source of
information to describe several symptoms of PD patients.
The combination of such sources of information allows to
perform an accurate quantification of the neurological state
of the patients.
The multimodal analysis that includes information
from different kind of sensors for the analysis of PD
has not been enough studied (Q. W. Oung, et al. 2015)
12 / 33

13. Methods: Speech analysis
Speech impairments can be assessed using four dimensions
(J. R. Orozco-Arroyave 2016)
Phonation
Articulation
Prosody Intelligibility
pataka pataka
13 / 33

14. Methods: Speech analysis
https://github.com/jcvasquezc/NeuroSpeech1
1J. R. Orozco-Arroyave, J. C. V´asquez-Correa, et. al. “NeuroSpeech: an
open-source software for Parkinson’s speech analysis”. In: Digital Signal Precessing
and SoftwareX, (Under review) (2017).
14 / 33

15. Methods: Gait analysis
Gait acquisition
15 / 33

16. Methods: Gait analysis
Heel
strike
Toe
off
Heel
strike
Stance time Swing time
Stride time & Stride length
Toe off / Heel strike
angles
( J. Klucken et al. 2013; P. A. P´erez-Toro 2016 )
16 / 33

17. Methods: Gait analysis
Time–frequency analysis
0 5 10 15 20
Time (s)
0
10
20
30
40
50
Frequency(Hz)
Healthy Control
0 5 10 15 20 25
Time (s)
Patient
Short time Fourier Transform
Wavelet transform
Modulation spectra
Wigner Ville distribution
17 / 33

18. Methods: Handwriting analysis
Handwriting acquisition
18 / 33

19. Methods: Handwriting analysis
Speed of the stroke
Acceleration
In–air movement
Pressure of the pen
Azimuth
(P. Drot´ar et al. 2016)
Static handwriting analysis
(Z. Naiquian et. al 2017 )
19 / 33

20. Methods: Multimodal analysis
GCCA
(J. C. V´asquez-Correa, et
al. 2017)
Early fusion
Weak learners
Multimodal convolutional
neural networks
Deep autoencoders.
20 / 33

21. Methods: Pattern analysis
Classical machine learning
Support vector
machines.
Support vector
regressors.
minimize
w,b,ξ
1
2 ||w||2
+ C N
i=1 ξi
subject to yi · (xT
i w + b) ≥ 1 − ξi,
ξi ≥ 0
21 / 33

22. Methods: Pattern analysis
Deep learning
Convolutional neural networks.
Recurrent neural networks and LSTM
Variational deep autoencoders
Convolutional layer 1 Convolutional layer 2Pooling layer Pooling layer 2 RNN-LSTM
Input
y
J hidden filters K hidden filters
22 / 33

23. Results: Multi-view learning using GCCA2
To obtain a new feature representation when multimodal
data is not available.
To predict missing information.
Machine learning methods are trained with the new feature
representation.
arg min
Uj
J
j=1
G − XjUj
2
F
s.t. GT
G = I
2J. C. V´asquez-Correa, et al. “Multi-view representation learning via GCCA for
multimodal analysis of Parkinson’s disease”. In: 42nd International Conference on
Acoustic, Speech, and Signal Processing (ICASSP). 2017.
23 / 33

24. Results: Multi-view learning using GCCA3
Baseline GCCA
Classification PD vs. HC 77% 78%
Neurological state prediction 0.36 0.40
Speech quality prediction 0.67 0.71
Table: Results GCCA
The proposed approach is suitable to map the features
from other modalities that are not always available.
3J. C. V´asquez-Correa, et al. “Multi-view representation learning via GCCA for
multimodal analysis of Parkinson’s disease”. In: 42nd International Conference on
Acoustic, Speech, and Signal Processing (ICASSP). 2017.
24 / 33

25. Results: CNNs for speech analysis4
Convolution layer I Convolution layer IIMax-pool. layer 1 Max-pool layer 2 Fully conected MLP
Input layer
PD
vs.
HC
Feature maps 1
Feature maps 2
convolutional neural networks (CNN) learns high–level
representations from the low–level raw data.
CNN is formed with an array of convolutional filters and
subsampling layers.
HL(i, j, d) = (I ∗ Kd )(i, j) = m n I(i + m, j + n)Kd (m, n)
4J. C. V´asquez-Correa, J. R. Orozco-Arroyave, and E. N¨oth. “Convolutional Neural
Network to Model Articulation Impairments in Patients with Parkinson’s Disease”. In:
18th International Conference of the Speech and Communication Association
(INTERSPEECH). 2017.
25 / 33

26. Results: CNNs for speech analysis5
Voiced-Unvoiced transitions are modeled with CNNs and
TFRs.
The STFT and the continuous wavelet transform (CWT)
are considered.
Speech of PD patients in three languages: Spanish,
German and Czech.
5J. C. V´asquez-Correa, J. R. Orozco-Arroyave, and E. N¨oth. “Convolutional Neural
Network to Model Articulation Impairments in Patients with Parkinson’s Disease”. In:
18th International Conference of the Speech and Communication Association
(INTERSPEECH). 2017.
26 / 33

27. Results: CNNs for speech analysis6
Language Accuracy
Spanish 85.9%
German 75.0%
Czech 89.4%
Table: Classification of PD vs. HC
using CNNs
50 100 150
Time (ms)
0
1000
2000
3000
4000
Frequency(Hz)
50 100 150
Time (ms)
Low Energy High Energy
Figure: Output of the CNN after
the last max–pool layer: PD
patient (left) and a HC speaker
(right)
6J. C. V´asquez-Correa, J. R. Orozco-Arroyave, and E. N¨oth. “Convolutional Neural
Network to Model Articulation Impairments in Patients with Parkinson’s Disease”. In:
18th International Conference of the Speech and Communication Association
(INTERSPEECH). 2017.
27 / 33

28. Ongoing work
Combination of convolutional and recurrent neural
networks for multimodal analysis of PD.
Convolutional layer 1 Convolutional layer 2Pooling layer Pooling layer 2 RNN-LSTM
Speech
y
J hidden filters K hidden filters
Gait
Handwriting
28 / 33

29. Ongoing work
Combination of convolutional and recurrent neural
networks for multimodal analysis of PD.
𝒙(𝑡)
𝒉(𝑡−1)
+
𝑼 𝑓 𝑾 𝑓
𝑓: forget gate
+
𝑼 𝑾
𝒙(𝑡)
𝒉(𝑡−1)
𝒙(𝑡)
𝒉(𝑡−1)
+
𝑼 𝑔 𝑾 𝑔
𝑔: input gate
∗+∗
𝒔(𝑡)
𝒔(𝑡−1)
𝑠: hidden state
𝑡𝑎𝑛ℎ
𝒉(𝑡)
𝒙(𝑡)
𝒉(𝑡−1)
+
𝑼 𝑞 𝑾 𝑞
𝑞: output gate
LSTM cell
29 / 33

30. Conclusion
Speech, handwriting and gait signals constitute a reliable
source of information to describe several symptoms of PD
patients.
The combination of such sources of information allows to
perform an accurate quantification of the neurological state
of the patients.
Several features and pattern analysis approaches could be
considered to improve the classification of the disease, and
the monitoring of the neurological state of the patients.
30 / 33

31. References I
J. Klucken et al. “Unbiased and mobile gait analysis detects motor impairment in Parkin-
son’s disease”. In: PloS one 8.2 (2013), e56956.
C. G. Goetz et al. “Movement Disorder Society-sponsored revision of the Unified Parkin-
son’s Disease Rating Scale (MDS-UPDRS): Scale presentation and clinimetric test-
ing results”. In: Movement disorders 23.15 (2008), pp. 2129–2170.
J. C. V´asquez-Correa, et al. “Multi-view representation learning via GCCA for multimodal
analysis of Parkinson’s disease”. In: 42nd International Conference on Acoustic,
Speech, and Signal Processing (ICASSP). 2017.
J. C. V´asquez-Correa, J. R. Orozco-Arroyave, and E. N¨oth. “Convolutional Neural Net-
work to Model Articulation Impairments in Patients with Parkinson’s Disease”. In:
18th International Conference of the Speech and Communication Association (IN-
TERSPEECH). 2017.
J. R. Orozco-Arroyave. Analysis of Speech of People with Parkinson’s Disease. Ger-
many: Logos Verlag Berlin, 2016.
J. R. Orozco-Arroyave, J. C. V´asquez-Correa, et. al. “NeuroSpeech: an open-source
software for Parkinson’s speech analysis”. In: Digital Signal Precessing and Soft-
wareX, (Under review) (2017).
O. Hornykiewicz. “Biochemical aspects of Parkinson’s disease”. In: Neurology 51.2
Suppl 2 (1998), S2–S9.
P. A. P´erez-Toro J. C. V´asquez-Correa, et. al. “An´alisis motriz en las extremidades in-
feriores para el monitoreo del estado neurol´ogico de pacientes con enfermedad de
Parkinson”. In: XXI Symposium on Image, Signal Processing and Artificial Vision
(STSIVA). 2016.

32. References II
P. Drot´ar et al. “Evaluation of handwriting kinematics and pressure for differential diag-
nosis of Parkinson’s disease”. In: Artificial intelligence in Medicine 67 (2016), pp. 39–
46.
Q. W. Oung, et al. “Technologies for assessment of motor disorders in Parkinson’s dis-
ease: a review”. In: Sensors 15.9 (2015), pp. 21710–21745.
Z. Naiquian et. al. “Toward Monitoring Parkinson’s through Analysis of Static Handwriting
Samples: A Quantitative Analytical Framework”. In: IEEE journal of biomedical and
health informatics 21.2 (2017), pp. 488–495.

33. Multimodal Analysis of Speech, Handwriting
and Gait for the Assessment of Patients with
Parkinson’s Disease
Student: Juan Camilo V´asquez Correa
Advisors:
Prof. Juan Rafael Orozco Arroyave1, Prof. Elmar N¨oth2
1GITA research group, University of Antioquia UdeA.
2Pattern recognition Lab. Friedrich Alexander Universit¨at. Erlangen-N¨urnberg.
jcamilo.vasquez@udea.edu.co
33 / 33