1. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Clustering Human Behaviors with Dynamic Time Warping
and Hidden Markov Models for a Video Surveillance System
Kan Ouivirach and Matthew N. Dailey
Computer Science and Information Management
Asian Institute of Technology
ECTI-CON
May 19-21, 2010
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
1 / 29

2. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Outline
1 Introduction
2 Human Behavior Pattern Clustering
3 Experimental Results
4 Conclusion
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
2 / 29

3. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction
Human behavior understanding is important for intelligent systems.
Difficult due to the wide range of activities possible in any given
context
Figure: Reprinted from http://www.sourcesecurity.com/
A classic work by Yamato et al. who model tennis actions using
hidden Markov models (HMMs)
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
3 / 29

4. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction
Human behavior understanding is important for intelligent systems.
Difficult due to the wide range of activities possible in any given
context
Figure: Reprinted from http://www.sourcesecurity.com/
A classic work by Yamato et al. who model tennis actions using
hidden Markov models (HMMs)
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
3 / 29

5. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction
Human behavior understanding is important for intelligent systems.
Difficult due to the wide range of activities possible in any given
context
Figure: Reprinted from http://www.sourcesecurity.com/
A classic work by Yamato et al. who model tennis actions using
hidden Markov models (HMMs)
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
3 / 29

6. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
To help security personnel work reliably and efficiently,
filter out typical events;
automatically present anomalous events to human operator.
Figure: Reprinted from http://sikafutu.com/
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
4 / 29

7. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
To help security personnel work reliably and efficiently,
filter out typical events;
automatically present anomalous events to human operator.
Figure: Reprinted from http://sikafutu.com/
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
4 / 29

8. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
To help security personnel work reliably and efficiently,
filter out typical events;
automatically present anomalous events to human operator.
Figure: Reprinted from http://sikafutu.com/
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
4 / 29

9. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
One limitation of most of the work
The number of “normal” behavior patterns need to be known
beforehand.
Nair and Clark (2002) use HMMs to model a common, predefined
activity in a scene.
Unsupervised analysis and clustering of behaviors for a variety of
purposes has started to draw attentions.
Li et al. (2006) cluster human gestures by constructing an affinity
matrix using dynamic time warping (DTW), and apply the
normalized-cut approach to cluster.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
5 / 29

10. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
One limitation of most of the work
The number of “normal” behavior patterns need to be known
beforehand.
Nair and Clark (2002) use HMMs to model a common, predefined
activity in a scene.
Unsupervised analysis and clustering of behaviors for a variety of
purposes has started to draw attentions.
Li et al. (2006) cluster human gestures by constructing an affinity
matrix using dynamic time warping (DTW), and apply the
normalized-cut approach to cluster.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
5 / 29

11. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
One limitation of most of the work
The number of “normal” behavior patterns need to be known
beforehand.
Nair and Clark (2002) use HMMs to model a common, predefined
activity in a scene.
Unsupervised analysis and clustering of behaviors for a variety of
purposes has started to draw attentions.
Li et al. (2006) cluster human gestures by constructing an affinity
matrix using dynamic time warping (DTW), and apply the
normalized-cut approach to cluster.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
5 / 29

12. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
One limitation of most of the work
The number of “normal” behavior patterns need to be known
beforehand.
Nair and Clark (2002) use HMMs to model a common, predefined
activity in a scene.
Unsupervised analysis and clustering of behaviors for a variety of
purposes has started to draw attentions.
Li et al. (2006) cluster human gestures by constructing an affinity
matrix using dynamic time warping (DTW), and apply the
normalized-cut approach to cluster.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
5 / 29

13. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
One limitation of most of the work
The number of “normal” behavior patterns need to be known
beforehand.
Nair and Clark (2002) use HMMs to model a common, predefined
activity in a scene.
Unsupervised analysis and clustering of behaviors for a variety of
purposes has started to draw attentions.
Li et al. (2006) cluster human gestures by constructing an affinity
matrix using dynamic time warping (DTW), and apply the
normalized-cut approach to cluster.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
5 / 29

14. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
Some recent related works using HMMs to cluster behavior patterns
Swears et al. (2008) propose hierarchical HMM-based clustering to
find motion trajectories and velocities in a highway interchange
scene.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
6 / 29

15. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
Some recent related works using HMMs to cluster behavior patterns
Swears et al. (2008) propose hierarchical HMM-based clustering to
find motion trajectories and velocities in a highway interchange
scene.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
6 / 29

16. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
New method for clustering human behaviors in the context of video
surveillance
Combination of clustering and HMMs to group human behaviors
Summary flow
1 After extracting sequences, we use DTW to measure the pairwise
similarity between sequences.
2 Construct an agglomerative hierarchical clustering dendrogram
based on the DTW similarity measure.
3 Recursively, find the optimal set of behavior clusters using HMMs.
Oates et al. (2001) first proposed the idea of using the DTW with
HMMs to cluster time series.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
7 / 29

17. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
New method for clustering human behaviors in the context of video
surveillance
Combination of clustering and HMMs to group human behaviors
Summary flow
1 After extracting sequences, we use DTW to measure the pairwise
similarity between sequences.
2 Construct an agglomerative hierarchical clustering dendrogram
based on the DTW similarity measure.
3 Recursively, find the optimal set of behavior clusters using HMMs.
Oates et al. (2001) first proposed the idea of using the DTW with
HMMs to cluster time series.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
7 / 29

18. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
New method for clustering human behaviors in the context of video
surveillance
Combination of clustering and HMMs to group human behaviors
Summary flow
1 After extracting sequences, we use DTW to measure the pairwise
similarity between sequences.
2 Construct an agglomerative hierarchical clustering dendrogram
based on the DTW similarity measure.
3 Recursively, find the optimal set of behavior clusters using HMMs.
Oates et al. (2001) first proposed the idea of using the DTW with
HMMs to cluster time series.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
7 / 29

19. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
New method for clustering human behaviors in the context of video
surveillance
Combination of clustering and HMMs to group human behaviors
Summary flow
1 After extracting sequences, we use DTW to measure the pairwise
similarity between sequences.
2 Construct an agglomerative hierarchical clustering dendrogram
based on the DTW similarity measure.
3 Recursively, find the optimal set of behavior clusters using HMMs.
Oates et al. (2001) first proposed the idea of using the DTW with
HMMs to cluster time series.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
7 / 29

20. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
New method for clustering human behaviors in the context of video
surveillance
Combination of clustering and HMMs to group human behaviors
Summary flow
1 After extracting sequences, we use DTW to measure the pairwise
similarity between sequences.
2 Construct an agglomerative hierarchical clustering dendrogram
based on the DTW similarity measure.
3 Recursively, find the optimal set of behavior clusters using HMMs.
Oates et al. (2001) first proposed the idea of using the DTW with
HMMs to cluster time series.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
7 / 29

21. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
New method for clustering human behaviors in the context of video
surveillance
Combination of clustering and HMMs to group human behaviors
Summary flow
1 After extracting sequences, we use DTW to measure the pairwise
similarity between sequences.
2 Construct an agglomerative hierarchical clustering dendrogram
based on the DTW similarity measure.
3 Recursively, find the optimal set of behavior clusters using HMMs.
Oates et al. (2001) first proposed the idea of using the DTW with
HMMs to cluster time series.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
7 / 29

22. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Introduction (cont.)
New method for clustering human behaviors in the context of video
surveillance
Combination of clustering and HMMs to group human behaviors
Summary flow
1 After extracting sequences, we use DTW to measure the pairwise
similarity between sequences.
2 Construct an agglomerative hierarchical clustering dendrogram
based on the DTW similarity measure.
3 Recursively, find the optimal set of behavior clusters using HMMs.
Oates et al. (2001) first proposed the idea of using the DTW with
HMMs to cluster time series.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
7 / 29

23. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Compared to the related works
Potential to improve upon the state of the art in intelligent video
surveillance applications by
Bootstrapping human behavior classification and anomaly detection
modules
Supporting incremental HMM learning (performing statistical tests
to select which cluster should be incrementally updated)
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
8 / 29

24. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Compared to the related works
Potential to improve upon the state of the art in intelligent video
surveillance applications by
Bootstrapping human behavior classification and anomaly detection
modules
Supporting incremental HMM learning (performing statistical tests
to select which cluster should be incrementally updated)
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
8 / 29

25. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Compared to the related works
Potential to improve upon the state of the art in intelligent video
surveillance applications by
Bootstrapping human behavior classification and anomaly detection
modules
Supporting incremental HMM learning (performing statistical tests
to select which cluster should be incrementally updated)
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
8 / 29

26. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Outline
1 Introduction
2 Human Behavior Pattern Clustering
3 Experimental Results
4 Conclusion
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
9 / 29

27. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Overview
We divide the proposed method into 2 phases.
1 Blob extraction
2 Behavior clustering
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
10 / 29

28. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Blob Extraction
CCTV�camera
Video
Background
Foreground model Background
Extraction Modeling
List�of�blobs
Single�Blob
Tracking
Blob�features
Vector
Quantization
Observation
symbols
Sequence
Aggregation
Discrete�symbol
sequences
Figure: Block Diagram of Blob Extraction
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
11 / 29

29. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Blob Extraction (cont.)
We represent a blob at time t by the feature vector
ft = xt yt st rt dxt dyt vt ,
where
(xt , yt ) is the centroid of the blob.
st is the size of the blob in pixels.
rt is the aspect ratio of the blob’s bounding box.
(dxt , dyt ) is the unit-normalized motion vector for the blob
compared to the previous frame.
vt is the blob’s speed compared to the previous frame.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
12 / 29

30. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Behavior Clustering
Discrete�symbol
sequences
Similarity
Measurement
Distance
matrix
Agglomerative
Hierarchical�Clustering
Dendrogram
HMM-based
Hierarchical�Clustering
Set�of�HMMs
Figure: Block Diagram of Behavior Clustering
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
13 / 29

31. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Behavior Clustering (cont.)
c����cluster�at�root
0
of�dendrogram
c
C���{��}
0
c����any�cluster
in�C
Train�a�HMM
on�the�sequences
in�c
Is�the�HMM Replace�c�in�C
No
a�sufficient�model with�the�children
of�the�sequences� of�c�from�
in�c? DTW�dendrogram
Yes
Add�the�trained�HMM
to�model�list�M
Remove�c�from�C
No
Is�C�empty?
Yes
Figure: Processing flow of the use of HMM clustering method
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
14 / 29

32. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Behavior Clustering (cont.)
How the processing flow works
Root
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
15 / 29

33. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Behavior Clustering (cont.)
How the processing flow works
The HMM is
sufficient?
Root
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
15 / 29

34. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Behavior Clustering (cont.)
How the processing flow works
Not
sufficient
Root
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
15 / 29

35. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Behavior Clustering (cont.)
How the processing flow works
Root
Child Child
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
15 / 29

36. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Behavior Clustering (cont.)
How the processing flow works
Root
The HMM is The HMM is
sufficient? sufficient?
Child Child
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
15 / 29

37. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Blob Clustering (cont.)
The HMM is not sufficient to model the sequences
When there are more than N sequences in a cluster whose
per-observation log-likelihood is less than a threshold.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
16 / 29

38. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Blob Clustering (cont.)
The HMM is not sufficient to model the sequences
When there are more than N sequences in a cluster whose
per-observation log-likelihood is less than a threshold.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
16 / 29

39. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Blob Clustering (cont.)
To determine the optimal rejection threshold
Use an approach similar to that of Oates et al. (2001).
Generate random sequences from the HMM.
Calculate µc and σc of the per-observation log-likelihood over the
set of generated sequences.
Let a threshold be pc = µc − zσc , where z is experimentally tuned.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
17 / 29

40. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Blob Clustering (cont.)
To determine the optimal rejection threshold
Use an approach similar to that of Oates et al. (2001).
Generate random sequences from the HMM.
Calculate µc and σc of the per-observation log-likelihood over the
set of generated sequences.
Let a threshold be pc = µc − zσc , where z is experimentally tuned.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
17 / 29

41. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Blob Clustering (cont.)
To determine the optimal rejection threshold
Use an approach similar to that of Oates et al. (2001).
Generate random sequences from the HMM.
Calculate µc and σc of the per-observation log-likelihood over the
set of generated sequences.
Let a threshold be pc = µc − zσc , where z is experimentally tuned.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
17 / 29

42. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Blob Clustering (cont.)
To determine the optimal rejection threshold
Use an approach similar to that of Oates et al. (2001).
Generate random sequences from the HMM.
Calculate µc and σc of the per-observation log-likelihood over the
set of generated sequences.
Let a threshold be pc = µc − zσc , where z is experimentally tuned.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
17 / 29

43. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Blob Clustering (cont.)
To determine the optimal rejection threshold
Use an approach similar to that of Oates et al. (2001).
Generate random sequences from the HMM.
Calculate µc and σc of the per-observation log-likelihood over the
set of generated sequences.
Let a threshold be pc = µc − zσc , where z is experimentally tuned.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
17 / 29

44. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Outline
1 Introduction
2 Human Behavior Pattern Clustering
3 Experimental Results
4 Conclusion
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
18 / 29

45. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Overview
Recorded videos at a resolution of 320 × 240 and 25 fps over 1 week.
Used a motion detection to save disk space.
Obtained videos corresponding to over 500 motion events, but
selected the 298 videos containing only a single motion.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
19 / 29

46. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Overview (cont.)
Found that at least 4 common behaviors:
Walking into the building (Walk-in)
Walking out of the building (Walk-out)
Parking a bicycle (Cycle-in)
Riding a bicycle out (Cycle-out)
Other less common activities:
Walking while telephoning, etc. (Other)
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
20 / 29

47. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Overview (cont.)
Figure: Example of common human activities in our testbed scene. (a)
Walking in. (b) Walking out. (c) Cycling in. (d) Cycling out.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
21 / 29

48. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Overview (cont.)
Our main hypothesis
Using DTW as a pre-process prior to HMM-based clustering should
improve the quality of the clusters in term of separating anomalous
from typical behaviors.
Compared to
Using only HMMs
Supervised classification with HMMs
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
22 / 29

49. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Overview (cont.)
Our main hypothesis
Using DTW as a pre-process prior to HMM-based clustering should
improve the quality of the clusters in term of separating anomalous
from typical behaviors.
Compared to
Using only HMMs
Supervised classification with HMMs
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
22 / 29

50. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Overview (cont.)
Our main hypothesis
Using DTW as a pre-process prior to HMM-based clustering should
improve the quality of the clusters in term of separating anomalous
from typical behaviors.
Compared to
Using only HMMs
Supervised classification with HMMs
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
22 / 29

51. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Overview (cont.)
Our main hypothesis
Using DTW as a pre-process prior to HMM-based clustering should
improve the quality of the clusters in term of separating anomalous
from typical behaviors.
Compared to
Using only HMMs
Supervised classification with HMMs
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
22 / 29

52. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Overview (cont.)
Performed 3 experiments.
Evaluated the results according to how well the induced categories
separate the anomalous sequences (hand-labeled with the category
“Other”) from typical sequences (Walk-in, Walk-out, Cycle-in,
Cycle-out).
1 Using our proposed method
2 Using only HMMs
3 Using HMMs with supervised learning
In all 3 experiments, we chose linear HMMs with 4 states based on
our previous empirical experience.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
23 / 29

53. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Overview (cont.)
Performed 3 experiments.
Evaluated the results according to how well the induced categories
separate the anomalous sequences (hand-labeled with the category
“Other”) from typical sequences (Walk-in, Walk-out, Cycle-in,
Cycle-out).
1 Using our proposed method
2 Using only HMMs
3 Using HMMs with supervised learning
In all 3 experiments, we chose linear HMMs with 4 states based on
our previous empirical experience.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
23 / 29

54. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Overview (cont.)
Performed 3 experiments.
Evaluated the results according to how well the induced categories
separate the anomalous sequences (hand-labeled with the category
“Other”) from typical sequences (Walk-in, Walk-out, Cycle-in,
Cycle-out).
1 Using our proposed method
2 Using only HMMs
3 Using HMMs with supervised learning
In all 3 experiments, we chose linear HMMs with 4 states based on
our previous empirical experience.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
23 / 29

55. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Overview (cont.)
Performed 3 experiments.
Evaluated the results according to how well the induced categories
separate the anomalous sequences (hand-labeled with the category
“Other”) from typical sequences (Walk-in, Walk-out, Cycle-in,
Cycle-out).
1 Using our proposed method
2 Using only HMMs
3 Using HMMs with supervised learning
In all 3 experiments, we chose linear HMMs with 4 states based on
our previous empirical experience.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
23 / 29

56. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Overview (cont.)
Performed 3 experiments.
Evaluated the results according to how well the induced categories
separate the anomalous sequences (hand-labeled with the category
“Other”) from typical sequences (Walk-in, Walk-out, Cycle-in,
Cycle-out).
1 Using our proposed method
2 Using only HMMs
3 Using HMMs with supervised learning
In all 3 experiments, we chose linear HMMs with 4 states based on
our previous empirical experience.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
23 / 29

57. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Overview (cont.)
Performed 3 experiments.
Evaluated the results according to how well the induced categories
separate the anomalous sequences (hand-labeled with the category
“Other”) from typical sequences (Walk-in, Walk-out, Cycle-in,
Cycle-out).
1 Using our proposed method
2 Using only HMMs
3 Using HMMs with supervised learning
In all 3 experiments, we chose linear HMMs with 4 states based on
our previous empirical experience.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
23 / 29

58. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Overview (cont.)
Our configuration
the number of deviant patterns allowed in a cluster N = 10
z = 2.0 for a threshold pc = µc − zσc
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
24 / 29

59. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Results for Experiment I
Clustering results for Experiment I (DTW+HMMs).
Cluster # Walk-in Walk-out Cycle-in Cycle-out Other
1 96 0 18 0 0
2 0 54 0 5 0
3 0 3 0 8 0
4 0 2 0 0 0
5 0 1 0 2 0
...
14 0 0 0 0 4
15 0 0 0 0 4
16 0 0 0 0 2
17 0 0 0 0 2
One-seq
clusters 4 17 34 21 4
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
25 / 29

60. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Results for Experiment II
Begin by training a single HMM on all sequences.
Assign every sequence with a per-observation log-likelihood above a
threshold pc to a cluster.
Repeat the process by training a new HMM on the remaining
sequences.
Clustering results for Experiment II (HMMs only).
Cluster # Walk-in Walk-out Cycle-in Cycle-out Other
1 15 77 49 43 16
2 80 0 11 2 0
3 5 0 0 0 0
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
26 / 29

61. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Results for Experiment III
Trained 4 HMMs on each of the four typical beahviors.
Maximize the F1 value to determine the best per-observation
log-likelihood threshold for each HMM.
For the best separation between the positive and negative test
patterns
Results for Experiment III (Supervised classification with HMMs).
Anomaly detection rate (%) False alarm rate (%)
50 24.6
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
27 / 29

62. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Outline
1 Introduction
2 Human Behavior Pattern Clustering
3 Experimental Results
4 Conclusion
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
28 / 29

63. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Conclusion
We have proposed and evaluated a new method for clustering
human behaviors.
Our method
provides an initial partitioning of a set of behavior sequences, then
automatically identifies where to cut off the hierarchical clustering
dendrogram.
could be used to bootstrap an anomaly detection module for
intelligent video surveillance applications.
shows a perfect separation between typical and anomalous behaviors
on real-world surveillance data without any information about the
labels.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
29 / 29

64. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Conclusion
We have proposed and evaluated a new method for clustering
human behaviors.
Our method
provides an initial partitioning of a set of behavior sequences, then
automatically identifies where to cut off the hierarchical clustering
dendrogram.
could be used to bootstrap an anomaly detection module for
intelligent video surveillance applications.
shows a perfect separation between typical and anomalous behaviors
on real-world surveillance data without any information about the
labels.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
29 / 29

65. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Conclusion
We have proposed and evaluated a new method for clustering
human behaviors.
Our method
provides an initial partitioning of a set of behavior sequences, then
automatically identifies where to cut off the hierarchical clustering
dendrogram.
could be used to bootstrap an anomaly detection module for
intelligent video surveillance applications.
shows a perfect separation between typical and anomalous behaviors
on real-world surveillance data without any information about the
labels.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
29 / 29

66. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Conclusion
We have proposed and evaluated a new method for clustering
human behaviors.
Our method
provides an initial partitioning of a set of behavior sequences, then
automatically identifies where to cut off the hierarchical clustering
dendrogram.
could be used to bootstrap an anomaly detection module for
intelligent video surveillance applications.
shows a perfect separation between typical and anomalous behaviors
on real-world surveillance data without any information about the
labels.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
29 / 29

67. Introduction Human Behavior Pattern Clustering Experimental Results Conclusion
Conclusion
We have proposed and evaluated a new method for clustering
human behaviors.
Our method
provides an initial partitioning of a set of behavior sequences, then
automatically identifies where to cut off the hierarchical clustering
dendrogram.
could be used to bootstrap an anomaly detection module for
intelligent video surveillance applications.
shows a perfect separation between typical and anomalous behaviors
on real-world surveillance data without any information about the
labels.
Clustering Human Behaviors with Dynamic Time Warping and Hidden Markov Models for a Video Surveillance System
29 / 29