Invited talk at the Workshop on Background Models Challenge, ACCV 2012, Daejon, Korea, November 2012.

1. Background Modeling and Foreground
Detection for Video Surveillance:
Recent Advances and Future Directions
Thierry BOUWMANS
Associate Professor
MIA Lab - University of La Rochelle - France

2. Plan
 Introduction
 Fuzzy Background Subtraction
 Background Subtraction via a Discriminative
Subspace Learning: IMMC
 Foreground Detection via Robust Principal
Component Analysis (RPCA)
 Conclusion - Perspectives
2

3. Goal
 Detection of moving objects in video sequence.
 Pixels are classified as:
Background(B) Foreground (F)
Séquence Pets 2006 : Image298 (720 x 576 pixels)
3

4. Background Subtraction Process
Incremental Algorithm
t >N Background
Maintenance
t ≥N t=t+1
Batch Algorithm
t ≤N
N images
Video Background F(t) Foreground
Initialization I(t+1) Detection Foreground
N+1
Mask
Classification task
4

5. Related Applications
 Video surveillance
 Optical Motion Capture
 Multimedia Applications
Séquence Danse [Mikic 2002] – Université de Californie SanJump [Mikic 2002]
Projet Aqu@theque – Université de La Rochelle
Projet ATON Séquence Diego
5

6. On the importance of the background
subtraction
Background
Processing
Subtraction
Acquisition
Convex Hull Tracking
6 Pattern Recognition

7. Challenges
 Critical situations which generate false
detections :
 Shadows Illumination variations…
-
Source : Séquence Pets 2006 Image 0298 (720 x 576 pixels)
7

8.  Multimodal Backgrounds
Rippling Water Camera Waving
Water Surface Jitter Trees
Source: http://perception.i2r.a-star.edu.sg/bk_model/bk_index.html
8

9. Statistical Background Modeling
 Background Subtraction Web Site: References (553),
datasets (10) and codes (27).
Source: http://sites.google.com/site/backgroundsubtraction/Home.html
(6256 Visitors, Source Google Analytics).
9

10. Plan
 Introduction
 Fuzzy Background Subtraction
 Background Subtraction via a Discriminative
Subspace Learning: IMMC
 Foreground Detection via Robust Principal
Component Analysis (RPCA)
 Conclusion - Perspectives
10

11. Fuzzy Background Subtraction
 A survey in Handbook on Soft Computing for Video
Surveillance, Taylor and Francis Group [HSCVS
2012]
 Three approaches developed at the MIA Lab:
 Background modeling by Type-2 Fuzzy Mixture of
Gaussians Model [ISVC 2008].
 Foreground Detection using the Choquet Integral
[WIAMIS 2008][FUZZ’IEEE 2008]
 Fuzzy Background Maintenance [ICIP 2008]
11

12. Weakness of the original MOG
1. False detections due to the matching test
kσ1 kσ 2 kσ 3
12

13. Weakness of the original MOG
2. False detections due to the presence of outliers in
the training step
Exact distribution
μ
μ min μ max
13

14. Mixture of Gaussians
with uncertainty on :
the mean and the variance [Zeng 2006]
(T2 FMOG-UM) (T2 FMOG-UV)
14

15. Mixture of Gaussians with uncertainty on
the mean
(T2 FMOG-UM)
X t ,c : Intensity vector in the RGB color space
15

16. Mixture of Gaussians with uncertainty on
the variance
(T2 FMOG-UV)
X t ,c : Intensity vector in the RGB color space
16

17. Classification B/F by T2-FMOG
Matching test:
Classification B/F as the MOG ⇒
17

18. Results on the “SHAH” dataset
(160 x 128 pixels) – Camera Jitter
Video at http://sites.google.com/site/t2fmog/
Original sequence MOG
T2 FMOG-UM (km=2) T2 FMOG-UV (kv=0.9)
18

19. Results on the “SHAH” dataset
(160 x 128 pixels) – Camera Jitter
Method Error Image Image Image Image Total Variation in %
Type 271 373 410 465 Error
MOG FN 0 1120 4818 2050
FP 2093 4124 2782 1589 18576
T2-FMOG-UM FN 0 1414 6043 2520
FP 203 153 252 46 10631 42,77
T2-FMOG-UV FN 0 957 2217 1069
FP 3069 1081 1119 1158 10670 42.56
19

20. Results on the “SHAH” dataset
(160 x 128 pixels) – Camera Jitter
[Stauffer 1999]
[Bowden 2001] – Initialization [Zivkovic 2004] – K is variable
20

21. Results on the sequence “CAMPUS”
(160 x 128 pixels) – Waving Trees
Video at http://sites.google.com/site/t2fmog/
Original Sequence MOG
T2 FMOG-UM (km=2) T2 FMOG-UV (kv=0.9)
21

22. Resultat on the sequence “Water
Surface” (160 x 128 pixels) – Water Surface
Video at http://sites.google.com/site/t2fmog/
Original Sequence MOG
T2 FMOG-UM (km=2) T2 FMOG-UV (kv=0.9)
22

23. Fuzzy Foreground Detection :
 Features: color, edge, stereo features, motion
features, texture.
 Multiple features:
More robustness in presence of illumination
changes, shadows and multimodal backgrounds
23

24. Choice of the features
 Color (3 components)
 Texture (Local Binary Pattern [Heikkila – PAMI 2006])
For each feature, a similarity (S) is computed
following its value in the background image and
its value in the current image.
24

25. Aggregation of the Color and Texture features with the
Choquet Integral
BG(t) I(t+1)
Color Features Texture Features
S
Similarity mesure
C,1
for the Color
SC,2 SC,3SimilarityTexture
ST measure
for the
Fuzzy Integral
Classification B/F
25 Foreground Mask

26. How to compute S for the Color and the
Texture?
TF TI
C F, k C I, k 0 ≤ T,C ≤ 255
Background Image Current Image
 C FBk
T,
 if CT,B < CTk
if F k < I , I
 C I Ik
T
For the
the ST = 1
 ,
SC ,k =  1 if CT,B = CTk
if F k = I , I 0≤S ≤1
 CI ,k
Color
Texture  I
T if CTk < C F ,k k=one of the color
 CF ,k if I , I < TB
T B components
26

27. Fuzzy operators
« Sugeno Integral» et «Choquet Integral»
 Uncertainty and imprecision
 Great flexibility
 Fast and simple operations
ordinal cardinal
27

28. Data Fusion using the Choquet Integral
Mesures floues :
Intégrale de
Choquet :
X = { x1 , x 2 , x 3 } {x } {x } {x } {x ,x } {x ,x } {x
1 2 3 1 2 1 3 2 , x3}
28

29. Fuzzy Foreground Detection
Classification using the Choquet integral
If C μ ( x , y ) < Th then ( x, y ) ∈ Background
else ( x, y ) ∈ Foreground
where Th is constant threshold. Cμ ( x, y) is the value of
the Choquet integral for the pixel (x,y)
29

30. Aggregation Color, Texture
 Aqu@thèque (384 x 288 pixels) - Ohta color space
Integral Choquet Sugeno
Color space Ohta Ohta
S(A,B) 0.40 0.27
a) Current image b) Ground truth
Comparison between the Sugeno and Choquet [Zhang 2006]
30 c) Choquet integral d) Sugeno integral

31. Aggregation Colors, Texture : Ohta, YCrCb, HSV
 Aqu@thèque (384 x 288 pixels)
Texture Color
{x } {x } {x } {x ,x } {x ,x } {x
1 2 3 1 2 1 3 2 , x3} X = { x1 , x 2 , x 3 }
0.6 0.3 0.1 0.9 0.7 0.4 1
0.5 0.4 0.1 0.9 0.6 0.5 1
Choquet - Ohta 0.2
0.5 0.3 0.8 0.7 0.5
Choquet - YCrCb 1
Choquet - HSV
0.5 0.39 0.11 0.89 0.61 0.5 1
0.53 0.34 0.13 0.87 0.66 0.47 1
Integral
Ohta YCrCb
Values of the fuzzy measures μ HSV
Color Space
S(A,B) 0.40 0.42 0.30
Evaluation of the Choquet integral for different color spaces
31

32. Aggregation Color, Texture
 VS-Pets 2003 (720 x 576)
Current Image Choquet - YCrCb Sugeno – Ohta [Zhang 2006]
32

33. Aggregation Colors : Pets 2006 (384 x 288 pixels)
Original sequence Ground truth
OR Sugeno Integral Choquet Integral
YCrCb
Ohta
HSV
33

34. Fuzzy Background maintenance
 No-selective rule
 Selective rule
Here, the idea is to adapt very quickly a pixel classified as
background and very slowly a pixel classified as foreground.
34

35. Fuzzy adaptive rule
and
Combination of the update rules of the selective scheme
35

36. Results on the Wallflower dataset
Sequence Time of Day
Original Image 1850 Ground Truth
No selective rule Selective rule Fuzzy adaptive rule
Similarity measure
No selective Selectiv Fuzzy adaptive
e
S(A,B)% 58.40 57.08 58.96
36

37. Computation Time
Algorithm Frames/Second
T2-FMOG-UM 11
T2-FMOG-UV 12
MOG 20
Choquet integral 31
Sugeno integral 22
OR 40
Resolution 384*288, RGB, Pentium 1,66GHz, RAM 1GB
37

38. Perspective
Assessment
s
 Fuzzy Background Modeling by T2-FMOG
 Multimodal Backgrounds
- Using fuzzy approaches in other statistical models.
 Fuzzy Foreground Detection using multi-features
- Using more than two features
- Fuzzy measures by learning
 Fuzzy Background Maintenance
38

39. Plan
 Introduction
 Fuzzy Background Subtraction
 Background Subtraction via a Discriminative
Subspace Learning: IMMC
 Foreground Detection via Robust Principal
Component Analysis (RPCA)
 Conclusion - Perspectives
39

40. Background Modeling and Foreground Detection
via a Discriminative Subspace Learning (MIA Lab)
 Reconstructive subspace learning models (PCA, ICA, IRT)
[RPCS 2009]
 Assumption: The main information contained in the training
sequence is the background meaning that the foreground
has a low contribution.
 However, this assumption is only verified when the moving
objects are either small or far away from the camera.
40

41. Discriminative Subspace Learning
 Advantages
 More efficient and often give better classification results.
 Robust supervised initialization of the background
 Incremental update of the eigenvectors and eigenvalues.
 Approach developed at the MIA Lab:
 Background initialization via MMC [MVA 2012]
 Background maintenance via Incremental Maximum
Margin Criterion (IMMC) [MVA 2012]
41

42. Background Subtraction via Incremental
Maximum Margin Criterion
 Denote the training video sequences S ={I1, ...IN}
where It is the frame at time t
N is the number of training frames.
 Let each pixel (x,y) be characterized by its intensity in the grey
scale and asssume that we have the ground truth corresponding to
this training video sequence, i.e we know for each pixel its class
label that can be foreground or background.
42

43. Background Subtraction via Incremental
Maximum Margin Criterion
 Thus, we compute respectively the inter-class scatter matrix Sb
and the intra-class scatter matrix Sw:
where c = 2
I is the mean of the intensity of the pixel (x,y) over the training video
Ii is the mean of samples belonging to class i
pi is the prior probability for a sample belonging to class i (Background,
Foreground).
43

44. Background Subtraction via Incremental
Maximum Margin Criterion
 Batch Maximum Margin Criterion algorithm.
 Extract the first leading eigenvectors that correspond to the
background. The corresponding eigenvalues are contained
in the matrix LM and the leading eigenvectors in the matrix
ΦM .
 The current image It can be approximated by the mean
background and weighted sum of the leading
eigenbackgrounds ΦM.
44

45. Background Subtraction via Incremental
Maximum Margin Criterion
 The coordinates in leading eigenbackground space of the current
image It can be computed :
 When wt is back projected onto the image space, the background
image is created :
45

46. Background Subtraction via Incremental
Maximum Margin Criterion
 Foreground detection
 Background maintenance via IMMC
46

47. Principle - Illustration
Current Image
IBackground
IForeground
Background image
Foreground mask
47

48. Results on the Wallflower dataset
Original image, ground truth , SG, MOG, KDE,
PCA, INMF, IRT, IMMC (30), IMMC (100)
48

49. Perspective
Assessment
s
 Advantages
 Robust supervised initialization of the background.
 Incremental update of the eigenvectors and
eigenvalues.
 Disadvantages
 Needs ground truth in the training step.
 Others Discriminative Subspace Learning
methods such as LDA.
49

50. Plan
 Introduction
 Fuzzy Background Subtraction
 Background Subtraction via a Discriminative
Subspace Learning: IMMC
 Foreground Detection via Robust Principal
Component Analysis (RPCA)
 Conclusion - Perspectives
50

51. Foreground Detection via Robust Principal
Component Analysis
 PCA (Oliver et al 1999): Not robust to outliers.
 Robust PCA (Candes et al. 2011): Decomposition
into low-rank and sparse matrices
 Approach developed at the MIA Lab:
 Validation [ICIP 2012][ICIAR 2012][ISVC 2012]
 RPCA via Iterative Reweighted Least Squares [BMC
2012]
51

52. Robust Principal Component Analysis
 Candes et al. (ACM 2011) proposed a convex optimization to address
the robust PCA problem. The observation matrix A is assumed
represented as:
where L is a low-rank matrix and S must be sparse matrix with a small
fraction of nonzero entries.
52 http://perception.csl.illinois.edu/matrix-rank/home.html

53. Robust Principal Component Analysis
 This research seeks to solve for L with the following optimization
problem:
where ||.||* and ||.||1 are the nuclear norm (which is the l1-norm of singular
value) and l1-norm, respectively, and λ > 0 is an arbitrary balanced
parameter.
 Under these minimal assumptions, this approach called Principal
Component Pursuit (PCP) solution perfectly recovers the low-rank and
the sparse matrices.
53

54. Algorithms for solving PCP
Time required to solve a 1000x1000=106 RPCA problem:
Algorithms Accuracy Rank ||E||_0 # iterations time (sec)
IT 5.99e-006 50 101,268 8,550 119,370.3
DUAL 8.65e-006 50 100,024 822 1,855.4
10,000
times
APG 5.85e-006 50 100,347 134 1,468.9
speedup!
APGP 5.91e-006 50 100,347 134 82.7
ALMP 2.07e-007 50 100,014 34 37.5
ADMP 3.83e-007 50 99,996 23 11.8
Source: Z. Lin , Y. Ma “The Pursuit of Low-dimensional Structures in High-dimensional
(Visual) Data: Fast and Scalable Algorithms”
Time required is still acceptable for ADM but for background
modeling and foreground detection?
54

55. Application to Background Modeling and
Foreground Detection
n is the amount of pixels in a frame (106)
m is the number of frames considered (200)
Computation time is 200* 12s= 40 minutes!!!
Source: http://perception.csl.illinois.edu/matrix-rank/home.html
55

56. PCP and its application to Background
Modeling and Foreground Detection
 Only visual validations are provided!!!
 Limitations:
 Spatio-temporal aspect: None!
 Real Time Aspect: PCP takes 40 minutes with the
ADM!!!
 Incremental Aspect: PCP is a batch algorithm. For
example, (Candes et al. 2011) collected 200 images.
56

57. PCP and its variants
How to improve PCP?
 Algorithms for solving PCP (17 Algorithms)
 Incremental PCP (5 papers)
 Real-Time PCP (2 papers)
Validation for background modeling and foreground detection
(3 papers) [ICIP 2012][ICIAR 2012][ISVC 2012]
Source: T. Bouwmans, Foreground Detection using Principal Component Pursuit: A Survey, under preparation.
57

58. PCP and its variants
Source: T. Bouwmans, Foreground Detection using Principal Component Pursuit: A Survey, under preparation.
58

59. Validation Background Modeling and Foreground
Detection: Qualitative Evaluation
Original image
Ground truth
PCA
RSL
PCP-EALM
PCP-IADM
PCP-LADM
PCP-LSADM
BPCP-IALM
59 Source: ICIP 2012, ICIAR 2012, ISVC 2012

60. Validation Background Modeling and Foreground
Detection : Quantitative Evaluation
F-Measure
Block PCP gives the best performance!
60 Source: ICIP 2012, ICIAR 2012, ISVC 2012

61. PCP and its application to Background
Modeling and Foreground Detection
 Recent improvements:
 BPCP (Tang et Nehorai (2012)) : Spatial but not incremental and
not real time!
 Recursive Robust PCP (Qiu and Vaswani (2012) ): Incremental but
not real time!
 Real Time Implementation on GPU (Anderson et al. (2012) ): Real
time but not incremental!
 What we can do?
 Research on real time incremental robust PCP!
61

62. Perspective
Conclusion
s
 Fuzzy Background Subtraction
 Background Subtraction via a Discriminative Subspace
Learning: IMMC
 Foreground Detection via Robust Principal Component
Analysis (RPCA)
 Fuzzy Learning Rate
 Other Discriminative Subspace Learning methods such
as LDA
 Incremental and real time RPCA
62

63. Publications
Chapter Fuzzy Background Subtraction
T. Bouwmans, “Background Subtraction For Visual Surveillance: A Fuzzy Approach”,
Handbook on Soft Computing for Video Surveillance, Taylor and Francis Group, Chapter
5, March 2012.
International Conferences :
F. El Baf, T. Bouwmans, B. Vachon, “Fuzzy Statistical Modeling of Dynamic
Backgrounds for Moving Object Detection in Infrared Videos”, CVPR 2009 Workshop ,
pages 1-6, Miami, USA, 22 June 2009.
F. El Baf, T. Bouwmans, B. Vachon, “Type-2 Fuzzy Mixture of Gaussians Model:
Application to Background Modeling”, ISVC 2008, pages 772-781, Las Vegas, USA,
December 2008
F. El Baf, T. Bouwmans, B. Vachon, “A Fuzzy Approach for Background Subtraction”,
ICIP 2008, San Diego, California, U.S.A, October 2008.
F. El Baf, T. Bouwmans, B. Vachon. " Fuzzy Integral for Moving Object Detection ",
IEEE-FUZZY 2008 , Hong Kong, China, June 2008.
F. El Baf, T. Bouwmans, B. Vachon, “Fuzzy Foreground Detection for Infrared Videos”,
CVPR 2008 Workshop , pages 1-6, Anchorage, Alaska, USA, 27 June 2008.
F. El Baf, T. Bouwmans, B. Vachon, “Foreground Detection using the Choquet Integral”,
International Workshop on Image Analysis for Multimedia Interactive Services, WIAMIS
2008, pages 187-190, Klagenfurt, Austria, May 2008.

64. Publications
Background Subtraction via IMMC
Journal
D. Farcas, C. Marghes, T. Bouwmans, “Background Subtraction via Incremental
Maximum Margin Criterion: A discriminative approach” , Machine Vision and
Applications , March 2012.
International Conferences :
C. Marghes, T. Bouwmans, "Background Modeling via Incremental Maximum Margin
Criterion", International Workshop on Subspace Methods, ACCV 2010 Workshop
Subspace 2010, Queenstown, New Zealand, November 2010.
D. Farcas, T. Bouwmans, "Background Modeling via a Supervised Subspace Learning",
International Conference on Image, Video Processing and Computer Vision, IVPCV
2010, pages 1-7, Orlando, USA , July 2010.

65. Publications
Chapter Foreground Detection via RPCA
C. Guyon, T. Bouwmans, E. Zahzah, “Robust Principal Component Analysis for
Background Subtraction: Systematic Evaluation and Comparative Analysis”, INTECH,
Principal Component Analysis, Book 1, Chapter 12, page 223-238, March 2012.
International Conferences :
C. Guyon, T. Bouwmans. E. Zahzah, “Foreground Detection via Robust Low Rank Matrix
Factorization including Spatial Constraint with Iterative Reweighted Regression”,
International Conference on Pattern Recognition, ICPR 2012, Tsukuba, Japan,
November 2012.
C. Guyon, T. Bouwmans. E. Zahzah, “Moving Object Detection via Robust Low Rank
Matrix Decomposition with IRLS scheme”, International Symposium on Visual
Computing, ISVC 2012,pages 665–674, Rethymnon, Crete, Greece, July 2012.
C. Guyon, T. Bouwmans, E. Zahzah, “Moving Object Detection by Robust PCA solved
via a Linearized Symmetric Alternating Direction Method”, International Symposium on
Visual Computing, ISVC 2012, pages 427-436, Rethymnon, Crete, Greece, July 2012.
C. Guyon, T. Bouwmans, E. Zahzah, "Foreground Detection by Robust PCA solved via a
Linearized Alternating Direction Method", International Conference on Image Analysis
and Recognition, ICIAR 2012, pages 115-122, Aveiro, Portugal, June 2012.
C. Guyon, T. Bouwmans, E. Zahzah, "Foreground detection based on low-rank and
block-sparse matrix decomposition", IEEE International Conference on Image
Processing, ICIP 2012 , Orlando, Florida, September 2012.