Paper Presentation, ISBC 2015 Workshop conjunction with AVSS 2015, Karlsruhe, Germany, 2015. Double-constrained RPCA based on Saliency Maps for Foreground Detection in Automated Maritime Surveillance

1. Double-constrained RPCA based on
Saliency Maps for Foreground Detection in
Automated Maritime Surveillance
Andrews Sobral, Thierry Bouwmans and El-hadi ZahZah
Ph.D. Student, Computer Vision
Lab. L3I/MIA – University of La Rochelle, France

2. Summary
▪ Context: Maritime Surveillance
▪ Understanding an Intelligent Video Surveillance Framework
▪ Introduction to Background Subtraction
▪ Robust Principal Component Analysis (RPCA)
▪ The SCM-RPCA approach
▪ Definition of shape and confidence map
▪ Solving the SCM-RPCA
▪ Experimental results
▪ Conclusions

3. Context: Maritime Surveillance
http://www.asv.fr/en/media
Images from:
▪ The automatic video surveillance of maritime
environment aims to:
▪ Detect and identify specific ships, vessels or boats on
the sea and their activity.
▪ Extend the capabilities to identify maritime risks: piracy,
trafficking, immigration, etc.
▪ However, the development of automatic video
surveillance applications for maritime environment
is a very difficult task due to the complexity of the
scenes: moving water, waves, etc.
▪ The motion of the objects of interest (i.e. ships or
boats) can be mixed with the dynamic behavior of the
background (non-regular patterns).

4. !
Video Content Analysis
(VCA) or Video Analytics
Behavior
Analysis
Image
acquisition and
preprocessing
Object
Detection
Object
Tracking
event location
Intrusion
detection
Collision
prevention
Target detection
and tracking
Anomaly
detection
Target behavior
analysis
Traffic data
collection and
analysis
activity report
Understanding an Intelligent Video Surveillance Framework
supervisor
Behind the Scenes of an Intelligent Video Surveillance
Framework
!
Example of automatic
incident detection
our focus

5. Introduction to Background Subtraction
Initialize
Background Model
frame model
Foreground
Detection
Background Model
Maintenance

6. Background subtraction methods
Traditional methods:
• Basic methods, mean and variance over time
• Fuzzy based methods
• Statistical methods
• Non-parametric methods
• Neural and neuro-fuzzy methods
Matrix and Tensor Factorization methods:
• Eigenspace-based methods (PCA / SVD)
• RPCA, LRR, NMF, MC, ST, etc.
• Tensor Decomposition, NTF, etc.
BGSLibrary (C++)
https://github.com/andrewssobral/bgslibrary
A large number of algorithms have been proposed for background subtraction
over the last few years:
LRSLibrary (MatLab)
https://github.com/andrewssobral/lrslibrary
our focus
Andrews Sobral and Antoine Vacavant. A comprehensive review of background subtraction algorithms
evaluated with synthetic and real videos. Computer Vision and Image Understanding (CVIU), 2014.
http://dx.doi.org/10.1016/j.cviu.2013.12.005

7. Robust Principal Component Analysis (RPCA)
▪ RPCA can be formulated as the problem of decomposing a data matrix (M) into two
components (L) and (S), where (M) is the sum of a low-rank matrix (L) and a sparse
matrix (S):
▪ M = L + S
Sparse error matrix
SL
Underlying low-rank matrix
M
Matrix of corrupted observations

8. Robust Principal Component Analysis (RPCA)
http://perception.csl.illinois.edu/matrix-rank/home.html
M L S
(tradeoff between rank and sparsity)

9. What about RPCA for dynamic background?
▪

10. The SCM-RPCA approach
▪ We propose to combine some ideas of Oreifej et al. (2013) and Yang et al. (2015).
▪ The weighting matrix proposed by Yang et al. (2015) can be used as a shape constraint (or region
constraint),
▪ While the confidence map proposed by Oreifej et al. (2013) reinforces the pixels belonging from
the moving objects.
▪ The original 3WD was modified adding the shape constraint as has been done in the
RMAMR.
▪ We chose to modify the 3WD instead of RMAMR due its capacity to deal more robustly with the
multimodality of the background.

11. Definition of shape and confidence map
▪ The second contribution of this paper refers to the way of building the shape constraint and
confidence map.
▪ Instead of using dense optical flow (temporal descriptor) as a preliminary step, we suggest to use a
saliency detector (spatial descriptor).
▪ In some cases where:
▪ a) the object of interest moves very slowly (i.e long distance boats) or
▪ b) the background is very dynamic (i.e boats in the sea), the optical flow may not be enough to ensure the object
detection.
▪ In addition, the computation of the dense optical flow usually request more computational cost than
saliency maps.
▪ In this work, the BMS method proposed by Zhang and Sclaroff (2014) was selected by its speed performance and
visual results.
http://cs-people.bu.edu/jmzhang/BMS/BMS.html

12. Block diagram of the SCM-RPCA
Is important to note that the double constraints (confidence map and shape) can be
built from two different types of source (i.e. from spatial, temporal, or spatio-temporal
information), but in this work we focus only on spatial saliency maps.

13. Solving the SCM-RPCA

14. Experimental results
UCSD MarDT
The UCSD and MarDT data sets consists of 18 and 28 video sequences, respectively, both
acquired from a stationary and moving camera, but in this work (due to page limits) we
have selected only four sequences from UCSD and three sequences from MarDT.
http://www.svcl.ucsd.edu/projects/background_subtraction/ucsdbgsub_dataset.htm
http://www.dis.uniroma1.it/~labrococo/MAR/index.htm

15. Quantitative results on UCSD data set
The quantitative results in Table 2 show that the SCM-RPCA outperforms the previous methods, with the
highest F-measure average over the selected video sequences.
Is important to note that all constrained RPCA evaluated in this paper have been used the saliency maps as
the input constraint.

16. Benchmark evaluation over UCSD data set
The algorithms were implemented in MATLAB (R2014a) running on a laptop computer with
Windows 7 Professional 64 bits, 2.7 GHz Core i7-3740QM processor and 32Gb of RAM.
We can see that the combination of shape constraint and confidence map did not changed
significantly the number of iterations and computation time over original 3WD.

17. Visual results on UCSD data set
From left to right: (a) input frame, (b) saliency map generated by BMS, (c) ground truth, (d) proposed
approach, (e) 3WD, and (f) RMAMR.

18. Visual results on MarDT data set
Is important to note that in the UCSD scenes we have used the original spatial saliency map provided
by BMS, while for the MarDT scenes we have subtracted its temporal median due to the high saliency
from the buildings around the river.

19. Conclusions
▪ The experimental results indicate a better enhancement of the object foreground mask
when compared with its direct competitors.
▪ As shown in quality evaluation, the combination with confidence map and shape
constraint can reduce the amount of false positive pixels.
▪ In the future work, we will investigate how the combination of different types of source
can improve the foreground detection.
▪ We are also interested to perform the decomposition incrementally as in Online RPCA proposed
by Feng et al. (2013).
▪ MATLAB codes available: https://sites.google.com/site/scmrpca