Background subtraction is a technique used to separate foreground objects from backgrounds in video frames. It works by comparing each frame to a background model and detecting differences which indicate moving foreground objects. Recursive techniques like mixtures of Gaussians model the background pixel values over time using multiple Gaussian distributions, allowing the background model to adapt to changing lighting conditions. Adaptive background/foreground detection uses a background model that evolves over time to distinguish foreground objects from the background in a robust way.
2. Background Subtraction
• As the name suggests, background subtraction
is the process of separating out foreground
objects from the background in a sequence of
video frames.
12/8/2011 2
4. Fundamental Logic
• Fundamental logic for detecting moving
objects from the difference between the
current frame and a reference frame, called
“background image” and this method is
known as FRAME DIFFERENCE METHOD
12/8/2011 4
8. Background Subtraction Principles
At ICCV 1999, MS Research presented a study, Wallflower: Principles and Practice
of Background Maintenance, by Kentaro Toyama, John Krumm, Barry
Brumitt, Brian Meyers. This paper compared many different background
subtraction techniques and came up with some principles:
P1:
P2:
P3:
P4:
12/8/2011 8
9. The Problem - Requirements
The background image is not fixed but must adapt to:
• Illumination changes
•Gradual •Sudden (such as clouds)
12/8/2011 9
10. • Motion changes
•Camera oscillations • High-frequencies background
objects (such as tree
branches, sea waves, and
similar)
12/8/2011 10
11. • Changes in the background geometry
•Parked cars
12/8/2011 11
12. Widely Used!
• Traffic monitoring (counting
vehicles, detecting & tracking
vehicles),
• Human action recognition
(run, walk, jump, squat),
• Human-computer interaction
(“human interface")
• Object tracking (watched tennis
lately),
• And in many other cool
applications of computer vision
such as digital forensics.
12/8/2011 12
13. Simple Approach
Image at time t: Background at time t:
I (x, y, t) B(x, y, t)
> Th
1. Estimate the background for time t.
2. Subtract the estimated background from the input frame.
3. Apply a threshold, Th, to the absolute difference to get the
foreground mask.
12/8/2011 13
14. The basic methods
• Frame difference:
| framei–framesi-1| > Th
• The estimated background is just the previous
frame
• It evidently works only in particular conditions
of objects’ speed and frame rate
• Very sensitive to the threshold Th
12/8/2011 14
15. Frame Differencing
• Background is estimated to be the previous
frame. Background subtraction equation then
becomes:
B(x ,y ,t) = I (x ,y ,t-1)
I (x, y, t) - I (x, y, t – 1) > Th
12/8/2011 15
16. Frame Differencing
• Depending on the object
structure, speed, frame rate and global
threshold, this approach may or may not be
useful (usually not).
> Th
12/8/2011 16
18. Thresholding
• The key parameter in the thresholding process is the
choice of the threshold value.
• Several different methods for choosing a threshold
exist,
– Users can manually choose a threshold value, or
– A thresholding algorithm can compute a value
automatically, which is known as automatic thresholding
12/8/2011 18
19. Motivation
• A robust background subtraction algorithm should handle:
lighting changes, repetitive motions from clutter and long-
term scene changes.
12/8/2011 19
20. Algorithm Overview
• The values of a particular pixel is modeled as a mixture of adaptive
Gaussians.
– Why mixture? Multiple surfaces appear in a pixel.
– Why adaptive? Lighting conditions change.
• At each iteration Gaussians are evaluated using a simple heuristic to
determine which ones are mostly likely to correspond to the
background.
• Pixels that do not match with the “background Gaussians” are
classified as foreground.
• Foreground pixels are grouped using 2D connected component
analysis.
12/8/2011 20
21. Advantages vs. Shortcomings
• Advantages:
– A different “threshold” is selected for each pixel.
– These pixel-wise “thresholds" are adapting by time.
– Objects are allowed to become part of the background
without destroying the existing background model.
– Provides fast recovery.
• Disadvantages:
– Cannot deal with sudden, drastic lighting changes.
– Initializing the Gaussians is important (median filtering).
– There are relatively many parameters, and they should be
selected intelligently.
12/8/2011 21
22. Challenges background subtraction
Algorithm
• It must be robust against changes in illumination.
• It should avoid detecting non-stationary background
objects such as swinging leaves, rain, snow ,and
shadow cast by moving objects.
• Finally, its internal background model should react
quickly to changes in background such as starting
and stopping of vehicles.
12/8/2011 22
23. BACKGROUND SUBTRACTION
ALGORITHMS
Flow diagram of a generic background subtraction algorithm.
12/8/2011 23
24. Background subtraction Algorithms
• The four major steps in a background
subtraction algorithm are:
1. Preprocessing,
2. Background modeling,
3. Foreground detection, and
4. Data validation.
12/8/2011 24
25. Background Modeling
• Background modeling is at the heart of any
background subtraction algorithm.
• Background model is that which robust
against environmental changes in the
background, but sensitive enough to identify
all moving objects of interest.
12/8/2011 25
26. Background Modeling
• We classify background modeling techniques
into two broad categories
– Non-recursive and
– Recursive
12/8/2011 26
27. Non-recursive Techniques
• A non-recursive technique uses a sliding
window approach for background estimation.
• Non-recursive techniques are highly adaptive
as they do not depend on the history beyond
those frames stored in the buffer.
12/8/2011 27
28. Non-recursive Techniques
• Some of the commonly-used non-recursive
techniques are:-
– Frame differencing
– Median filter
– Mean filter
12/8/2011 28
30. Median filter
• The background estimate is defined to be the
median at each pixel location of all the frames
in the buffer.
• The assumption is that the pixel stays in the
background for more than half of the frames
in the buffer.
12/8/2011 30
31. Median Filter
• Assuming that the background is more likely to
appear in a scene, we can use the median of the
previous n frames as the background model:
12/8/2011 31
34. Mean Filter
• In this case the background is the mean of the
previous n frames:
12/8/2011 34
35. For n = 10:
Estimated Background Foreground Mask
For n = 20:
Estimated Background Foreground Mask
12/8/2011 35
36. For n = 50:
Estimated Background Foreground Mask
12/8/2011 36
37. Kalman filter
• Kalman filter is a widely-used recursive
technique for tracking linear dynamical
systems under Gaussian noise. Many different
versions have been proposed for background
modeling, differing mainly in the state spaces
used for tracking.
12/8/2011 37
38. Mixture of Gaussians (MoG)
• Unlike Kalman filter which tracks the evolution of a single
Gaussian, the MoG method tracks multiple Gaussian
distributions simultaneously.
• MoG maintains a density function for each pixel.
• It is capable of handling multi-modal background
distributions.
• Since MoG is parametric, the model parameters can be
adaptively updated without keeping a large buffer of video
frames.
12/8/2011 38
39. Mixture of Gaussians
• the pixel process of
a pixel Z is the set of
the last t values at
the pixel
• {Z0, Z1, ..., Zt−1}
12/8/2011 39
40. Gaussian Mixture Model (GMM)
• A GMM is a mixture
pdf which is a linear
combination of K
Gaussian pdfs.
• wi = 1
• each pixel is given
one GMM
12/8/2011 40
41. Background/Foreground Detection
• There are two types of B/F detection methods:
– Non-adaptive: It depend on certain numbers of video
frames and do not maintain a background model in the
algorithm.
• B/F detection based on two frames
• B/F detection based on three frames
– Adaptive: It maintain a background model and the
parameters of the background model evolve over time.
• Adaptive B/F detection Statistical
• B/F detection based on Gaussian model
12/8/2011 41
42. Drawback of non-adaptive methods
• The non-adaptive methods are useful only in
– high-supervised,
– short-term tracking applications without
significant changes in the video scene.
– When errors happen, it requires manual re-
initialization.
– Without re-initialization, errors in the background
accumulate over time.
12/8/2011 42
43. Adaptive Background/Foreground
Detection
• Adaptive B/F detection is chosen for more and
more VCA applications because of the limitation
of non-adaptive methods. A standard adaptive
B/F detection method maintains a background
model within the system.
12/8/2011 43
44. Video Content Analysis (VCA) is the capability of
automatically analyzing video to detect and determine
temporal events not based on a single image
12/8/2011 44
45. The algorithm of adaptive B/F
detection
• The algorithm of adaptive B/F detection is
described below.
– fi : A pixel in a current frame, where i is the frame
index.
– µ : A pixel of the background model (fi and m are
located at the same location).
– di: Absolute difference between fi and m.
– bi: B/F mask - 0: background. 0 x ff: foreground.
– T: Threshold
– α: Learning rate of the background.
12/8/2011 45
46. Adaptive B/F Detection
1. di = |fi - µ|
2. If di > T , fi belongs to the foreground; otherwise, it
belongs to the background.
Adaptive B/F Detection
12/8/2011 46
47. Examples of Adaptive B/F Detection
Examples of Adaptive B/F Detection
12/8/2011 47