Presentation of Visual Tracking

Presentation of Personal Work
Yosen Chen

Yosen Chen, Copyright Reserved
Human Machine Interaction
Basic Visual Tracker
Multi-Cue Particle Filter Tracker
Consistent Labeling in Visual Surveillance
System
Multi-Human Tracker
Multi-Camera Multi-Human Tracker
Consistent Labeling with People Re-Entering
Problem
Outline
2

Implementation Tool
Microsoft Visual C++ with OpenCV Library
IPC (Interprocess Communication) Library
open source @ Carnegie Mellon University
Yosen Chen, Copyright Reserved 3

System
Problem
Outline
4

Intelligent Interactive Engine
Image processing
Webcam
Image data
Show message
& animation
Display Central
Server
Webcam
Our goal is to construct an interactive engine which utilizes
target motion as instruction input to the server.

System
Problem
Outline
6

Basic Visual Tracking Framework
(Color)
Camera Real-time
image
Feature Extraction
RGB
to
HSV
Hue
Saturation
Value
256-to-16
color Qtz
Similarity Evaluation
Reference
Image
Reference
16-color Histogram
Hue
Saturation
Value
Reference Color
Backprojection Probability
distribution
figure
Probability Peak
Search/Climbing
Target Information
Estimation
(Position, size,...)
If iterative
Reference
16-color Histogram
Reference Color
Backprojection
Input Hue
Image
Probability
distribution
figure
Color
similarity
score
Iterative Search
(Motion Prediction)
Previous
Current
Next
Camera View
Region of Interest
7

Demo Video: Ball Tracking
追球後段
Peak climbing: MeanShift

Related Works
Varieties of Feature Extraction/Evaluation
Feature evaluation: template matching
Background subtraction
Reference color model: YCrCb, RGB
Varieties of Peak Search/Climbing
Gravity center
Max-Likelihood
MeanShift
Cb
Cr
9

Summary
Advantages:
Simple & Fast  low-cost
Each pixel (RGBHSV, backprojection) is processed
independently  good for parallel processing
Drawbacks:
Hard to combine with more object features
 EX: edge information (object shape, contour)
 All object features 2D probability distribution figure
Hard to cope with higher-dimension target state
 EX: object orientation (rotation angle)
??suitable
10

System
Problem
Outline
11

SIR+SIS Particle Filter Tracking
State candidates
(Particle set) Color
Possible region
sampling
SIR
Real-time
State Candidate
Prediction
Frame
Multi-Cue Candidate
Evaluation
   
 1
m m
t tp X X 
 
  
1 1
, , ,
1,...,
mm
t t
X x y r
m N
 

 
 m
t tp O X State Candidate
Resampling
   
  
 
 
,
equal weight
m m
t t t
k
t
X p O X
X
tO
SIS
Shape
Motion
weighting
Particles from last candidate set
Particles generated
from current image
Tracking result
on Frame
tO
 m
tX
Best State Candidate
Sequential Importance Sampling
(SIS) Particle Filter
Sampling Importance Resampling
(SIR) Particle Filter
Color backproject
+ filtering
+ best contour finding
Initial State Estimation State Prediction State Evaluation 
Find Best Candidate
Resampling Last Candidates
based on weights
State Prediction State Evaluation 
Find Best Candidate
Last candidate set
{state candidate, weight}
12

Multi-Cue Likelihood Evaluation
For each state
candidate
Inner color extraction
Color
Similarity
Correlation-based
comparison
Color weight
Reference color modelColor
Multi-Cue Candidate
Evaluation
 
 m
t tp O XShape
Motion
weighting
 m
tX
tO
inner
outer
High
continuity
Low
continuity
id
maxd
ig
 
  
, , ,
mm
t t
X x y r 
Shape
Similarity
Motion
continuity
(logic check)
outer color extraction
Shape distance
State difference with
last tracking result
id
 Ih i
 Oh i
 REFh i
Edge strength ig
Max matched size 1 null
MAX
Nr
r N
 
  
 
Symmetry 1i N id d  
, , ,x y r    
Shape weight
Motion weight

Overall
candidate score
 
 m
t tp O X
13

Demo Video: 4D Bottle Tracking
(x, y, r, θ)
Candidate group behavior
Bottle tracking video
Blue: SIS (from current color projection)
Green: SIR (from last candidate set)
Red: Best candidate as tracking result

Interaction
System
Interaction
Triggering
Object
Checking
Tracking
& Interaction
Losing
Handling
Start
Motion
detection by
Optical flow
Image
Initialize/Reset
all parameters
Motion
detected?
Show Checking
Box
Record target
information and
features
Target Info is
robust?
Tracking
algorithm
Quality index
> threshold ?
Wait until user
puts target in
“Losing Box”
Is target put
back?
Visual reality
interacting
program
image
Yes
No Yes
Yes
No
No
Yes
No
Image

Demo Video:
Interactive Game – Brick Breaker
Interaction system: 4 status switching
4 degrees of freedom: location x & y, size, rotation angle
16
Motion Interaction Function Interactive Game Example

Summary
High flexibility
State dimension
Motion model
Easy to add new features & tune weights in tracker
for different applications and scenarios
Trade-off:
Feature description vs. Computation time
Impoverishment(intensive) vs. Degeneracy(sparse)
 Many particles concentrate on high-probability region
 Many particles are evaluated at low-probability region
 Number of particles ～ Complexity

System
Problem
Outline
18

Related applications:
Visual surveillance
Remote monitoring
Intelligent security system
Introduction
19Yosen Chen, Copyright Reserved

System
Problem
Outline
20

Multi-Human Tracking Flow
Prediction Evaluation Resampling
…
State joint evaluation
SIR Head tracking
Multi-human
tracking result
Real-time
captured image
Skin color
blob finding
SIS Face detection
Shape &
location filter
ROI location
Sampling
Face detector
(OpenCV)
Losing Track
Detector
(score,
counter)
Delete tracker
Create new tracker
1. SIS face detector  SIR human trackers
2. Multi-human state joint evaluation
3. Bad evaluated tracker  losing track
detector  face detector  delete tracker
21

Multi-Cue Likelihood Evaluation
for Head Tracking
High color likelihood Low color likelihood
outer
inner
Skin color mask
Hair color mask
outer
inner
Skin color mask
Hair color mask
  1
, , t
x y r 
 , , t
x y r
  1
, , t
x y r 
 , , t
x y r
High continuity Low continuity
1n
2n
6n
9n
13n
Shape distance
1n
2n
6n
9n
13n
Num of Matched point
ig
Edge strength 1n 2n
6n
9n
13n
symmetry
Color
•Masked skin color comparison
•Masked hair color comparison
Texture
•Texture SAD similarity
Edge
•Shape distance
•Number of matched point
•Edge strength
•Contour symmetry fitting
Motion
•State continuity
22

Face Detection Implementation Details
SIS Face Detection Flow
=======================Pseudo code========================
cvCalcBackProject;
cvEqualizeHist + cvThreshold;
cvResize + cvErode + cvDilate;
cvFindContours(LinkedList detect_contour);
for( ; detect_contour != 0 ; detect_contour = detect_contour->next )
{
if (Area < threshold) continue;
Use 8 points average to approximate center (xA, yA);
if (distance between all target T and (xA, yA) > threshold)
{
calculate geometric center (xC, yC), deviation σX, σY of detect_contour;
if ((σX > 0.7*σY) && (σY > 0.8*σX) &&
(σX /sqrt(Area) < 0.3) && (σY /sqrt(Area) < 0.4))
{
Draw k random samples (x, y) ~ N((xC, yC), (σX, σY));
where k = (Area/MIN_AREA).
if (frontalFaceTest(x, y)->detect() || profileFaceTest(x, y)->detect())
return location (x, y) as start point of new tracker;
}
}
}
return NOT_DETECTED;
Skin color
blob finding
SIS Face detection
Shape &
location filter
ROI location
Sampling
Face detector
(OpenCV)
Losing Track
Detector
Create tracker
Delete tracker
23

Face Detection Demo
Skin color backproject
Skin color
blob finding
SIS Face detection
Shape &
location filter
ROI location
Sampling
Face detector
(OpenCV)
Losing Track
Detector
Create tracker
Delete tracker
24Skin color backproject (down-size & morphology)
Last detecting region
Blue region: Last detected face
Red region: color is matched with skin color.
Tracking System GUI panel

People Overlapping Problem
When people get close to each other…
State candidates are attracted by other people
because trackers don’t know each other!!
Prediction
Evaluation
(individual)
Resampling
Original Head Trackers
25

Solutions of People Overlapping
Prediction
Evaluation
(Individual
+ joint)
ResamplingJoint state
check
Depth order
check
Body occlusion
handling
Target information
storage
Modified Head Trackers for People Overlapping
For people re-entering
problem
1. Use joint evaluation to avoid candidates merging for people overlapping
2. Solution#1 (left): give higher joint score for candidates further from other people 
reduce risk of candidates merging but also lose tracking accuracy
3. Solution#2 (right): kill candidates too close to other people  better tracking accuracy
26
resVideo_FurtherIsGood2.mpg resVideo_CloseThenLowWeight2.mpg

Implementation
Prediction
Evaluation
(Individual
+ joint)
check
Depth order
check
Image distance check “if △x2 + △y2 < threshold, …” è group together
target1 target2 target3 target4 target5
Grouping (2nd-order tree)
root leaf leafleaf
Target A
candidate 1
(far from B)
Target B
last best
candidate
Target A
candidate 2
(near B)
Pointer to
target1
Candidate Evaluation (individual + joint)
Individual: (as previous mentioned)
Color, shape, motion
Joint: (not be attracted by other close targets)
Distances to all other targets in the same group
Pointer to
target2
………
Depth order: deform 2nd-tree into array, then quick sort by depth features
Sort by depth
features
Depth features:
For best candidates of each target
Texture changing rate
Contour matching score
Distance from camera to target ground points
Image source
1:Front
2:Rear
Candidate A2 has high penalty
(since Candidate A2 are too close to B !!)
27

Depth Order Evaluation with Heights
Depth order estimation with height knowledge is accurate
Even when people are not overlapping.

Summary of Multi-Human Tracker
We extend the concept of single-object tracking
to multi-object tracker
We propose a entire tracking system
Face detection  multi-human trackers  detect
people leaving
Address the solution of people overlapping
Robust multi-human tracker is indeed the basis
of correspondence process within multi-camera
tracking system

System
Problem
Outline
30

Multi-Camera Tracking System
Captured image
Local camera #A
multi-head tracker
 1 2,A B
t tz z2tH
1
A
tz 2
A
tz
2
B
tz
1
B
tz
Local camera #B
multi-head tracker
Captured image
Central Server
Multi-camera multi-observation
correspondence
Person#1 Person#2
 1 2 1,A B
t tz ztH
Identify the relationship of observations
in different cameras
Local trackers: multi-human tracking and occlusion handling
Central server: observation correspondence across cameras
31

: image observation
: 3D line from image head center
j
rt
j
rt
O
L
 “Are these cameras tracking the same people? ”
 Let be the rth observation tracked by camera at time
Problem of Observation Correspondence
across Cameras
 ,j j j
rt rt rtO Lz j t
How many
people exist in
surveillance
area?
Ground plane
Camera j
Camera k
1
j
tO
2
j
tO
3
j
tO1
k
tO
2
k
tO
3
k
tO
1
j
tL
2
j
tL
3
j
tL
1
k
tL
2
k
tL
3
k
tL

 Determine observation correspondence using Multi-Feature Similarity
 Multi-Feature Observation Similarity contains:
Head line intersection verification
Body color matching by blocks
Confidence level
Label history consistency
How to Determine Observation
Correspondence across Cameras
Correspondence Flow
Calculate 3D
geometric intersection
Confidence
level
Are they from the
same person?
Camera#A Camera#B
Body Color
comparison by block
Label history
consistency
2D Body image
block allocation
No, if no reasonable
intersection
No, if color unmatched
Yes, 2 observations
are from the same
person

 2D Position on image plane  3D head line in real space
 3D ground point in real space  2D position on image plane
observation r
Optic center Image plane of
camera j
Ground plane
Optic center
observation q
Image plane of
camera k
3D Geometric Correspondence across
Cameras
Calibrated camera system
Image 2D↔3D coordinate transfer
is constructed off-line
Head Point
height
Estimated Ground Point

How to Implement 3D-2D Mapping
Class: Plane3D
Member functions:
isPointOnPlane
getDistFromPoint
getProjPointFromPoint
Class: line3D
Member functions:
getPositionFromX,Y,Z
isPointOnLine
getDistFromPoint
getProjPointFromPoint
getLinePointOnPlane
getMinDistPointFromLine
Class: CameraLine3D
Member functions:
setByImageLocation
setBy3DPosition
How to verify the
mapping accuracy?
3D Optic point
(≈ Camera location)
Class: CameraLine3D
2D image
plane
Reference plane
in 3D space
3D Normal vector
Reference 3D
plane center
(0,0)
derived
35
CAM2 CAM1
Locate 3D points/lines on 2D image planes

 Human Body Model Based on Head Tracking
Matching body color
 Human Body is separated into
4 blocks
ground-point
head
ground-point
head
Image
boundary
Observed ground-point Unobserved ground-point
j
rtz k
qtz
Camera kCamera j
qtrt
3D Geometric-based Body Image Block
Allocation
For some observations, not all the body blocks are unobservable.
Here we take the number of observable blocks as confidence level of correspondence result

Observation Correspondence across Cameras
Object Conf = 3
Object Conf = 2
Matching body color (Region-based body color comparison)
Depth order:
Front
Depth order:
RearBody Occlusion Handling (based on depth order and 3D-2D block mapping )
Body appearance is available for
correspondence process
Body appearance is not
available (Color turns darker)

Body Occlusion Handling
Prediction
Evaluation
(Individual
+ joint)
check
Depth order
check
Body occlusion
handling
Target information
storage

Labeling Consistency with Confidence Level
Observation correspondence process should follow rules stated below
Rule1. Claiming “match” for one pair will also imply claiming “collision” for all
corresponding residual pairs, with the same confidence level
Rule2. Only higher-confidence decisions can enforce changes of lower-confidence ones
√ √ √
A B C
√ 1 0 0 -4
√ 2 -4 -4 4
√ 3 0 0 -4
√ √ X
A B C
√ 1 0 0 0
√ 2 -4 -4 0
√ 3 0 0 0
√ √
A B
√ 1 -3 0
√ 2 -4 -4
√ 3 3 -3
Camera#1
Camera#2
Conf = 4Conf = -4
1 2
3
A B
C
Camera#1
Camera#2
Conf = -4
1 2
3
A B
Camera#1
Camera#2
Conf = 3
Conf = -4
Conf = -3
1 2
3
A B 39

System
Problem
Outline
40

Consistent Labeling with People Re-Entering
Captured image
Local camera #A
multi-head tracker
 1 2,A B
t tz z2tH
1
A
tz 2
A
tz
2
B
tz
1
B
tz
Local camera #B
multi-head tracker
Captured image
Central Server
Multi-camera multi-observation
correspondence
Person#1 Person#2
 1 2 1,A B
t tz ztH
Identify the relationship of observations
in different cameras
3tX2tX1tX
associate with target databases
41

Association Processes with Matching Confidence Orders
Hierarchical Association with target databases
 Objects is associated to target databases  1
m
ht h
H   1
n
it i
X 
Consistent Labeling Using Target Database
3tX
2tX
1tX
4tX
1tH
i
TARL
High
Low
1
OBJL
1
TARL
2
TARL
3
TARL
4
TARL
compare with
descendent order
: target confidence level (in database)
: object confidence level (from observation)
i
TAR
i
OBJ
L
L
if association success
update related database
else
create new database
PS: database association process is
only for objects with conf. level ≧2

Association Process
 Association process:
 To determine if observed object and database are from the same
person
 Features are similar to observation correspondence process
 People height
 Body color comparison by blocks
 Association history consistency
Observation-Database Association Flow:
Check height
difference
Are they from the same person?
Body Color
comparison by block
Association history
consistency
No, if diff > threshold No, if color unmatched
Yes, observations &
this database set are
from the same person
2tX
1tH
From Camera
observations
People information
already stored in
databases 43

How to Model & Update Database
Adaptive Gaussian Mixture Model
3 color candidates (histogram & weight) to describe 1 body block
color
Candidate’s histograms & weights are updated based on current
color of MATCHED observations
Color comparison with observed color (Define MATCHED or not)
 Bhattacharyya distance from 3 candidates
 3 weighted Gaussian kernels
Color histogram (1-D for demo)
Candidate
weight
Candidate color
Adaptive Gaussian Mixture Model
Candidate#1 Candidate#3 Candidate#2
Candidate color & weight will be updated based on observations
Candidate
variance
Inspired by
[Stauffer., CVPR 2000]

Database Operation Demonstration
Histogram candidatesCamera#2 histogramCamera#1 histogram
Matched with camera#2
Matched with camera#1

Database Demonstration (cont’)
46Yosen Chen, Copyright Reserved

Multi-People Correspondence
1. Multi-people depth order
2. body occlusion handling
3. Multi-camera multi-people correspondence is correct

Multi-People Re-entering
System Operation Flow:
1. Observation correspondence & Body occlusion handling
2. Associate with current databases  (Create then) update new information
3. Return people IDs (show by color) to each tracker of each camera

Future Works…
Extend to the 3rd Camera
Challenge of communication load, algorithm complexity, and
tracking/labeling stability
Performance Evaluation of Target Database
Comparison with related works
More challenges:
Outdoors Surveillance (human body detection)
Uncalibrated multi-camera system (Training)
Motive camera (rotation 3D-2D mapping)

BACKUP SLIDES

Motion Detection : Optical Flow
(Modified from OpenCV Sample Code)
Input gray-scale
image
Optical point
extraction
Optical point
mapping & filtering
Motion
detection criteria
Reset criteria
Interaction
trigger
Optical flow
Resettrigger
Previous gray-scale
image
Store
Optical Flow:
1. A technique to create edge points then map them between adjacent images.
(edge tracking)
2. Better to analyze group behavior rather than individual (not accurate)
3. Can be utilized as motion detection tool
1. Not enough effective optical points on screen
2. Time out then refresh
1. Average velocity of moving points > threshold
2. Number of moving points > threshold
Moving points:
points with velocity > 3*average velocity

BACKUP SLIDES END

Presentation of Visual Tracking

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (10)

Similar to Presentation of Visual Tracking

Similar to Presentation of Visual Tracking (20)

Recently uploaded

Recently uploaded (20)

Presentation of Visual Tracking