[Partly in Japanese] I present it in 関東コンピュータビジョン勉強会(2015/07/25). Main References: http://crcv.ucf.edu/projects/GMMCP-Tracker/CVPR15_GMMCP_Presentation.pptx If you find a problem, please let me know. Thanks!

1. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
2015 / 7 / 26 (Fri.)
関東コンピュータビジョン勉強会
発表者: @hokkun_cv
GMMCP-Tracker:
Globally Optimal Generalized Maximum
Multi Clique Problem for Multiple Object Tracking
1
Afshin Dehghan, Shayan Modiri Assari, Mubarak Shah
University of Central Florida

2. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
About me
• 東大院・学際情報学府・M2
• 相澤研究室所属
• 普段は食べものの研究をしています
• 2014/5のCV勉強会（CNNについて）ぶりの発表
参加です
2
• Preferred Networksでインターン→アルバイト中
• メンターが@tabe2314さん
• 今日はその課題の中で出てきたタスクに関連する
論文を紹介します

3. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
対象とする問題
• Multiple Object Tracking (MOT)
• YouTubeデモ (GMMCP)
3
※筆者は物体追跡については専門ではないので細かいとこ
ろに誤りがある可能性があります．遠慮無く指摘をお願い
致します．

4. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
ちなみに
• 筆者らはMultiple Object Trackingにかかわる論文
をもうひとつCVPR2015で発表している（強い）
• Target Identity-aware Network Flow for Online Multiple
Target Tracking
• 筆頭著者も一緒(Ph.Dの学生，ちなみに去年も2本筆頭で発表．強い）
4

5. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
その他
• H. Possegger et al., In Defense of Color-based
Model-free Tracking
• モデルフリートラッキング（非detection based)
• T. Liu et al., Real-time part-based visual tracking
via adaptive correlation filters
• パートベースのトラッキング
• S. Tang et al., Subgraph Decomposition for Multi-
Target Tracking
5

6. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Data Association (Naïvest)
6
Frame n Frame n+1
Bipartite
Matching
Problem

7. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Tracking
7
Detection
Data
Association
http://crcv.ucf.edu/projects/GMMCP-Tracker/CVPR15_GMMCP_Presentation.pptx

8. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Tracking
8
Detection
Data
Association
http://crcv.ucf.edu/projects/GMMCP-Tracker/CVPR15_GMMCP_Presentation.pptx

9. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Data Association (Naïvest)
9
Frame n Frame n+1
Bipartite
Matching
Problem

10. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Data Association (Naïvest)
10
Frame n Frame n+1
Bipartite
Matching
Problem

11. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Data Association (Network Flow)
11
Frame n Frame n+1 Frame n+2 Frame n+3
sources
sinks
minimum-cost
maximum-flow
problem
• incorporating
motion feature
• multi-commodity
network

12. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
12
Frame
1
Frame
2
Frame
3

13. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
However,
• Data association with network flow is simplified
formulation of this problem
• Assuming no simplification is closer to the
tracking scenario in real world.
13

14. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Data Association (Not Simplify)
14
Frame n
Frame n+1
Frame n+2
Frame n+3
重み
=
0.95
重み
=
0.10
うまいこと重みが最大
になるクリークを探す

15. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Preliminary: clique (クリーク)
• 任意の2点を結ぶ枝がある頂点集合のこと
• see wikipedia in detail
• 今回は「各クラスタから1つのノードを選んでで
きる部分グラフ」という理解でOK
15

16. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Data Association (Not Simplify)
16
Frame n
Frame n+1
Frame n+2
Frame n+3
Input: k-partite complete
graph (完全k部グラフ)
A person form a clique
↓
maximum clique
problem

17. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
GMCP Tracker[1]
• The same team s ECCV 2012 paper
• They formulate MOT as generalized maximum
clique problem. (cf. former page)
17[1] Amir Roshan Zamir et al., GMCP-Tracker: Global Multi-object Tracking Using Generalized Minimum Clique Graphs, ECCV, 2012.

18. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
However (2),
• Due to complexity of the model, these
approaches have been solved by approximate
solutions.
• GMCP Tracker also used a greedy local
neighborhood search, which is prone to local
minima.
• GMCP Tracker doesn t follow a joint optimization
for all the tracks simultaneously (one by one).
18

19. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Contribution
1. this approach doesn t involve any simplification
neither in formulation nor in optimization
(Binary Integer Problem).
2. they propose a more efficient occlusion
handling strategy, which can handle long-term
occlusions (e.g. 150 frames) and can speed-up
the whole algorithm.
19

20. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Contribution
1. this approach doesn t involve any simplification
neither in formulation nor in optimization
(Binary Integer Problem).
2. they propose a more efficient occlusion
handling strategy, which can handle long-term
occlusions (e.g. 150 frames) and can speed-up
the whole algorithm.
20

21. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
21
Low-level Tracklets
Segment 01 Segment 05
Segment 06 Segment 10
Mid-level Tracklets
Final Trajectories
GMMCP GMMCP
Input Video
Human
Detection
Detected Humans

22. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
22
Low-level Tracklets
Segment 01 Segment 05
Segment 06 Segment 10
Mid-level Tracklets
Final Trajectories
GMMCP GMMCP
Input Video
Human
Detection
Detected Humans

23. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Step 0: Low-level Tracklet
• In GMCP, the nodes at first step are each
detections.
23
Frames
1-­‐10
• In GMMCP, the nodes are (low-level) tracklet
• How to find: bounding boxes that overlap more than
60% between two frames are regarded as being
connected.

24. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
24
Low-level Tracklets
Segment 01 Segment 05
Segment 06 Segment 10
Mid-level Tracklets
Final Trajectories
GMMCP GMMCP
Input Video
Human
Detection
Detected Humans

25. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Step 1: Mid-level Tracklet
25
• 各クラスタ（青円）からひとつのノード（赤線）
を選び，クリークを作る
Frames
1-­‐10
Frames
11-­‐20
Frames
21-­‐30
Frames
31-­‐40
Frames
41-­‐50
Frames
51-­‐60

26. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Step 1: Mid-level Tracklet
26
• エッジの重み = (見た目特徴) + (動き特徴)
• これを基に最適化をすると・・
Frames
1-­‐10
Frames
11-­‐20
Frames
21-­‐30
Frames
31-­‐40
Frames
41-­‐50
Frames
51-­‐60

27. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Step 1: Mid-level Tracklet
27
• このような三人の軌跡が同時に検出できる
• オクルージョンに対応するため，ダミーノードを
入れてある
Frames
1-­‐10
Frames
11-­‐20
Frames
21-­‐30
Frames
31-­‐40
Frames
41-­‐50
Frames
51-­‐60

28. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
28
Low-level Tracklets
Segment 01 Segment 05
Segment 06 Segment 10
Mid-level Tracklets
Final Trajectories
GMMCP GMMCP
Input Video
Human
Detection
Detected Humans

29. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Step 2: Final Trajectories
• The another but similar problem with step 1.
• They solve GMMCP:
• Nodes are Mid-level Tracklet
• For appearance feature, they use median (or average)
feature among detections in each frame
• For motion feature, they use middle point of mid-level
tracklet as the location of each node
29

30. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Appearance Affinity
• Feature: Invariant Color Histogram [2]
• Deformation and viewpoint invariant
• Affinity: Histogram Intersection
30[1] J. Domke et al., Deformation and Viewpoint Invariant Color Histogram, BMVC, 2006
min(H1[i], H2[i])

31. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Motion Affinity
31[1] J. Domke et al., Deformation and Viewpoint Invariant Color Histogram, BMVC, 2006
今の位置
前の位置＋速度度から
予想される位置

32. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Optimization
• GMMCP is NP Hard, but they solve without any
simplification.
• They formulate GMMCP as Binary Integer
Problem (BIP, 0-1整数計画問題)
32

33. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
33http://www.dais.is.tohoku.ac.jp/ shioura/teaching/dais08/dais02.pdf

34. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
34http://www.dais.is.tohoku.ac.jp/ shioura/teaching/dais08/dais02.pdf

35. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Optimization
• GMMCP is NP Hard, but they solve without any
simplification.
• They formulate GMMCP as Binary Integer
Problem (BIP, 0-1整数計画問題)
35
• これは実は組合せ最適化と言われる問題
• cf. 0-1ナップザック問題，巡回セールスマン問
題

36. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
BIP in this case
• C is weight matrix (?)
• x is boolean column vector
• the elements of x is all of edges and nodes
• Ax = b is equality constraints
• Mx <= n is inequality constraints
36

37. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
37
各クラスタごとに1に
なってるのは定数K
Notation
: i th node in j th cluster
: edge between and h: Number of clusterseij
mn
vm
n
vi
j
vi
j
あるノードから伸び
るエッジはh-1(かゼ
ロ）
クリークを作ってい
るかどうか
３種の制約

38. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Contribution
1. this approach doesn t involve any simplification
neither in formulation nor in optimization
(Binary Integer Problem).
2. they propose a more efficient occlusion
handling strategy, which can handle long-term
occlusions (e.g. 150 frames) and can speed-up
the whole algorithm.
38

39. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Occlusion Handling
• Detector can detect not all the persons in each
frame
• Occlusion, Detection Error, …
• They add Dummy Node to each cluster
• Cost of dummy edge ( = edge connected to
dummy node) is fixed value.
39

40. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Occlusion Handling
40
= さっきまで出てきてた重み ( 見た目 + 動き )cj1
cj2 = 定数c_d
cj3 , cj4 = 0

41. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Occlusion Handling
• How/How many do we add dummy nodes?
• Many dummy nodes increase computational
complexity
• cf. case of GMCP:
• They add dummy node by the motion-based way
• ある答えに対して等速度運動を仮定して，大きくハズレ
てしまうようなクラスタにダミーノードを足す
• Many dummy nodes increase computational
complexity (大事なので2度)
41

42. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Occlusion Handling
• Aggregated Dummy Nodes (ADN)
• no longer be boolean variable
• can take any integer value
• add only one ADN to each cluster
• Not connected to other nodes!
• New Solution: Mixed-Binary-Integer Programming
42
Constraint 1 Constraint 2 Constraint 3
各クラスタごとに1に
なってるのは定数K
あるクラスタから伸
びるエッジは1か0
クリークを作ってい
るかどうか

43. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Occlusion Handling
43
cj1
cj2 はない (
cj3 , cj4= 0
cd
2
cj3 , cj4 =
= さっきまで出てきてた重み ( 見た目 + 動き )

44. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
ここからひたすら結果
44

45. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Metrics
45

46. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
46
Dataset Method MOTA MOTP MT ML IDS
TownCenter
MPT 72.9 71.3 - - -
GMCP 75.59 71.93 - - -
Ours 77.37 66.38 86.09 4.35 68
TUDCrossing
MWIS
85.9 73 - - 2
GMCP 91.63 75.6 - - 0
Ours 91.9 70 75 0 2
TUDStadmitte
DLP 79.3 73.9 - - 4
GMCP 77.7 63.4 - - 0
Ours 82.4 73.9 80 0 0
Parking
Lot1
H2T 88.4 81.9 78.57 0 21
GMCP 90.43 74.1 - - -
Ours 92.9 73.6 92.86 0 4

47. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
47
Dataset Method MOTA MOTP MT ML IDS
ParkingLot2
KSP 45.4 57.8 46.15 0 531
DCT 60.1 56.1 76.92 0 234
CMOT 80.7 58 84.62 0 61
GMCP 75.6 58.1 61.54 0 76
IHTLS 78.8 57.9 84.62 0 50
Ours 87.6 58.1 92.31 0 7
Pizza
KSP 51.8 65.7 39.13 0 249
DCT 53.5 65.8 69.57 0 185
IHTLS 57.6 66.8 43.48 4.35 105
CMOT 56.9 63.3 30.43 4.35 87
GMCP 57.6 68.6 26.9 4.35 52
Ours 59.5 64.1 30.43 0 55

48. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
時間的評価
48

49. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
時間的評価
49

50. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
時間的評価
50

51.
GMMCP
Tracker:
Globally
Op3mal
Generalized
Maximum
Mul3
Clique
Problem
for
Mul3ple
Object
Tracking
TUD-Stadmitte
Mid-­‐level
Tracklets
Final
Trajectories

52.
GMMCP
Tracker:
Globally
Op3mal
Generalized
Maximum
Mul3
Clique
Problem
for
Mul3ple
Object
Tracking

53.
GMMCP
Tracker:
Globally
Op3mal
Generalized
Maximum
Mul3
Clique
Problem
for
Mul3ple
Object
Tracking

54. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
まとめ (拝借)
• Formulate MOT as GMMCP
• a new graph theoretic problem
• Formulate GMMCP as a MBIP
• GMMCP is NP Hard but no approximate solutions
• An efficient occlusion handling through AND
• Performance close to real-time
• Improving state-of-art on several sequences

55. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
project page
• 本物のproject pageが情報量多くてこれがCVPR複
数本2年連続で通す人のページか，と思いました
• http://crcv.ucf.edu/projects/GMMCP-Tracker/
55

56. GMMCP-Tracker: Globally Optimal Generalized Maximum Multi Clique Problem for Multiple Object Tracking
Implementation Detail
• Detection: DPM
• K: target-specific
• (1st layer ) number of cluster: 5 (2-6で実験)
• (2nd layer) number of cluster: 6
56