4. Background: Deformable Parts Model
• Strong low-level features based on
histograms of oriented gradients (HOG)
• Efficient matching algorithms for deformable part-
based models (pictorial structures)
• Discriminative learning with latent variables (latent
SVM)
• Where to look? Every where (the sliding window
approach)
• mean Average Precision (mAP): 33.7% - 33.4%
P.F. Felzenszwalb et al., “Object Detection with Discriminatively Trained Part-Based Models”, PAMI 2010.
J.J. Lim et al., “Sketch Tokens: A Learned Mid-level Representation for Contour and Object Detection”, CVPR 2013.
X. Ren et al., “Histograms of Sparse Codes for Object Detection”, CVPR 2013.
5. Background: Selective search
• Alternative to exhaustive search
with sliding window.
• Starting with over-segmentation,
merge similar regions and produce region
proposals.
van de Sande et al., “Segmentation as Selective Search for Object Recognition”, ICCV 2011.
6. Deep Learning happened, again!
Krizhevsky et al., “ImageNet Classification with Deep Convolutional Neural Networks”, NIPS 2012.
ImageNet 2012 :whole-image classification with 1000 categories
Model Top-1(val) Top-5(val) Top-5(test)
1 CNN 40.7% 18.2% -
5 CNNs 38.1% 16.4% 16.4%
1 CNN (pre-trained) 39.0% 16.6% -
7 CNNs (pre-trained) 36.7% 15.4% 15.3%
• Can it be used in object recognition?
• Problems:
• localization: Where is the object?
• annotation: Labeled data is scarce.
• Expensive Computation for dense
search.
7. R-CNN: Region proposals + CNN
localization featureextraction classification
Approach Summery selective search deep learning
CNN
binary linear SVM
13. ConvNet
ConvNet
ConvNet
SVMs
Warped image regions
Forward each region
through ConvNet
Bbox reg
Bbox reg
Bbox reg SVMs
SVMs
Apply boundingboxregressors
Classify regions withSVMs
Regions of Interest (RoI)
from a proposal method
(~2k)
Input image
R-CNN
Girshick et al. CVPR14.
15. • Ad hoc training objectives
• Fine-tune network with softmax classifier (log loss)
• Train post-hoc linear SVMs (hingeloss)
• Train post-hoc bounding-box regressors (squaredloss)
What’s wrong with R-CNN?
16. • Ad hoc training objectives
• FineHtunenetwork with softmax classifier (log loss)
• Train postHhoclinear SVMs (hingeloss)
• Train postHhocboundingHbox regressors (squaredloss)
• Training is slow (84h), takes a lot of disk space
What’s wrong with R-CNN?
17. • Ad hoc training objectives
• FineHtune network with softmax classifier (log loss)
• Train postHhoclinear SVMs (hingeloss)
• Train postHhocboundingHboxregressions (least squares)
• Training is slow (84h), takes a lot of disk space
• Inference (detection) is slow
• 47s / image with VGG16 [Simonyan & Zisserman. ICLR15]
• Fixed by SPP-net[He et al. ECCV14]
~2000 ConvNet forward passes per image
What’s wrong with R-CNN?
20. ConvNet
Input image
Forward whole image through ConvNet
“conv5” feature map of imageRegions of
Interest (RoIs)
from a proposal
method
SPP-net
He et al. ECCV14.
21. ConvNet
Input image
Forward whole image through ConvNet
“conv5” feature map of imageRegions of
Interest (RoIs)
from a proposal
method
Spatial Pyramid Pooling (SPP) layer
SPP-net
He et al. ECCV14.
22. Input image
Regions of
Interest (RoIs)
from a proposal
method
ConvNet
SVMs Classify regions withSVMs
FullyHconnected layers
Spatial Pyramid Pooling (SPP) layer
“conv5” feature map of image
Forward whole image through ConvNet
FCs
SPP-net
He et al. ECCV14.
23. Input image
Regions of
Interest (RoIs)
from a proposal
method
ConvNet
SVMs Classify regions withSVMs
FullyHconnected layers
Spatial Pyramid Pooling (SPP) layer
“conv5” feature map of image
Forward whole image through ConvNet
FCs
Bbox reg
Apply boundingbox regressorsSPP-net
He et al. ECCV14.
24. What’s good about SPP-net?
• Fixes one issue with R-CNN:makes testing fast
ConvNet
SVMs
FCs
Bbox reg
Region-wise
computation
Image-wise
computation
(shared)
25. What’s wrong with SPP-net?
• Inherits the rest of R-CNN’sproblems
• Ad hoc trainingobjectives
• Training is slow (25h), takes a lot of disk space
• Introduces a new problem: cannot update
parameters below SPP layer during training
26. SPP-net: the main limitation
ConvNet
He et al. ECCV14.
SVMs
Trainable
(3 layers)
Frozen
(13 layers)
FCs
Bbox reg
SPPisnotdifferentiable
28. Fast R-CNN
• Fast test-time,like SPP-net
• One network, trained in one stage
29. Fast R-CNN
• Fast test-time,like SPP-net
• One network, trained in one stage
• Higher mean average precision than R-CNN and SPP-net
30. Fast R-CNN (test time)
ConvNet
Forward whole image through ConvNet
“conv5” feature map of imageRegions of
Interest (RoIs)
from a proposal
method
Input image
31. ConvNet
Forward whole image through ConvNet
“conv5” feature map of image
“RoI Pooling” (singleHlevel SPP) layer
Input image
Regions of
Interest (RoIs)
from a proposal
method
Fast R-CNN (test time)
32. Linear +
softmax
FCs FullyHconnected layers
“RoI Pooling” (singleHlevel SPP) layer
“conv5” feature map of image
Forward whole image through ConvNet
Input image
Softmax classifier
Regions of
Interest (RoIs)
from a proposal
method
ConvNet
Fast R-CNN (test time)
33. ConvNet
Forward whole image through ConvNet
“conv5” feature map of image
“RoI Pooling” (single-level SPP) layer
Linear +
softmax
FCs FullyHconnected layers
Softmax classifier
Regions of
Interest (RoIs)
from a proposal
method
Linear
Input image
Bounding-box regressors
Fast R-CNN (test time)
35. Log loss + smooth L1 loss
Linear +
softmax
FCs
Linear
ConvNet
Multi-taskloss
Fast R-CNN (training)
36. Log loss + smooth L1 loss
Linear +
softmax
FCs
Linear
Trainable
Multi-taskloss
ConvNet
Fast R-CNN (training)
37. What is missing from the previous
architectures?
• All the previous architectures relies on an external region
proposal algorithm.
• Proposed regions are independent from the network loss.
• No control over the regions quality.
39. Faster R-CNN
• Fast test-time,like FastR-CNN
• One network, trained in one stage
40. • Fast test-time,like FastR-CNN
• One network, trained in one stage
• Higher mean average precision than R-CNN,SPP-net,
Fast-RCNN
Faster R-CNN
41. • Fast test-time,like FastR-CNN
• One network, trained in one stage
• Higher mean average precision than R-CNN , SPP-
net, Fast-RCNN
• HaveadedicatedRegionProposalNetwork(RPN)trainedto
optimizethenetworkloss.
Faster R-CNN
51. Problem definition
• All region based CNN object detector are dependent on the quality of
the region proposal algorithm.
• Although in the Faster R-CNN, the region proposal network was trained
to minimize a multi-task loss function (log-loss and bounding-box
regression), still ,in my experiments, the best proposed regions are ill-
localized.
57. Better regions with Bayesian
Optimization
Now the goal becomes sampling new solution 𝑦 𝑛+1 with
high chance that it will maximizes the value of 𝑓𝑛+1
58. Better regions with Bayesian
Optimization
Given the ability to query a our CNN for region scores
we can repeat the following:
59. 1. Given existing regions/scores •
Better regions with Bayesian
Optimization
Given the ability to query a our CNN for region scores
we can repeat the following:
60. 1. Given existing regions/scores •
2. Wefit a model
Given the ability to query a our CNN for region scores
we can repeat the following:
Better regions with Bayesian
Optimization
61. 1. Given existing regions/scores •
2. Wefit a model
3. Introduce the chanceutility function
Given the ability to query a our CNN for region scores
we can repeat the following:
Better regions with Bayesian
Optimization
62. 1. Given existing regions/scores •
2. Wefit a model
3. Introduce the chanceutility function
4. Locatethe maximum of the utility
Given the ability to query a our CNN for region scores
we can repeat the following:
Better regions with Bayesian
Optimization
63. 1. Given existing regions/scores •
2. Wefit a model
3. Introduce the chanceutility function
4. Locatethe maximum of the utility
5. Observe the new regionscore
Given the ability to query a our CNN for region scores
we can repeat the following:
Better regions with Bayesian
Optimization
64. 1. Given existing regions/scores •
2. Wefit a model
3. Introduce the chanceutility function
4. Locatethe maximum of the utility
5. Observe the new regionscore
6. Update the model.
Given the ability to query a our CNN for region scores
we can repeat the following:
Better regions with Bayesian
Optimization
65. 1. Given existing regions/scores •
2. Wefit a model
3. Introduce the chanceutility function
4. Locatethe maximum of the utility
5. Observe the new regionscore
6. Update the model.
7. Repeatstep 2.
Given the ability to query a our CNN for region scores
we can repeat the following:
Better regions with Bayesian
Optimization
66. 1. Given existing regions/scores •
2. Wefit a model
3. Introduce the chanceutility function
4. Locatethe maximum of the utility
5. Observe the new regionscore
6. Update the model.
7. Repeatstep 2.
Given the ability to query a our CNN for region scores
we can repeat the following:
Better regions with Bayesian
Optimization
67. Example of BO applied
to R-CNN
Yuting Zhang, Kihyuk Sohn, Ruben Villegas, Gang Pan, and Honglak Lee.