SlideShare uma empresa Scribd logo
1 de 55
Baixar para ler offline
Descriptors



 Cordelia Schmid
 Tinne Tuytelaars
Krystian Mikolajczyk
Overview – descriptors
•  Introduction

•  Modern descriptors

•  Comparison and evaluation
Descriptors

                                             Eliminate rotational   Compute appearance
Extract affine regions   Normalize regions                             descriptors
                                                + illumination




                                                                     SIFT (Lowe ’04)
Descriptors - history
•  Normalized cross-correlation (NCC) [~ 60s]
•  Gaussian derivative-based descriptors
   –  Differential invariants [Koenderink and van Doorn’87]
   –  Steerable filters [Freeman and Adelson’91]
•  Moment invariants [Van Gool et al.’96]
•  SIFT [Lowe’99]
•  Shape context [Belongie et al.’02]
•  Gradient PCA [Ke and Sukthankar’04]

•  SURF descriptor [Bay et al.’08]
•  DAISY descriptor [Tola et al.’08, Windler et al’09]
•  …….
SIFT descriptor [Lowe’99]

•     Spatial binning and binning of the gradient orientation
•     4x4 spatial grid, 8 orientations of the gradient, dim 128
•     Soft-assignment to spatial bins
•     Normalization of the descriptor to norm one (robust to illumination)
•     Comparison with Euclidean distance


     image patch           gradient x


                   →                    →
                       y
Local descriptors - rotation invariance
•  Estimation of the dominant orientation
   –  extract gradient orientation
   –  histogram over gradient orientation
   –  peak in this histogram
                                                                         0   2π


•  Rotate patch in dominant direction
         The image cannot be displayed. Your computer may not have
         enough memory to open the image, or the image may have
         been corrupted. Restart your computer, and then open the
         file again. If the red x still appears, you may have to delete
         the image and then insert it again.




•  Plus: invariance
•  Minus: less discriminant, additional noise
Evaluation
•  Descriptors should be
   –  Distinctive (! importance of the measurement region)
   –  Robust to changes on viewing conditions as well as to errors of
      the detector
   –  Compactness, speed of computation


•  Detection rate (recall)                      1
   –  #correct matches / #correspondences
•  False positive rate
   –  #false matches / #all matches
•  Variation of the distance threshold
   –  distance (d1, d2) < threshold
                                                                        1

  [K. Mikolajczyk & C. Schmid, PAMI’05]
Viewpoint change (60 degrees)
                                    Detector:
*   *                               Hessian-Affine
Scale change (factor 2.8)
                                  Detector:
* *                               Hessian-Affine
Evaluation - conclusion
•  SIFT based descriptors perform best

•  Significant difference between SIFT and low dimension
   descriptors as well as cross-correlation

•  Robust region descriptors better than point-wise
   descriptors

•  Performance of the descriptor is relatively independent of
   the detector
Recent extensions to SIFT
•  Color SIFT [Sande et al. 2010]

•  Normalizing SIFT with square root transformation




                   [Arandjelovic et Zisserman’12]
Descriptors – dense extraction
•  Many conclusions for descriptors applied to sparse
   detectors also hold for dense extraction
   –  For example normalizing SIFT with the square root improves in
      image retrieval and classification


•  Image retrieval: sparse versus dense

   SIFT + Fisher vector for retrieval on the Holidays dataset
                 Hessian-Affine   MAP = 0.54
                 Dense            MAP = 0.62
Overview – descriptors
•  Introduction

•  Modern descriptors

•  Comparison and evaluation
Overview

•  Introduction

•  Modern descriptors

•  Comparison and evaluation
Modern	
  descriptors	
  
•    Efficient	
  descriptors	
  
•    Compact	
  binary	
  descriptors	
  
•    More	
  robust	
  descriptors	
  
•    Learned	
  descriptors	
  
DAISY	
  
                                                                                                                   Cita%ons:	
  	
  
       •    Op<mized	
  for	
  dense	
  sampling	
                                                                 150	
  	
  (2012)	
  

       •    Log-­‐polar	
  grid	
  
       •    Gaussian	
  smoothing	
  
       •    Dealing	
  with	
  occlusions	
  




Engin	
  Tola,	
  Vincent	
  Lepe<t,	
  and	
  Pascal	
  Fua,	
  DAISY:	
  An	
  Efficient	
  Dense	
  Descriptor	
  Applied	
  to	
  
Wide-­‐Baseline	
  Stereo,	
  TPAMI	
  32(5),	
  2010.	
  	
  	
  
SURF:	
  
                      Speeded	
  Up	
  Robust	
  Features	
  
       •  Approximate	
  deriva<ves	
  with	
  Haar	
  wavelets	
  
       •  Exploit	
  integral	
  images	
                  Cita%ons:	
  	
  
                                                                                                                4500	
  	
  (2012)	
  




Herbert	
  Bay,	
  Andreas	
  Ess,	
  Tinne	
  Tuytelaars,	
  Luc	
  Van	
  Gool	
  "SURF:	
  Speeded	
  Up	
  Robust	
  Features",	
  
Computer	
  Vision	
  and	
  Image	
  Understanding	
  (CVIU),	
  Vol.	
  110,	
  No.	
  3,	
  pp.	
  346-­‐-­‐359,	
  2008	
  
SURF:	
  
      Speeded	
  Up	
  Robust	
  Features	
  
•  Orienta<on	
  assignment	
  
U-­‐SURF:	
  
                           Upright	
  SURF	
  
•  Rota<on	
  invariance	
  is	
  o`en	
  not	
  needed.	
  
  	
  


             	
   Don’t	
  use	
  more	
  invariance	
  	
  
                    	
  


         than	
  needed	
  for	
  a	
  given	
  applica<on	
  	
  
                                    	
  

•  Orienta<on	
  es<ma<on	
  takes	
  <me.	
  
•  Orienta<on	
  es<ma<on	
  is	
  o`en	
  a	
  source	
  of	
  
   errors.	
  	
  
Beyond	
  the	
  classics	
  
•    Efficient	
  descriptors	
  
•    Compact	
  binary	
  descriptors	
  
•    More	
  robust	
  descriptors	
  
•    Learned	
  descriptors	
  
Fast	
  and	
  compact	
  descriptors	
  
•  (Very)	
  large	
  scale	
  applica<ons	
  
   –  >	
  memory	
  issues,	
  computa<on	
  <me	
  issues	
  
•  Mobile	
  phone	
  applica<ons	
  
Fast	
  and	
  compact	
  descriptors	
  
•  Binary	
  descriptors	
  
•  Comparison	
  of	
  pairs	
  of	
  intensity	
  values	
  
     	
  -­‐	
  LBP	
  
     	
  -­‐	
  BRIEF	
  
     	
  -­‐	
  ORB	
  
     	
  -­‐	
  BRISK	
  
	
  
LBP:	
  
                                  Local	
  Binary	
  Paeerns	
  
        •  First	
  proposed	
  for	
  texture	
  recogni<on	
  in	
  1994.	
  




                                                                                                             Cita%ons:	
  	
  
                                                                                                             2500	
  	
  (2012)	
  
T.	
  Ojala,	
  M.	
  Pie<käinen,	
  and	
  D.	
  Harwood	
  (1994),	
  "Performance	
  evalua<on	
  of	
  texture	
  measures	
  
with	
  classifica<on	
  based	
  on	
  Kullback	
  discrimina<on	
  of	
  distribu<ons",	
  ICPR	
  1994,	
  pp.582-­‐585.	
  
M	
  Heikkilä,	
  M	
  Pie<käinen,	
  C	
  Schmid,	
  Descrip<on	
  of	
  interest	
  regions	
  with	
  LBP,	
  Paeern	
  
recogni<on	
  42	
  (3),	
  425-­‐436	
  
BRIEF:	
  
                         Binary	
  Robust	
  Independent	
  
                            Elementary	
  Features	
  
      •  Random	
  selec<on	
  of	
  pairs	
  
         of	
  intensity	
  values.	
  
      •  Fixed	
  sampling	
  paeern	
  
         of	
  128,	
  256	
  or	
  512	
  pairs.	
  
      •  Hamming	
  distance	
  to	
  	
  
         compare	
  descriptors	
  (XOR).	
  
                                                                                                                 Cita%ons:	
  	
  
                                                                                                                 149	
  	
  (2012)	
  

M.	
  Calonder,	
  V.	
  Lepe<t,	
  C.	
  Strecha,	
  P.	
  Fua,	
  BRIEF:	
  Binary	
  Robust	
  Independent	
  Elementary	
  
Features,	
  11th	
  European	
  Conference	
  on	
  Computer	
  Vision,	
  2010.	
  
D-­‐BRIEF:	
  
                                   Discrimina<ve	
  BRIEF	
  	
  

      •  Learn	
  linear	
  projec<ons	
  that	
  map	
  image	
  
         patches	
  to	
  a	
  more	
  discrimina<ve	
  subspace	
  
      •  Exploit	
  integral	
  images	
  




T.	
  Trzcinski	
  and	
  V.	
  Lepe<t,	
  Efficient	
  Discrimina<ve	
  Projec<ons	
  for	
  Compact	
  Binary	
  Descriptors	
  
European	
  Conference	
  on	
  Computer	
  Vision	
  (ECCV)	
  2012	
  
ORB:	
  
               Oriented	
  FAST	
  and	
  Rotated	
  BRIEF	
  
      •  Add	
  rota<on	
  invariance	
  to	
  BRIEF	
                                                        Cita%ons:	
  	
  
                                                                                                              43	
  	
  (2012)	
  
      •  Orienta<on	
  assignment	
  based	
  	
  
         on	
  the	
  intensity	
  centroid	
  

      	
  




Ethan	
  Rublee,	
  Vincent	
  Rabaud,	
  Kurt	
  Konolige,	
  Gary	
  Bradski,	
  ORB:	
  an	
  efficient	
  alterna<ve	
  to	
  SIFT	
  
or	
  SURF,	
  ICCV	
  2011	
  
ORB:	
  
     Oriented	
  FAST	
  and	
  Rotated	
  BRIEF	
  
•  Select	
  a	
  good	
  set	
  of	
  pairwise	
  comparisons:	
  
   minimize	
  correla<on	
  under	
  various	
  
   orienta<on	
  changes	
  




                High	
  variance	
  	
     High	
  variance	
  +	
  uncorrelated	
  	
  
BRISK:	
  	
  
           Binary	
  Robust	
  Invariant	
  Scalable	
  
                                 Keypoints	
  
                                                  Cita%ons:	
  	
  
      •  Regular	
  grid	
  	
       	
           20	
  	
  (2012)	
  

      •  Orienta<on	
  assignment	
  based	
  on	
  dominant	
  
         gradient	
  direc<on	
  (using	
  long-­‐distance	
  pairs)	
  



      •  512	
  bit	
  descriptor	
  based	
  on	
  
         short-­‐distance	
  pairs	
  
Stefan	
  Leutenegger,	
  Margarita	
  Chli	
  and	
  Roland	
  Y.	
  Siegwart,	
  BRISK:	
  Binary	
  Robust	
  Invariant	
  
Scalable	
  Keypoints,	
  ICCV	
  2011	
  
Various	
  others	
  
•  FREAK:	
  Fast	
  Re<na	
  Keypoint	
  
•  CARD:	
  Compact	
  and	
  Real<me	
  Descriptor	
  
•  LDB:	
  Local	
  Difference	
  Binary	
  	
  
   	
  
FREAK:	
  	
  
                                     Fast	
  Re<na	
  Keypoint	
  	
  
                                                 	
  
       •  Inspired	
  by	
  the	
  human	
  visual	
  system	
  




A.	
  Alahi,	
  R.	
  Or<z,	
  and	
  P.	
  Vandergheynst.	
  FREAK:	
  Fast	
  Re<na	
  Keypoint.	
  In	
  IEEE	
  Conference	
  on	
  
Computer	
  Vision	
  and	
  Paeern	
  Recogni<on	
  2012.	
  
CARD:	
  	
  
Compact	
  and	
  Real<me	
  Descriptor	
  
–  Look	
  up	
  tables	
     	
  
–  learning-­‐based	
  sparse	
  hashing	
  
LDB:	
  
                                Local	
  Difference	
  Binary	
  




Xin	
  Yang	
  and	
  Kwang-­‐Ting	
  Cheng,	
  LDB:	
  An	
  Ultra-­‐Fast	
  Feature	
  for	
  Scalable	
  Augmened	
  Reality	
  on	
  
Mobile	
  Devices,	
  Interna<onal	
  Symposium	
  on	
  Mixed	
  and	
  Augmented	
  Reality	
  2012	
  (ISMAR	
  2012)	
  
Beyond	
  the	
  classics	
  
•    Efficient	
  descriptors	
  
•    Compact	
  binary	
  descriptors	
  
•    More	
  robust	
  descriptors	
  
•    Learned	
  descriptors	
  
LIOP:	
  
                 Local	
  Intensity	
  Order	
  Paeern	
  for	
  
                           Feature	
  Descrip<on	
  
                                                                            (and	
  predecessors	
  MROGH	
  and	
  MRRID)	
  


      •  Robustness	
  to	
  monotonic	
  intensity	
  changes	
  
      •  Data-­‐driven	
  division	
  into	
  cells	
  




Zhenhua	
  Wang	
  Bin	
  Fan	
  Fuchao	
  Wu,	
  Local	
  Intensity	
  Order	
  Paeern	
  for	
  Feature	
  Descrip<on,	
  ICCV	
  
2011.	
  
LIOP:	
  
Local	
  Intensity	
  Order	
  Paeern	
  for	
  
          Feature	
  Descrip<on	
  
Beyond	
  the	
  classics	
  
•    Efficient	
  descriptors	
  
•    Compact	
  binary	
  descriptors	
  
•    More	
  robust	
  descriptors	
  
•    Learned	
  descriptors	
  
Winder	
  &	
  Brown	
  
       •  Learn	
  configura<on	
  and	
  other	
  parameters	
  
          from	
  training	
  data	
  obtained	
  from	
  3D	
   Cita%ons:	
  	
  
          reconstruc<ons	
                                       194	
  (2012)	
  




M.	
  Brown,	
  G.	
  Hua	
  and	
  S.	
  Winder,	
  Discriminant	
  Learning	
  of	
  Local	
  Image	
  Descriptors..	
  
IEEE	
  Transac<ons	
  on	
  Paeern	
  Analysis	
  and	
  Machine	
  Intelligence.	
  2010.	
  
Winder	
  &	
  Brown	
  
•  Training	
  data	
  =	
  set	
  of	
  corresponding	
  image	
  
   patches	
  
Descriptor	
  Learning	
  	
  
                                 Using	
  Convex	
  Op<misa<on	
  
       •  Convex	
  learning	
  of	
                                                                                Pre-­‐rec<fied	
  	
  
                                                                                                                   keypoint	
  patch	
  	
  

               –  spa<al	
  pooling	
  regions	
  
               –  dimensionality	
  reduc<on	
                                                                 Non-­‐linear	
  transform	
  


       •  Learning	
  from	
  very	
  weak	
  	
  
          supervision	
                                                                    learning	
              Spa<al	
  pooling	
  	
  



                                                                                                                Normalisa<on	
  and	
  
                                                                                                                    cropping	
  



                                                                                                                   Dimensionality	
  
                                                                                           learning	
                reduc<on	
  



                                                                                                                 Descriptor	
  vector	
  

K.	
  Simonyan	
  et	
  al.,	
  Descriptor	
  Learning	
  Using	
  Convex	
  Op<misa<on,	
  ECCV	
  2012	
  
Learning	
  Spa<al	
  Pooling	
  Regions	
  
•  Selec%on	
  from	
  a	
  large	
  pool	
  using	
  L1	
  regularisa%on	
  
•  So`-­‐margin	
  constraints:	
  	
  
    –  squared	
  L2	
  distance	
  between	
  descriptors	
  of	
  matching	
  feature	
  
       pairs	
  should	
  be	
  smaller	
  than	
  that	
  of	
  non-­‐matching	
  pairs	
  
•  Convex	
  objec<ve	
  (op<mised	
  with	
  a	
  proximal	
  method)	
  




       768-­‐D	
                    576-­‐D	
                              320-­‐D	
  
    pooling	
  region	
  configura%ons	
  learnt	
  with	
  different	
  levels	
  of	
  sparsity	
  
Overview

•  Introduction

•  Modern descriptors

•  Comparison and evaluation
Sepng	
  up	
  an	
  evalua<on	
  
•  Which	
  problem?	
  Performance	
  in	
  different	
  applica<on/niches	
  may	
  
   vary	
  significantly.	
  
     –  Category	
  recogni<on,	
  	
  
     –  Matching,	
  	
  
     –  Retrieval	
  	
  

•  What	
  dataset?	
  
     –  Pascal	
  VOC	
  2007	
  
     –  Oxford	
  image	
  pairs	
  
     –  Oxford	
  -­‐	
  Paris	
  buildings	
  

•  Protocol	
  and	
  criteria?	
  
     –  Public	
  dataset,	
  	
  
     –  Avoiding	
  risk	
  to	
  over-­‐fipng/op<mizing	
  to	
  the	
  data	
  
     	
  
Detector	
  evalua<ons	
  

                                                             homography


                                              A
                                                                                  B
                                                         B




                                            Two points are correctly matched if
                # correct matches
    precision =                              T=40%
                   # all matches                         A∩ B
                                                              >T
                                                         A∪ B
                  # correct matches
recall =
           # ground truth correspondences
Previous	
  Evalua<ons	
  
•  2D	
  Scene	
  –	
  Homography	
  
       –  C.	
  Schmid,	
  R.	
  Mohr,	
  and	
  C.	
  Bauckhage,	
  “Evalua<on	
  of	
  interest	
  point	
  
          detectors,”	
  IJCV,	
  2000.	
  
       –  K.	
  Mikolajczyk	
  and	
  C.	
  Schmid,	
  “A	
  performance	
  evalua<on	
  of	
  local	
  descriptors,”	
  
          CVPR,	
  2003.	
  
       –  T.	
  Kadir,	
  M.	
  Brady,	
  and	
  A.	
  Zisserman,	
  “An	
  affine	
  invariant	
  method	
  for	
  selec<ng	
  
          salient	
  regions	
  in	
  images,”	
  in	
  ECCV,	
  2004.	
  
       –  K.	
  Mikolajczyk,	
  T.	
  Tuytelaars,	
  C.	
  Schmid,	
  A.	
  Zisserman,	
  J.	
  Matas,	
  F.	
  
          Schaffalitzky,T.	
  Kadir,	
  and	
  L.	
  Van	
  Gool,	
  “A	
  comparison	
  of	
  affine	
  region	
  
          detectors,”	
  IJCV,	
  2005.	
  
       –  A.	
  Haja,	
  S.	
  Abraham,	
  and	
  B.	
  Jahne,	
  Localiza<on	
  accuracy	
  of	
  region	
  detectors,	
  
          CVPR	
  2008	
  	
  
       –  T.	
  Dickscheid,	
  	
  FSchindler,	
  Falko,	
  W.	
  Förstner,	
  Coding	
  Images	
  with	
  Local	
  
          Features,	
  IJCV	
  2011	
  


	
  
Previous	
  Evalua<ons	
  
•  3D	
  Scene	
  -­‐	
  epipolar	
  constraints	
  
       –  F.	
  Fraundorfer	
  and	
  H.	
  Bischof,	
  “Evalua<on	
  of	
  local	
  detectors	
  on	
  non-­‐planar,	
  
          scenes,”	
  in	
  AAPR,	
  2004.	
  	
  
       –  P.	
  Moreels	
  and	
  P.	
  Perona,	
  “Evalua<on	
  of	
  features	
  detectors	
  and	
  descriptors	
  
          based	
  on	
  3D	
  objects,”	
  IJCV,	
  2007.	
  
       –  S.	
  Winder	
  and	
  M.	
  Brown,	
  “Learning	
  local	
  image	
  descriptors,”	
  CVPR,	
  
          2007,2009.	
  
       –  Dahl,	
  A.L.,	
  Aanæs,	
  H.	
  and	
  Pedersen,	
  K.S.	
  (2011):	
  Finding	
  the	
  Best	
  Feature	
  
          Detector-­‐Descriptor	
  Combina<on.	
  3DIMPVT,	
  2011.	
  	
  


	
  
Recent	
  Evalua<ons	
  
•  Recent	
  detectors	
  
O.	
  Miksik	
  and	
  K.	
  Mikolajczyk,	
  Evalua<on	
  of	
  Local	
  Detectors	
  and	
  Descriptors	
  for	
  Fast	
  Feature	
  
Matching,	
  ICPR	
  2012	
  




ECCV	
  2012	
  	
  Modern	
  features:	
  
                                                                                                                                         	
  46/60	
  
…	
  Detectors.	
  
Recent	
  detector	
  evalua<ons	
  
•  Completeness	
  (coverage),	
  complementarity	
  between	
  
   detectors	
  




	
  




 T.	
  Dickscheid,	
  	
  FSchindler,	
  Falko,	
  W.	
  Förstner,	
  Coding	
  Images	
  with	
  Local	
  Features,	
  IJCV	
  2011	
  
        –  Completeness,	
  Edgelap	
  (Mikolajczyk),	
  Salient	
  (Kadir),	
  MSER	
  (Matas)	
  
        –  Complementrity,	
  MSER	
  +	
  SFOP	
  	
  
Descriptor	
  Evalua<ons	
  
•  Matching	
  precision	
  and	
  recall	
  




O.	
  Miksik	
  and	
  K.	
  Mikolajczyk,	
  Evalua<on	
  of	
  Local	
  Detectors	
  and	
  Descriptors	
  for	
  Fast	
  
Feature	
  Matching,	
  ICPR	
  2012	
  
Recent	
  Descriptor	
  Evalua<ons	
  
•  Computa%on	
  %mes	
  for	
  the	
  different	
  descriptors	
  for	
  1000	
  
   SURF	
  keypoints	
  
O.	
  Miksik	
  and	
  K.	
  Mikolajczyk,	
  Evalua<on	
  of	
  Local	
  Detectors	
  and	
  Descriptors	
  for	
  Fast	
  Feature	
  
Matching,	
  ICPR	
  2012	
  
J.	
  Heinly	
  E.	
  Dunn,	
  J-­‐M.	
  Frahm,	
  Compara<ve	
  Evalua<on	
  of	
  Binary	
  Features,	
  ECCV2012	
  	
  
Previous	
  Evalua<ons	
  
•    Image/object	
  categories	
  
      –  K.	
  Mikolajczyk,	
  B.	
  Leibe,	
  and	
  B.	
  Schiele,	
  “Local	
  features	
  for	
  object	
  class	
  
         recogni<on,”	
  in	
  ICCV,	
  2005	
  
      –  E.	
  Seemann,	
  B.	
  Leibe,	
  K.	
  Mikolajczyk,	
  and	
  B.	
  Schiele,	
  “An	
  evalua<on	
  of	
  local	
  
         shape-­‐based	
  features	
  for	
  pedestrian	
  detec<on,”	
  in	
  BMVC,	
  2005.	
  
      –  M.	
  Stark	
  and	
  B.	
  Schiele,	
  “How	
  good	
  are	
  local	
  features	
  for	
  classes	
  of	
  geometric	
  
         objects,”	
  in	
  ICCV,	
  2007.	
  
      –  K.	
  E.	
  A.	
  van	
  de	
  Sande,	
  T.	
  Gevers	
  and	
  C.	
  G.	
  M.	
  Snoek,	
  Evalua<on	
  of	
  Color	
  
         Descriptors	
  for	
  Object	
  and	
  Scene	
  Recogni<on.	
  CVPR,	
  2008.	
  	
  
Approach	
  
•  Bags-­‐of-­‐features	
  
     1.      Interest	
  point	
  /	
  region	
  detector	
  
     2.      Descriptors	
  
     3.      K-­‐means	
  clustering	
  (4000	
  clusters)	
  
     4.      Histogram	
  of	
  cluster	
  occurrences	
  (NN	
  assignment)	
  
     5.      Chi-­‐square	
  distance	
  and	
  RBF	
  kernel	
  for	
  KDA	
  or	
  SVM	
  classifier	
  	
  




  • 	
  J.	
  Zhang	
  and	
  M.	
  Marszalek	
  and	
  S.	
  Lazebnik	
  and	
  C.	
  Schmid,	
  	
  
  Local	
  Features	
  and	
  Kernels	
  for	
  Classifica<on	
  of	
  Texture	
  and	
  Object	
  Categories:	
  	
  
  A	
  Comprehensive	
  Study,	
  IJCV,	
  2007	
  
  • 	
  K.	
  E.	
  A.	
  van	
  de	
  Sande,	
  T.	
  Gevers	
  and	
  C.	
  G.	
  M.	
  Snoek,	
  	
  
  Evalua<on	
  of	
  Color	
  Descriptors	
  for	
  Object	
  and	
  Scene	
  Recogni<on.	
  CVPR,	
  2008	
  
Evalua<on	
  
•  PASCAL	
  VOC	
  measures	
  
   –  Average	
  precision	
  for	
  every	
  object	
  category	
  
   –  Mean	
  average	
  precision	
  	
                    Category	
     AP	
  




                                           output=>	
  
     precision	
  




                     AP	
  




                       recall	
  
MAP	
  Ranking	
  
          color/gray,	
  density,	
  dimensionality	
  ...	
  
          MAP	
                           #dimensions	
                              density	
  




•  SIFT	
  s<ll	
  dominates	
  (Histograms	
  of	
  gradient	
  loca<ons	
  and	
  orienta<ons)	
  
•  Opponent	
  chroma<c	
  space	
  (normalized	
  red-­‐green,	
  blue-­‐yellow,	
  and	
  intensity	
  Y	
  
Grayvalue	
  descriptors	
  
            MAP	
  Ranking	
   #dimensions	
                                                   density	
  




	
  
•  Observa<ons	
  
       •    Color	
  improves	
  
       •    All	
  based	
  on	
  histograms	
  of	
  gradient	
  loca<ons	
  and	
  orienta<ons	
  
       •    Dimensionality	
  not	
  much	
  correlated	
  with	
  the	
  performance	
  
       •    Density	
  Strongly	
  correlated	
  (the	
  more	
  the	
  beeer)	
  
       •    Results	
  biased	
  by	
  density	
  
       •    Implementa<on	
  details	
  maeer	
  
Break	
  !	
  

Mais conteúdo relacionado

Mais procurados

Edge Detection and Segmentation
Edge Detection and SegmentationEdge Detection and Segmentation
Edge Detection and SegmentationA B Shinde
 
Edge Detection using Hough Transform
Edge Detection using Hough TransformEdge Detection using Hough Transform
Edge Detection using Hough TransformMrunal Selokar
 
Lossless predictive coding in Digital Image Processing
Lossless predictive coding in Digital Image ProcessingLossless predictive coding in Digital Image Processing
Lossless predictive coding in Digital Image Processingpriyadharshini murugan
 
03 digital image fundamentals DIP
03 digital image fundamentals DIP03 digital image fundamentals DIP
03 digital image fundamentals DIPbabak danyal
 
"Introduction to Feature Descriptors in Vision: From Haar to SIFT," A Present...
"Introduction to Feature Descriptors in Vision: From Haar to SIFT," A Present..."Introduction to Feature Descriptors in Vision: From Haar to SIFT," A Present...
"Introduction to Feature Descriptors in Vision: From Haar to SIFT," A Present...Edge AI and Vision Alliance
 
Fidelity criteria in image compression
Fidelity criteria in image compressionFidelity criteria in image compression
Fidelity criteria in image compressionKadamPawan
 
Computer Vision: Feature matching with RANSAC Algorithm
Computer Vision: Feature matching with RANSAC AlgorithmComputer Vision: Feature matching with RANSAC Algorithm
Computer Vision: Feature matching with RANSAC Algorithmallyn joy calcaben
 
Predictive coding
Predictive codingPredictive coding
Predictive codingp_ayal
 
Fingerprint Recognition Technique(PPT)
Fingerprint Recognition Technique(PPT)Fingerprint Recognition Technique(PPT)
Fingerprint Recognition Technique(PPT)Sandeep Kumar Panda
 
Feature detection - Image Processing
Feature detection - Image ProcessingFeature detection - Image Processing
Feature detection - Image ProcessingRitesh Kanjee
 
Hough Transform By Md.Nazmul Islam
Hough Transform By Md.Nazmul IslamHough Transform By Md.Nazmul Islam
Hough Transform By Md.Nazmul IslamNazmul Islam
 
Ip elements of image processing
Ip elements of image processingIp elements of image processing
Ip elements of image processingNishirajNath
 
Image feature extraction
Image feature extractionImage feature extraction
Image feature extractionRishabh shah
 
Human visual perception
Human visual perceptionHuman visual perception
Human visual perceptionritanyan
 
Elements of visual perception
Elements of visual perceptionElements of visual perception
Elements of visual perceptionDr INBAMALAR T M
 
Hidden surfaces
Hidden surfacesHidden surfaces
Hidden surfacesMohd Arif
 
Feature Matching using SIFT algorithm
Feature Matching using SIFT algorithmFeature Matching using SIFT algorithm
Feature Matching using SIFT algorithmSajid Pareeth
 

Mais procurados (20)

Edge Detection and Segmentation
Edge Detection and SegmentationEdge Detection and Segmentation
Edge Detection and Segmentation
 
Edge Detection using Hough Transform
Edge Detection using Hough TransformEdge Detection using Hough Transform
Edge Detection using Hough Transform
 
Lossless predictive coding in Digital Image Processing
Lossless predictive coding in Digital Image ProcessingLossless predictive coding in Digital Image Processing
Lossless predictive coding in Digital Image Processing
 
03 digital image fundamentals DIP
03 digital image fundamentals DIP03 digital image fundamentals DIP
03 digital image fundamentals DIP
 
"Introduction to Feature Descriptors in Vision: From Haar to SIFT," A Present...
"Introduction to Feature Descriptors in Vision: From Haar to SIFT," A Present..."Introduction to Feature Descriptors in Vision: From Haar to SIFT," A Present...
"Introduction to Feature Descriptors in Vision: From Haar to SIFT," A Present...
 
Fidelity criteria in image compression
Fidelity criteria in image compressionFidelity criteria in image compression
Fidelity criteria in image compression
 
Computer Vision: Feature matching with RANSAC Algorithm
Computer Vision: Feature matching with RANSAC AlgorithmComputer Vision: Feature matching with RANSAC Algorithm
Computer Vision: Feature matching with RANSAC Algorithm
 
Predictive coding
Predictive codingPredictive coding
Predictive coding
 
Digital Holography
Digital HolographyDigital Holography
Digital Holography
 
Fingerprint Recognition Technique(PPT)
Fingerprint Recognition Technique(PPT)Fingerprint Recognition Technique(PPT)
Fingerprint Recognition Technique(PPT)
 
Feature detection - Image Processing
Feature detection - Image ProcessingFeature detection - Image Processing
Feature detection - Image Processing
 
Hough Transform By Md.Nazmul Islam
Hough Transform By Md.Nazmul IslamHough Transform By Md.Nazmul Islam
Hough Transform By Md.Nazmul Islam
 
Ip elements of image processing
Ip elements of image processingIp elements of image processing
Ip elements of image processing
 
Image feature extraction
Image feature extractionImage feature extraction
Image feature extraction
 
Human visual perception
Human visual perceptionHuman visual perception
Human visual perception
 
GRPHICS06 - Shading
GRPHICS06 - ShadingGRPHICS06 - Shading
GRPHICS06 - Shading
 
Shading methods
Shading methodsShading methods
Shading methods
 
Elements of visual perception
Elements of visual perceptionElements of visual perception
Elements of visual perception
 
Hidden surfaces
Hidden surfacesHidden surfaces
Hidden surfaces
 
Feature Matching using SIFT algorithm
Feature Matching using SIFT algorithmFeature Matching using SIFT algorithm
Feature Matching using SIFT algorithm
 

Semelhante a Modern features-part-2-descriptors

Computer Vision descriptors
Computer Vision descriptorsComputer Vision descriptors
Computer Vision descriptorsWael Badawy
 
Gdc2012 frames, sparsity and global illumination
Gdc2012 frames, sparsity and global illumination Gdc2012 frames, sparsity and global illumination
Gdc2012 frames, sparsity and global illumination Manchor Ko
 
Semantic Retrieval and Automatic Annotation: Linear Transformations, Correlat...
Semantic Retrieval and Automatic Annotation: Linear Transformations, Correlat...Semantic Retrieval and Automatic Annotation: Linear Transformations, Correlat...
Semantic Retrieval and Automatic Annotation: Linear Transformations, Correlat...Jonathon Hare
 
Digital image classification
Digital image classificationDigital image classification
Digital image classificationAleemuddin Abbasi
 
Convolutional Patch Representations for Image Retrieval An unsupervised approach
Convolutional Patch Representations for Image Retrieval An unsupervised approachConvolutional Patch Representations for Image Retrieval An unsupervised approach
Convolutional Patch Representations for Image Retrieval An unsupervised approachUniversitat de Barcelona
 
Art of Feature Engineering for Data Science with Nabeel Sarwar
Art of Feature Engineering for Data Science with Nabeel SarwarArt of Feature Engineering for Data Science with Nabeel Sarwar
Art of Feature Engineering for Data Science with Nabeel SarwarSpark Summit
 
CVPR2008 tutorial generalized pca
CVPR2008 tutorial generalized pcaCVPR2008 tutorial generalized pca
CVPR2008 tutorial generalized pcazukun
 
Passive stereo vision with deep learning
Passive stereo vision with deep learningPassive stereo vision with deep learning
Passive stereo vision with deep learningYu Huang
 
Enhanced characterness for text detection in the wild
Enhanced characterness for text detection in the wildEnhanced characterness for text detection in the wild
Enhanced characterness for text detection in the wildPrerana Mukherjee
 
Oxford 05-oct-2012
Oxford 05-oct-2012Oxford 05-oct-2012
Oxford 05-oct-2012Ted Dunning
 
priyankamainthesisppt.pptx
priyankamainthesisppt.pptxpriyankamainthesisppt.pptx
priyankamainthesisppt.pptxsaiproject
 
04-Timo_Kunkel.pdf
04-Timo_Kunkel.pdf04-Timo_Kunkel.pdf
04-Timo_Kunkel.pdftristone1
 
IEEE DSP Workshop 2011
IEEE DSP Workshop 2011IEEE DSP Workshop 2011
IEEE DSP Workshop 2011Behzad Dogahe
 
IEEE DSP Workshop 2011
IEEE DSP Workshop 2011IEEE DSP Workshop 2011
IEEE DSP Workshop 2011dogahe
 
Lec16: Medical Image Registration (Advanced): Deformable Registration
Lec16: Medical Image Registration (Advanced): Deformable RegistrationLec16: Medical Image Registration (Advanced): Deformable Registration
Lec16: Medical Image Registration (Advanced): Deformable RegistrationUlaş Bağcı
 

Semelhante a Modern features-part-2-descriptors (20)

PPT s11-machine vision-s2
PPT s11-machine vision-s2PPT s11-machine vision-s2
PPT s11-machine vision-s2
 
Computer Vision descriptors
Computer Vision descriptorsComputer Vision descriptors
Computer Vision descriptors
 
Gdc2012 frames, sparsity and global illumination
Gdc2012 frames, sparsity and global illumination Gdc2012 frames, sparsity and global illumination
Gdc2012 frames, sparsity and global illumination
 
GIS fundamentals - raster
GIS fundamentals - rasterGIS fundamentals - raster
GIS fundamentals - raster
 
Semantic Retrieval and Automatic Annotation: Linear Transformations, Correlat...
Semantic Retrieval and Automatic Annotation: Linear Transformations, Correlat...Semantic Retrieval and Automatic Annotation: Linear Transformations, Correlat...
Semantic Retrieval and Automatic Annotation: Linear Transformations, Correlat...
 
Digital image classification
Digital image classificationDigital image classification
Digital image classification
 
Convolutional Patch Representations for Image Retrieval An unsupervised approach
Convolutional Patch Representations for Image Retrieval An unsupervised approachConvolutional Patch Representations for Image Retrieval An unsupervised approach
Convolutional Patch Representations for Image Retrieval An unsupervised approach
 
Art of Feature Engineering for Data Science with Nabeel Sarwar
Art of Feature Engineering for Data Science with Nabeel SarwarArt of Feature Engineering for Data Science with Nabeel Sarwar
Art of Feature Engineering for Data Science with Nabeel Sarwar
 
CVPR2008 tutorial generalized pca
CVPR2008 tutorial generalized pcaCVPR2008 tutorial generalized pca
CVPR2008 tutorial generalized pca
 
Passive stereo vision with deep learning
Passive stereo vision with deep learningPassive stereo vision with deep learning
Passive stereo vision with deep learning
 
Enhanced characterness for text detection in the wild
Enhanced characterness for text detection in the wildEnhanced characterness for text detection in the wild
Enhanced characterness for text detection in the wild
 
16 17 bag_words
16 17 bag_words16 17 bag_words
16 17 bag_words
 
Oxford 05-oct-2012
Oxford 05-oct-2012Oxford 05-oct-2012
Oxford 05-oct-2012
 
DNN Model Interpretability
DNN Model InterpretabilityDNN Model Interpretability
DNN Model Interpretability
 
WT in IP.ppt
WT in IP.pptWT in IP.ppt
WT in IP.ppt
 
priyankamainthesisppt.pptx
priyankamainthesisppt.pptxpriyankamainthesisppt.pptx
priyankamainthesisppt.pptx
 
04-Timo_Kunkel.pdf
04-Timo_Kunkel.pdf04-Timo_Kunkel.pdf
04-Timo_Kunkel.pdf
 
IEEE DSP Workshop 2011
IEEE DSP Workshop 2011IEEE DSP Workshop 2011
IEEE DSP Workshop 2011
 
IEEE DSP Workshop 2011
IEEE DSP Workshop 2011IEEE DSP Workshop 2011
IEEE DSP Workshop 2011
 
Lec16: Medical Image Registration (Advanced): Deformable Registration
Lec16: Medical Image Registration (Advanced): Deformable RegistrationLec16: Medical Image Registration (Advanced): Deformable Registration
Lec16: Medical Image Registration (Advanced): Deformable Registration
 

Mais de zukun

My lyn tutorial 2009
My lyn tutorial 2009My lyn tutorial 2009
My lyn tutorial 2009zukun
 
ETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCVETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCVzukun
 
ETHZ CV2012: Information
ETHZ CV2012: InformationETHZ CV2012: Information
ETHZ CV2012: Informationzukun
 
Siwei lyu: natural image statistics
Siwei lyu: natural image statisticsSiwei lyu: natural image statistics
Siwei lyu: natural image statisticszukun
 
Lecture9 camera calibration
Lecture9 camera calibrationLecture9 camera calibration
Lecture9 camera calibrationzukun
 
Brunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer visionBrunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer visionzukun
 
Modern features-part-4-evaluation
Modern features-part-4-evaluationModern features-part-4-evaluation
Modern features-part-4-evaluationzukun
 
Modern features-part-3-software
Modern features-part-3-softwareModern features-part-3-software
Modern features-part-3-softwarezukun
 
Modern features-part-1-detectors
Modern features-part-1-detectorsModern features-part-1-detectors
Modern features-part-1-detectorszukun
 
Modern features-part-0-intro
Modern features-part-0-introModern features-part-0-intro
Modern features-part-0-introzukun
 
Lecture 02 internet video search
Lecture 02 internet video searchLecture 02 internet video search
Lecture 02 internet video searchzukun
 
Lecture 01 internet video search
Lecture 01 internet video searchLecture 01 internet video search
Lecture 01 internet video searchzukun
 
Lecture 03 internet video search
Lecture 03 internet video searchLecture 03 internet video search
Lecture 03 internet video searchzukun
 
Icml2012 tutorial representation_learning
Icml2012 tutorial representation_learningIcml2012 tutorial representation_learning
Icml2012 tutorial representation_learningzukun
 
Advances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer visionAdvances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer visionzukun
 
Gephi tutorial: quick start
Gephi tutorial: quick startGephi tutorial: quick start
Gephi tutorial: quick startzukun
 
EM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysisEM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysiszukun
 
Object recognition with pictorial structures
Object recognition with pictorial structuresObject recognition with pictorial structures
Object recognition with pictorial structureszukun
 
Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities zukun
 
Icml2012 learning hierarchies of invariant features
Icml2012 learning hierarchies of invariant featuresIcml2012 learning hierarchies of invariant features
Icml2012 learning hierarchies of invariant featureszukun
 

Mais de zukun (20)

My lyn tutorial 2009
My lyn tutorial 2009My lyn tutorial 2009
My lyn tutorial 2009
 
ETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCVETHZ CV2012: Tutorial openCV
ETHZ CV2012: Tutorial openCV
 
ETHZ CV2012: Information
ETHZ CV2012: InformationETHZ CV2012: Information
ETHZ CV2012: Information
 
Siwei lyu: natural image statistics
Siwei lyu: natural image statisticsSiwei lyu: natural image statistics
Siwei lyu: natural image statistics
 
Lecture9 camera calibration
Lecture9 camera calibrationLecture9 camera calibration
Lecture9 camera calibration
 
Brunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer visionBrunelli 2008: template matching techniques in computer vision
Brunelli 2008: template matching techniques in computer vision
 
Modern features-part-4-evaluation
Modern features-part-4-evaluationModern features-part-4-evaluation
Modern features-part-4-evaluation
 
Modern features-part-3-software
Modern features-part-3-softwareModern features-part-3-software
Modern features-part-3-software
 
Modern features-part-1-detectors
Modern features-part-1-detectorsModern features-part-1-detectors
Modern features-part-1-detectors
 
Modern features-part-0-intro
Modern features-part-0-introModern features-part-0-intro
Modern features-part-0-intro
 
Lecture 02 internet video search
Lecture 02 internet video searchLecture 02 internet video search
Lecture 02 internet video search
 
Lecture 01 internet video search
Lecture 01 internet video searchLecture 01 internet video search
Lecture 01 internet video search
 
Lecture 03 internet video search
Lecture 03 internet video searchLecture 03 internet video search
Lecture 03 internet video search
 
Icml2012 tutorial representation_learning
Icml2012 tutorial representation_learningIcml2012 tutorial representation_learning
Icml2012 tutorial representation_learning
 
Advances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer visionAdvances in discrete energy minimisation for computer vision
Advances in discrete energy minimisation for computer vision
 
Gephi tutorial: quick start
Gephi tutorial: quick startGephi tutorial: quick start
Gephi tutorial: quick start
 
EM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysisEM algorithm and its application in probabilistic latent semantic analysis
EM algorithm and its application in probabilistic latent semantic analysis
 
Object recognition with pictorial structures
Object recognition with pictorial structuresObject recognition with pictorial structures
Object recognition with pictorial structures
 
Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities Iccv2011 learning spatiotemporal graphs of human activities
Iccv2011 learning spatiotemporal graphs of human activities
 
Icml2012 learning hierarchies of invariant features
Icml2012 learning hierarchies of invariant featuresIcml2012 learning hierarchies of invariant features
Icml2012 learning hierarchies of invariant features
 

Modern features-part-2-descriptors

  • 1. Descriptors Cordelia Schmid Tinne Tuytelaars Krystian Mikolajczyk
  • 2. Overview – descriptors •  Introduction •  Modern descriptors •  Comparison and evaluation
  • 3. Descriptors Eliminate rotational Compute appearance Extract affine regions Normalize regions descriptors + illumination SIFT (Lowe ’04)
  • 4. Descriptors - history •  Normalized cross-correlation (NCC) [~ 60s] •  Gaussian derivative-based descriptors –  Differential invariants [Koenderink and van Doorn’87] –  Steerable filters [Freeman and Adelson’91] •  Moment invariants [Van Gool et al.’96] •  SIFT [Lowe’99] •  Shape context [Belongie et al.’02] •  Gradient PCA [Ke and Sukthankar’04] •  SURF descriptor [Bay et al.’08] •  DAISY descriptor [Tola et al.’08, Windler et al’09] •  …….
  • 5. SIFT descriptor [Lowe’99] •  Spatial binning and binning of the gradient orientation •  4x4 spatial grid, 8 orientations of the gradient, dim 128 •  Soft-assignment to spatial bins •  Normalization of the descriptor to norm one (robust to illumination) •  Comparison with Euclidean distance image patch gradient x → → y
  • 6. Local descriptors - rotation invariance •  Estimation of the dominant orientation –  extract gradient orientation –  histogram over gradient orientation –  peak in this histogram 0 2π •  Rotate patch in dominant direction The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again. •  Plus: invariance •  Minus: less discriminant, additional noise
  • 7. Evaluation •  Descriptors should be –  Distinctive (! importance of the measurement region) –  Robust to changes on viewing conditions as well as to errors of the detector –  Compactness, speed of computation •  Detection rate (recall) 1 –  #correct matches / #correspondences •  False positive rate –  #false matches / #all matches •  Variation of the distance threshold –  distance (d1, d2) < threshold 1 [K. Mikolajczyk & C. Schmid, PAMI’05]
  • 8. Viewpoint change (60 degrees) Detector: * * Hessian-Affine
  • 9. Scale change (factor 2.8) Detector: * * Hessian-Affine
  • 10. Evaluation - conclusion •  SIFT based descriptors perform best •  Significant difference between SIFT and low dimension descriptors as well as cross-correlation •  Robust region descriptors better than point-wise descriptors •  Performance of the descriptor is relatively independent of the detector
  • 11. Recent extensions to SIFT •  Color SIFT [Sande et al. 2010] •  Normalizing SIFT with square root transformation [Arandjelovic et Zisserman’12]
  • 12. Descriptors – dense extraction •  Many conclusions for descriptors applied to sparse detectors also hold for dense extraction –  For example normalizing SIFT with the square root improves in image retrieval and classification •  Image retrieval: sparse versus dense SIFT + Fisher vector for retrieval on the Holidays dataset Hessian-Affine MAP = 0.54 Dense MAP = 0.62
  • 13. Overview – descriptors •  Introduction •  Modern descriptors •  Comparison and evaluation
  • 14. Overview •  Introduction •  Modern descriptors •  Comparison and evaluation
  • 15. Modern  descriptors   •  Efficient  descriptors   •  Compact  binary  descriptors   •  More  robust  descriptors   •  Learned  descriptors  
  • 16. DAISY   Cita%ons:     •  Op<mized  for  dense  sampling   150    (2012)   •  Log-­‐polar  grid   •  Gaussian  smoothing   •  Dealing  with  occlusions   Engin  Tola,  Vincent  Lepe<t,  and  Pascal  Fua,  DAISY:  An  Efficient  Dense  Descriptor  Applied  to   Wide-­‐Baseline  Stereo,  TPAMI  32(5),  2010.      
  • 17. SURF:   Speeded  Up  Robust  Features   •  Approximate  deriva<ves  with  Haar  wavelets   •  Exploit  integral  images   Cita%ons:     4500    (2012)   Herbert  Bay,  Andreas  Ess,  Tinne  Tuytelaars,  Luc  Van  Gool  "SURF:  Speeded  Up  Robust  Features",   Computer  Vision  and  Image  Understanding  (CVIU),  Vol.  110,  No.  3,  pp.  346-­‐-­‐359,  2008  
  • 18. SURF:   Speeded  Up  Robust  Features   •  Orienta<on  assignment  
  • 19. U-­‐SURF:   Upright  SURF   •  Rota<on  invariance  is  o`en  not  needed.       Don’t  use  more  invariance       than  needed  for  a  given  applica<on       •  Orienta<on  es<ma<on  takes  <me.   •  Orienta<on  es<ma<on  is  o`en  a  source  of   errors.    
  • 20. Beyond  the  classics   •  Efficient  descriptors   •  Compact  binary  descriptors   •  More  robust  descriptors   •  Learned  descriptors  
  • 21. Fast  and  compact  descriptors   •  (Very)  large  scale  applica<ons   –  >  memory  issues,  computa<on  <me  issues   •  Mobile  phone  applica<ons  
  • 22. Fast  and  compact  descriptors   •  Binary  descriptors   •  Comparison  of  pairs  of  intensity  values    -­‐  LBP    -­‐  BRIEF    -­‐  ORB    -­‐  BRISK    
  • 23. LBP:   Local  Binary  Paeerns   •  First  proposed  for  texture  recogni<on  in  1994.   Cita%ons:     2500    (2012)   T.  Ojala,  M.  Pie<käinen,  and  D.  Harwood  (1994),  "Performance  evalua<on  of  texture  measures   with  classifica<on  based  on  Kullback  discrimina<on  of  distribu<ons",  ICPR  1994,  pp.582-­‐585.   M  Heikkilä,  M  Pie<käinen,  C  Schmid,  Descrip<on  of  interest  regions  with  LBP,  Paeern   recogni<on  42  (3),  425-­‐436  
  • 24. BRIEF:   Binary  Robust  Independent   Elementary  Features   •  Random  selec<on  of  pairs   of  intensity  values.   •  Fixed  sampling  paeern   of  128,  256  or  512  pairs.   •  Hamming  distance  to     compare  descriptors  (XOR).   Cita%ons:     149    (2012)   M.  Calonder,  V.  Lepe<t,  C.  Strecha,  P.  Fua,  BRIEF:  Binary  Robust  Independent  Elementary   Features,  11th  European  Conference  on  Computer  Vision,  2010.  
  • 25. D-­‐BRIEF:   Discrimina<ve  BRIEF     •  Learn  linear  projec<ons  that  map  image   patches  to  a  more  discrimina<ve  subspace   •  Exploit  integral  images   T.  Trzcinski  and  V.  Lepe<t,  Efficient  Discrimina<ve  Projec<ons  for  Compact  Binary  Descriptors   European  Conference  on  Computer  Vision  (ECCV)  2012  
  • 26. ORB:   Oriented  FAST  and  Rotated  BRIEF   •  Add  rota<on  invariance  to  BRIEF   Cita%ons:     43    (2012)   •  Orienta<on  assignment  based     on  the  intensity  centroid     Ethan  Rublee,  Vincent  Rabaud,  Kurt  Konolige,  Gary  Bradski,  ORB:  an  efficient  alterna<ve  to  SIFT   or  SURF,  ICCV  2011  
  • 27. ORB:   Oriented  FAST  and  Rotated  BRIEF   •  Select  a  good  set  of  pairwise  comparisons:   minimize  correla<on  under  various   orienta<on  changes   High  variance     High  variance  +  uncorrelated    
  • 28. BRISK:     Binary  Robust  Invariant  Scalable   Keypoints   Cita%ons:     •  Regular  grid       20    (2012)   •  Orienta<on  assignment  based  on  dominant   gradient  direc<on  (using  long-­‐distance  pairs)   •  512  bit  descriptor  based  on   short-­‐distance  pairs   Stefan  Leutenegger,  Margarita  Chli  and  Roland  Y.  Siegwart,  BRISK:  Binary  Robust  Invariant   Scalable  Keypoints,  ICCV  2011  
  • 29. Various  others   •  FREAK:  Fast  Re<na  Keypoint   •  CARD:  Compact  and  Real<me  Descriptor   •  LDB:  Local  Difference  Binary      
  • 30. FREAK:     Fast  Re<na  Keypoint       •  Inspired  by  the  human  visual  system   A.  Alahi,  R.  Or<z,  and  P.  Vandergheynst.  FREAK:  Fast  Re<na  Keypoint.  In  IEEE  Conference  on   Computer  Vision  and  Paeern  Recogni<on  2012.  
  • 31. CARD:     Compact  and  Real<me  Descriptor   –  Look  up  tables     –  learning-­‐based  sparse  hashing  
  • 32. LDB:   Local  Difference  Binary   Xin  Yang  and  Kwang-­‐Ting  Cheng,  LDB:  An  Ultra-­‐Fast  Feature  for  Scalable  Augmened  Reality  on   Mobile  Devices,  Interna<onal  Symposium  on  Mixed  and  Augmented  Reality  2012  (ISMAR  2012)  
  • 33. Beyond  the  classics   •  Efficient  descriptors   •  Compact  binary  descriptors   •  More  robust  descriptors   •  Learned  descriptors  
  • 34. LIOP:   Local  Intensity  Order  Paeern  for   Feature  Descrip<on   (and  predecessors  MROGH  and  MRRID)   •  Robustness  to  monotonic  intensity  changes   •  Data-­‐driven  division  into  cells   Zhenhua  Wang  Bin  Fan  Fuchao  Wu,  Local  Intensity  Order  Paeern  for  Feature  Descrip<on,  ICCV   2011.  
  • 35. LIOP:   Local  Intensity  Order  Paeern  for   Feature  Descrip<on  
  • 36. Beyond  the  classics   •  Efficient  descriptors   •  Compact  binary  descriptors   •  More  robust  descriptors   •  Learned  descriptors  
  • 37. Winder  &  Brown   •  Learn  configura<on  and  other  parameters   from  training  data  obtained  from  3D   Cita%ons:     reconstruc<ons   194  (2012)   M.  Brown,  G.  Hua  and  S.  Winder,  Discriminant  Learning  of  Local  Image  Descriptors..   IEEE  Transac<ons  on  Paeern  Analysis  and  Machine  Intelligence.  2010.  
  • 38. Winder  &  Brown   •  Training  data  =  set  of  corresponding  image   patches  
  • 39. Descriptor  Learning     Using  Convex  Op<misa<on   •  Convex  learning  of   Pre-­‐rec<fied     keypoint  patch     –  spa<al  pooling  regions   –  dimensionality  reduc<on   Non-­‐linear  transform   •  Learning  from  very  weak     supervision   learning   Spa<al  pooling     Normalisa<on  and   cropping   Dimensionality   learning   reduc<on   Descriptor  vector   K.  Simonyan  et  al.,  Descriptor  Learning  Using  Convex  Op<misa<on,  ECCV  2012  
  • 40. Learning  Spa<al  Pooling  Regions   •  Selec%on  from  a  large  pool  using  L1  regularisa%on   •  So`-­‐margin  constraints:     –  squared  L2  distance  between  descriptors  of  matching  feature   pairs  should  be  smaller  than  that  of  non-­‐matching  pairs   •  Convex  objec<ve  (op<mised  with  a  proximal  method)   768-­‐D   576-­‐D   320-­‐D   pooling  region  configura%ons  learnt  with  different  levels  of  sparsity  
  • 41. Overview •  Introduction •  Modern descriptors •  Comparison and evaluation
  • 42. Sepng  up  an  evalua<on   •  Which  problem?  Performance  in  different  applica<on/niches  may   vary  significantly.   –  Category  recogni<on,     –  Matching,     –  Retrieval     •  What  dataset?   –  Pascal  VOC  2007   –  Oxford  image  pairs   –  Oxford  -­‐  Paris  buildings   •  Protocol  and  criteria?   –  Public  dataset,     –  Avoiding  risk  to  over-­‐fipng/op<mizing  to  the  data    
  • 43. Detector  evalua<ons   homography A B B Two points are correctly matched if # correct matches precision = T=40% # all matches A∩ B >T A∪ B # correct matches recall = # ground truth correspondences
  • 44. Previous  Evalua<ons   •  2D  Scene  –  Homography   –  C.  Schmid,  R.  Mohr,  and  C.  Bauckhage,  “Evalua<on  of  interest  point   detectors,”  IJCV,  2000.   –  K.  Mikolajczyk  and  C.  Schmid,  “A  performance  evalua<on  of  local  descriptors,”   CVPR,  2003.   –  T.  Kadir,  M.  Brady,  and  A.  Zisserman,  “An  affine  invariant  method  for  selec<ng   salient  regions  in  images,”  in  ECCV,  2004.   –  K.  Mikolajczyk,  T.  Tuytelaars,  C.  Schmid,  A.  Zisserman,  J.  Matas,  F.   Schaffalitzky,T.  Kadir,  and  L.  Van  Gool,  “A  comparison  of  affine  region   detectors,”  IJCV,  2005.   –  A.  Haja,  S.  Abraham,  and  B.  Jahne,  Localiza<on  accuracy  of  region  detectors,   CVPR  2008     –  T.  Dickscheid,    FSchindler,  Falko,  W.  Förstner,  Coding  Images  with  Local   Features,  IJCV  2011    
  • 45. Previous  Evalua<ons   •  3D  Scene  -­‐  epipolar  constraints   –  F.  Fraundorfer  and  H.  Bischof,  “Evalua<on  of  local  detectors  on  non-­‐planar,   scenes,”  in  AAPR,  2004.     –  P.  Moreels  and  P.  Perona,  “Evalua<on  of  features  detectors  and  descriptors   based  on  3D  objects,”  IJCV,  2007.   –  S.  Winder  and  M.  Brown,  “Learning  local  image  descriptors,”  CVPR,   2007,2009.   –  Dahl,  A.L.,  Aanæs,  H.  and  Pedersen,  K.S.  (2011):  Finding  the  Best  Feature   Detector-­‐Descriptor  Combina<on.  3DIMPVT,  2011.      
  • 46. Recent  Evalua<ons   •  Recent  detectors   O.  Miksik  and  K.  Mikolajczyk,  Evalua<on  of  Local  Detectors  and  Descriptors  for  Fast  Feature   Matching,  ICPR  2012   ECCV  2012    Modern  features:    46/60   …  Detectors.  
  • 47. Recent  detector  evalua<ons   •  Completeness  (coverage),  complementarity  between   detectors     T.  Dickscheid,    FSchindler,  Falko,  W.  Förstner,  Coding  Images  with  Local  Features,  IJCV  2011   –  Completeness,  Edgelap  (Mikolajczyk),  Salient  (Kadir),  MSER  (Matas)   –  Complementrity,  MSER  +  SFOP    
  • 48. Descriptor  Evalua<ons   •  Matching  precision  and  recall   O.  Miksik  and  K.  Mikolajczyk,  Evalua<on  of  Local  Detectors  and  Descriptors  for  Fast   Feature  Matching,  ICPR  2012  
  • 49. Recent  Descriptor  Evalua<ons   •  Computa%on  %mes  for  the  different  descriptors  for  1000   SURF  keypoints   O.  Miksik  and  K.  Mikolajczyk,  Evalua<on  of  Local  Detectors  and  Descriptors  for  Fast  Feature   Matching,  ICPR  2012   J.  Heinly  E.  Dunn,  J-­‐M.  Frahm,  Compara<ve  Evalua<on  of  Binary  Features,  ECCV2012    
  • 50. Previous  Evalua<ons   •  Image/object  categories   –  K.  Mikolajczyk,  B.  Leibe,  and  B.  Schiele,  “Local  features  for  object  class   recogni<on,”  in  ICCV,  2005   –  E.  Seemann,  B.  Leibe,  K.  Mikolajczyk,  and  B.  Schiele,  “An  evalua<on  of  local   shape-­‐based  features  for  pedestrian  detec<on,”  in  BMVC,  2005.   –  M.  Stark  and  B.  Schiele,  “How  good  are  local  features  for  classes  of  geometric   objects,”  in  ICCV,  2007.   –  K.  E.  A.  van  de  Sande,  T.  Gevers  and  C.  G.  M.  Snoek,  Evalua<on  of  Color   Descriptors  for  Object  and  Scene  Recogni<on.  CVPR,  2008.    
  • 51. Approach   •  Bags-­‐of-­‐features   1.  Interest  point  /  region  detector   2.  Descriptors   3.  K-­‐means  clustering  (4000  clusters)   4.  Histogram  of  cluster  occurrences  (NN  assignment)   5.  Chi-­‐square  distance  and  RBF  kernel  for  KDA  or  SVM  classifier     •   J.  Zhang  and  M.  Marszalek  and  S.  Lazebnik  and  C.  Schmid,     Local  Features  and  Kernels  for  Classifica<on  of  Texture  and  Object  Categories:     A  Comprehensive  Study,  IJCV,  2007   •   K.  E.  A.  van  de  Sande,  T.  Gevers  and  C.  G.  M.  Snoek,     Evalua<on  of  Color  Descriptors  for  Object  and  Scene  Recogni<on.  CVPR,  2008  
  • 52. Evalua<on   •  PASCAL  VOC  measures   –  Average  precision  for  every  object  category   –  Mean  average  precision     Category   AP   output=>   precision   AP   recall  
  • 53. MAP  Ranking   color/gray,  density,  dimensionality  ...   MAP   #dimensions   density   •  SIFT  s<ll  dominates  (Histograms  of  gradient  loca<ons  and  orienta<ons)   •  Opponent  chroma<c  space  (normalized  red-­‐green,  blue-­‐yellow,  and  intensity  Y  
  • 54. Grayvalue  descriptors   MAP  Ranking   #dimensions   density     •  Observa<ons   •  Color  improves   •  All  based  on  histograms  of  gradient  loca<ons  and  orienta<ons   •  Dimensionality  not  much  correlated  with  the  performance   •  Density  Strongly  correlated  (the  more  the  beeer)   •  Results  biased  by  density   •  Implementa<on  details  maeer