Tutorial from PyData London: An Introduction to Video Action Recognition by Bertrand Nouvel, Jonathan Kelsey and Bernard Hernandez from Wide IO

1. Introduction to
Action Recognition
in Python
@wideio
Bertrand NOUVEL, bn@wide.io
Jonathan KELSEY, jk@wide.io
Bernard HERNANDEZ, bh@wide.io
PYDATA LONDON 2014

2. Outline
- (While you download the data) Forewords & Overview
- PART 1: Video-processing in python
- PART 2: The pipeline in details
- PART 3: Putting it together
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

3. FOREWORDS
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

4. GETTING READY
1)
2)
GET THE ABSOLUTELY ESSENTIAL PACKAGES :
Python 2.x / 3.x
Numy/Scipy
PIL
OpenCV2
ANACONDA
WAKARI
GET THE SOURCE CODE
git clone https://bitbucket.org/wideio/pydata.git
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

5. WHO WE ARE
WIDE IO - Democratising the best algorithms
startup
+
consultancy
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

6. Why do Computer Vision in Python?
Reflexive
Multiparadigm
Computer vision is difficult.
“””
Readable
Python is 1000 slower than C++
Lots of packages
Lightweight
Best community
Operator
overloading
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

7. ACTION RECOGNITION
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

8. What is action recognition ?
Classification task
SEMANTIC GAP
KTH, Human action dataset (Laptev)
Classify actions
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

9. Types of Systems
FEATURE BASED
ACTION SPECIFIC
HOLISTIC
APPROACHES
APPEARANCE
BASED
Many priors
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
DEEP LEARNING
Less priors

10. The Very-Traditional Pipeline
RAW DATA
FEATURE-EXTRACTION
MACHINE LEARNING
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

11. The Very-Traditional Pipeline
VIDEO
FRAMES
SPARSIFICATION
CLASSIFICATION
METHOD
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

12. The Very-Traditional Pipeline
VIDEO
FRAMES
BAGS OF KEYPOINTS
SVM
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

13. PART I
PROCESSING
VIDEOS
TOWARDS FRAMEWORKS

14. Reading Videos
-
PYFFMPEG (outdated)
-
OPENCV
MLT
-
ON WAKARI
anaconda$ vi io/player_mlt.py
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

15. Reading Videos
-
PYFFMPEG (outdated)
-
OPENCV
MLT
-
ON WAKARI
anaconda$ vi io/player_cv.py
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

16. Displaying Images
Alternatives
Recommended
anaconda$
python io/display_pyglet.py
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

17. Harris Corner Detector
Detecting interest points:
imsmooth=scipy.ndimage.gaussian_filter
def harris(I,alpha=0.04,si=1):
Ix,Iy = scipy.gradient(I)
H11 = imsmooth(Ix*Ix, si)
H12 = imsmooth(Ix*Iy, si)
H22 = imsmooth(Iy*Iy, si)
return ((H11*H22 - H12**2)
- alpha*(H11+H22)**2)
anaconda$
python keypoints/harris.py
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

18. Optical flow - wrapper based code
anaconda$
python keyponats/custom_feature.py
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

19. Spatio-temporal keypoints
Use the two previous elements to compute keypoints that contain information about the movement
anaconda$
python keyponats/custom_feature.py
Ideas for extension: Make the same with a pyramidal approach.
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

20. PART II
UNDERSTANDING THE KEY
ELEMENTS OF THE PIPELINE

21. Feature extractions
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

22. Feature extractions
SIFT (128)
Descriptor vector
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
SURF (64)

23. Algorithms:
Type (density, connectivity, …)
Dimension (descriptor, position, …)
Recursive clustering
Connectivity (ward)
Centroid (k-means)
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
Density (optics)

24. WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

25. WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

26. WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

27. Support Vector Machines
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

28. Support Vector Machines
Weighted-Support Vector Machines
Different support
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

29. Support Vector Machines
Weighted-Support Vector Machines
Different support
Different relevance (outliers)
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

30. PART III
PUTTING EVERYTHING
TOGETHER

31. Feature detection
PCA Projection
Clustering
Eigen Vectors & Mean
Graphic Model
Model.model
centers
BOW
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

32. Feature detection stack for image
PCA project with graphic model
Distance of projection to BOW cluster centers with metric
Invert with 'discriminator' to turn into weighting
Normalise (Area=1)
Mean of stack to create histogram
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

33. Training data
Training Labels
Testing data
Testing labels
Feature computer
SVM predict all
Feature stack
SVM auto train
Param grid
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL

34. Introduction to
Action Recognition
in Python
@wideio
Bertrand NOUVEL, bn@wide.io
Jonathan KELSEY, jk@wide.io
Bernard HERNANDEZ, bh@wide.io
PYDATA LONDON 2014