"Multiple Uses of Pipelined Video Pre-Processor Hardware in Vision Applications," a Presentation from Analog Devices

Copyright © 2014 Analog Devices Inc. 1
Rajesh Mahapatra
May 29, 2014
Multiple Uses of Pipelined Video Pre-
processor Hardware in Vision
Applications

ADI is an established global leader with unparalleled signal processing expertise and a
deep commitment to our customers’ success.
ADI’s revenue base is comprised of a wide range of industries and applications, with a
deep focus on signal processing products and technologies.
• Global Leader
• Leader in high-performance signal processing product categories across all end markets and
geographies
• Founded in 1965 to produce modular precision op amps
• Obsessive vision to help engineers solve signal processing challenges
• ADI is NASDAQ listed and a component of the S&P 500
• 2013 revenue of $2.6 billion
• Who we are
• ADI’s culture of innovation is driven by a desire to
make our 60,000 plus customers successful at
gaining a competitive edge in their markets
• More than half of ADI employees are engineers
• Dedicated to Breakthrough Innovations by providing:
• Complete signal chain solutions
• System-level integration and software
• Superior services and user experiences
ADI At a Glance

• What are these Computer Vision Applications?
• Traffic Sign Speed Limit Recognition
• Face Detection
• Defect Detection
• Iris Detection
• Brief introduction to ADSP-BF60x and PVP (Pipelined Vision Processor)
• Some ways to solve these problems for applications
• Analyze how PVP can be leveraged to solve traditional approach with
massive gain in performance
Introduction

Traffic Sign Recognition — Speed Limit
90
A Speed limit recognition system is expected to detect
the speed limit sign
90

• Face Detection
• Localize the problem for Face Recognition
• Enable/Disable device operations based on whether a person is
looking at the device or not
• People Counting
• Camera focus
• Many others…
Face Detection

• Defect Detection
• Production Assembly Line
• Sorting
• Iris Detection
• ROI for Iris recognition
• Eye movement tracking
Defect Detection & Iris Detection

ADSP-BF609/8/7/6 Block Diagram
Part of Video Subsystem Part of Safety Concept For Industrial & Automotive

PVP Functional Blocks
InputDataFormatters
Conv 1
Conv 2
Conv 3
Conv 4
Arithmetics
Cartesian
to
Polar
Threshold
Threshold
Edge
Classifier
Integral
Image 2
Up/Down
Scaler
OutputDataFormatters
Memory
Pipe
Camera
Pipes
 Each PVP Block can be assigned to either
 Camera Pipes
 Or Memory Pipe
 Each PVP Block can be used only once
Integral
Image 1

Serialization of the Memory Pipe Tasks (1/3)
InputDataFormatters
Conv
Conv
Cartesian
to
Polar
Threshold
Camera
Threshold Binarization
Edge
Classifier
Lane Departure Warning
People DetectionIntegral
Image
 Memory Pipe
 one input DMA and one output DMA
 can be time shared by multiple tasks
Memory
Template MatchingUp / Down
Scaler

Serialization of Memory Pipe Tasks (2/3)
InputDataFormatters
Conv
Conv
Cartesian
to
Polar
Threshold
Camera
Edge
Classifier
people DetectionIntegral
Image
 Memory Pipe
Memory
ConvolutionConv

Serialization of Memory Pipe Tasks (3/3)
InputDataFormatters
Conv
Conv
Cartesian
to
Polar
Threshold
Camera
Edge
Classifier
People DetectionIntegral
Image
 Memory Pipe
Memory
Arithmetics
DivisionConv

• PVP was architected to be a flexible image processing engine
• Ability to run up to five concurrent vision algorithms
• Generally useful functional blocks include convolution, scalers, and
arithmetic blocks
• Optimized to save memory bandwidth
• PVP provides acceleration into the following 3 major areas:
• Object Detection
• Object Classification
• Object Recognition
• All of these functions are used in Driver Assistance, Security &
surveillance, machine vision, optical character recognition,
biometrics….
PVP Accelerator : 8+ GMACs < 80mW
Useful for multiple & concurrent analytics functions

Other implementations often approximate magnitude by
PVP does real
22
yxM 
yxM 
)arctan(
x
y

16 bits
5 bits
Absolute Magnitude and Gradient of Edge
Sobel Edge Detection: Cartesian-to-Polar
Transform
5x5 CNV
Sobel Sx
5x5 CNV
Sobel Sy
Polar Magnitude
and Angle Unit
(PMA)
THC
Binary ThresholdPPI
Edge Map
|H|
θ
Horizontal Component of Gradient
Vertical Component of Gradient
1= Edge
0= No Edge

Problem Statement — 1
90
A Speed limit recognition system is expected to
detect the speed limit sign
90

Solution — Step 1 & 2 — Find Edge and Circle
• Edge Detection
• Sobel or Canny Edge Detectors (OpenCV functions)
• Circle Detection
• OpenCV — Hough Transform for Circle
• Shape Detector — ADI software

• One solution is to use a classifier — OpenCV functions
• Treat speed limit recognition as a statistical pattern recognition
problem
• There will be a feature generator and a classifier
Solution Step 3 — Detect Numbers or Signs
Feature
Generator
Classifier
Features
70

Edge
Detection
Training
Database
Circle
Detection
Feature
Generation
Classification Tracking
Pre-Processing
Post-Processing
Grey
scale
Input
TSR
Output
• Edge pixels are typically a small percentage of input pixel — say 10%
• Core needs to process this small percentage only
• Edge Detection can be done by Sobel or Canny. Needs to run on all input pixels or
all pixels in a ROI.

Math Operations Required
Edge
Detection
Training
Database
Circle
Detection
Feature
Generation
Pre-Processing
Post-Processing
Grey
scale
Input
TSR
Output
5x5 CNV
Sobel Sx
5x5 CNV
Sobel Sy
Polar Magnitude
and Angle Unit
(PMA)
THC
Binary ThresholdPPI
Edge Map
|H|
θ
Gradient
Map
Hx Sobel
5x5
Hy Sobel
5x5
Cartesian
To
Polar
Gx
thetaGy
rho
Non-Max
Suppression
+
Hysteresis
Threshold
Gaussian
Lowpass
5x5
Functionality Math ops for HD
(1280x960) Sensor
5x5 Gaussian 921M MAC
Sobel Hx and Hy 1800M MAC
Cordic Magnitude and
Phase
Square root + arc
tan (Theta)
16 Level Threshold
plus RLE
Non max suppression

Can We Make Solutions Easier Using PVP?
We can use 2 convolution blocks and one Cartesian to Polar block
and one threshold block for our operation
InputDataFormatters
Conv 1
Conv 2
Conv 3
Conv 4
Arithmetics
Cartesian
to
Polar
Threshold
Threshold
Edge
Classifier
Integral
Image 2
Up/Down
Scaler
OutputDataFormatters
Integral
Image 1

Edge
Detection
Training
Database
Circle
Detection
Feature
Generation
PVP Core
Grey
scale
Input
TSR
Output
(1280x960) Sensor
5x5 Gaussian 921M MAC
Phase
Square root + arc
tan (Theta)
16 Level Threshold
plus RLE
Non max suppression
Speed Limit Recognition H/W S/w Partition

Edge
Detection
Circle
Detection
Outer
Boundary
Detection
Pre-Processing
Post-Processing
PVP
Core
Iris Detection — H/w S/w partiiton

Face Detection H/w S/w Partition
5x5 CNV
Sobel Sx
5x5 CNV
Sobel Sy
Polar Magnitude
and Angle Unit
(PMA)
THC
Binary ThresholdPPI
Edge Map
|H|
θ
(1280x960) Sensor
Phase
Square root + arc
tan (Theta)
16 Level Threshold
plus RLE
Non max suppression
Edge
Detection
Training
Database
Object Detector
Pre-Processing Post-Processing
FD
OutputIntegral
Integral

• The techniques we have used in this presentation can also be applied
for other applications such as
• Pedestrian detection using Histogram of Gradient approach
• Somnolence Detection kind of application
• License Plate Recognition
• Face Detection
• Finding angle of random rotation for currency sorting
• These techniques are standard techniques and can be applied in
multiple vendor solutions
Other Applications

• Having a standard set of data base that can be used for commercial
applications for validating the solutions would be truly awesome
• Having standard benchmarks for performance would also make our work
lot easier.
“Wish List”

• Several vision applications like Defect detection, Face detection are
complex to be implemented on embedded solutions. And usually have
other additional processing needs on processors.
• Effective partitioning of the complexity between hardware and
software will lead to
• hardware doing a bulk of the brute force fixed operations “for
free”. Software overhead is almost zero for this.
• Software focuses on results of pre-processing. Hence works on
smaller data and can do lot more complex and concurrent
operations.
• PVP and the dual core on Analog Devices ADSP-BF609 supports this
partitioning.
• Software Solutions from Analog Devices makes it easier to build these
applications.
Lessons Learned

• Many software modules like Video Codecs, Audio Codecs, Speech
Codecs, Video and Audio pre and post processors, VOIP modules
available for free download from
• http://www.analog.com/BlackfinModules
• http://www.analog.com/SharcModules
• For questions mail to processor.tools.support@analog.com
• Access Engineer Zone community at http://ez.analog.com/
Please visit demos booths of Analog Devices 
Resource Slide

Backup Slides

HOG Based Object Detection
Blackfin CorePVP
Hx Sobel
CNV0
HoG
features
ClassifierPMA PEC
theta
rho
4-directions
Hy Sobel
CNV1
Quad
8-bit IIM
Blackfin CorePVP
Hx Sobel
CNV0
HoG
features
Classifier
PMA PEC
theta
rho
32-directions
Hy Sobel
CNV1
1 2 3 4 5 6 7 8
0
2
4
6
8
10
12
Compute
Histogram
HoG = Histogram of Gradients

"Multiple Uses of Pipelined Video Pre-Processor Hardware in Vision Applications," a Presentation from Analog Devices

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a "Multiple Uses of Pipelined Video Pre-Processor Hardware in Vision Applications," a Presentation from Analog Devices

Semelhante a "Multiple Uses of Pipelined Video Pre-Processor Hardware in Vision Applications," a Presentation from Analog Devices (20)

Mais de Edge AI and Vision Alliance

Mais de Edge AI and Vision Alliance (20)

Último

Último (20)

"Multiple Uses of Pipelined Video Pre-Processor Hardware in Vision Applications," a Presentation from Analog Devices