1. Beetlebox provides the ClickCV computer vision library and tools to develop real-time low latency computer vision applications using FPGAs without requiring hardware expertise. 2. FPGAs offer high performance through parallel processing and precise control over latency, throughput, and power consumption compared to CPUs, but traditionally required hardware expertise to program. 3. Xilinx's Vitis tools including the Vitis Vision library, Vitis AI, and unified software development platform allow development of FPGA applications using C/C++ and popular frameworks like OpenCV, reducing the need for hardware expertise.

1. Andrew Swirski,
Founder and Executive Director,
a.swirski@beetlebox.org
Real Time Low Latency Computer Vision
Unit 1. 10,
Chester House,
Kennington Park,
1-3 Brixton Road,
London,
SW9 6DE

2. • Meeting the increasing demands of Computer Vision
• What are FPGAs and why do they perform better than
CPUs
• Why in previous years FPGAs have been restricted to
hardware engineers
• The new generation of Xilinx software development tools:
• Vitis Unified Software Development platform
• Vitis Vision Library
• Vitis AI
• ClickCV: Beetlebox’s computer vision library
• Electronic Image Stabilisation with Sundance
• Getting involved with our Early Access
Introduction

3. Our vision of the future puts enormous
demand on embedded devices to understand
the world around them
Autonomous delivery drone
Our vision of the future

4. What do we mean by high performance?
1. Throughput:
• Achievable Resolution
• Achievable Frames Per Second (FPS)
2. Latency:
• Consistent end-to-end latency
3. Power Consumption:
• Battery Life
• Heat Production
What is high performance?

5. What are FPGAs
What are FPGAs
Inputs/Outputs
Memory
Logic
DSP Engines

6. What happens on a CPU
int main() {
int a[5], b[5], output[5];
for(int i=0;i<5;i++){
output[i]=a[i]+b[i];
}
}
main:
str fp, [sp, #-4]!
add fp, sp, #0
sub sp, sp, #68
mov r3, #0
str r3, [fp, #-8]
.L3:
ldr r3, [fp, #-8]
cmp r3, #4
bgt .L2
ldr r3, [fp, #-8]
lsl r3, r3, #2
sub r2, fp, #4
add r3, r2, r3
ldr r2, [r3, #-24]
ldr r3, [fp, #-8]
lsl r3, r3, #2
sub r1, fp, #4
add r3, r1, r3
ldr r3, [r3, #-44]
add r2, r2, r3
ldr r3, [fp, #-8]
lsl r3, r3, #2
sub r1, fp, #4
add r3, r1, r3
str r2, [r3, #-64]
ldr r3, [fp, #-8]
add r3, r3, #1
str r3, [fp, #-8]
b .L3
.L2:
mov r3, #0
mov r0, r3
add sp, fp, #0
ldr fp, [sp], #4
bx lr
• Variable latency

7. What happens on a FPGA
int main() {
int a[5], b[5], output[5];
for(int i=0;i<5;i++){
output[i]=a[i]+b[i];
}
} + +
a[0] b[0] a[1] b[1] a[2] b[2] a[3] b[3] a[4] b[4]
+ + +
output[0] output[1] output[2] output[3] output[4]
• Control over the design
• Control over the latency
• Control over the clock (power consumption)

8. With great control comes great responsibility
• Need hardware specialists:
• Hardware Description Languages: Verilog or VHDL
• Design and Verification is a time-consuming
process
• Newer tools (High Level Synthesis) do allow for
the uses of C/C++, but:
• Limited subset of C
• No standard libraries
• Need to still understand the hardware
Restricted to Hardware Engineers

9. The new generation of tools are now focused on System-on-
Chip (SoC)
• Xilinx’s Zynq series and the Zynq Ultrascale+ series contain
ARM cores
• Benefits of a CPU host code + FPGA accelerated code
The new generation of tools
CPU FPGA

10. Providing a familiar software development
environment
• Develop, compile and debug code using C, C++ or OpenCL
either through the provided IDE or using a makefile build
flow
• Embedded platforms run a version of Linux known as
PetaLinux
• Use Standard Libraries and Tools such as GStreamer,
OpenCV and FFMPEG
• Gain access to open-source Xilinx Vitis Accelerated
Libraries
Xilinx Vitis Unified Software Platform

11. Vitis Vision library provides a library of low-
level computer vision kernels
• Provides a strong video pipeline framework
• Great for forming basic pipelines: e.g. colour thresholding
Xilinx Vitis Vision Library
CPU FPGA CPU
Collect
Data
Computer
Vision
Pipeline
Output
Data
30 FPSVideo
In

12. What we can do with Vitis Vision
Xilinx Vitis Vision Library
Edge Detection Colour Detection

13. Allows the deployment of models from deep
learning frameworks such as Tensorflow and
Caffe on to a specialized processor
• Optimizes your model to run on FPGAs:
• Prune
• Quantisation
Xilinx Vitis AI Development Environment
CPU FPGA CPU
Collect
Data Pre-process
Output
Data
30 FPSVitis AI
DPU

14. What we can do with Vitis AI
Xilinx Vitis AI
Object prediction Bounding Boxes Semantic Segmentation

15. High performance, high level FPGA kernels without the need
for hardware expertise
• Provides high-level, out-of-the-box functionality using
software only to get computer vision developed faster
• Also provides the low-level kernels needed to build
custom systems or for building test systems
• Integrates with industry standard software such as
OpenCV and GStreamer
Beetlebox ClickCV: Accelerated Computer Vision

16. Partnered with Sundance Microprocessor technologies
• Robotics and vision hardware experts
• Robots and drones often suffer from
shaky camera, especially when there
is no room or power for a gimbal
• Cameras with in-built stabilization,
such as GoPros are often impractical
• Stabilise video purely through the
video itself
• No sensor data
Electronic Image Stabilisation (EIS)
VCS-1

17. Electronic Image Stabilisation (EIS)

18. Currently running the ClickCV Early Access Programme
• High Performance with no need for computer vision hardware
specialists:
• Out-of-the-box functionality
• We can develop bespoke hardware for a specific solution
• Provide custom systems to get testing on FPGAs up and
running fast
• Let your software engineers use industry-standard tools, such
as OpenCV, GStreamer and FFMPEG
• Areas where we are looking next:
• Super Resolution
• Template Matching
• SLAM
• Deep Learning
ClickCV Early Access

19. Our Team
Andrew Swirski
MD
• MEng Electronic
Engineering at Imperial
College London
• Design Engineer at Intel
Matthew Simpson
Head of Systems
Dr Christos-Savvas Bouganis
Consultant
• Reader and Director of the
Integrated Digital Systems Lab
at Imperial College:
• Computer Vision, Image
Processing, Machine Learning
and SLAM on FPGAs
• MSc Electronic
Engineering at Imperial
College London
• CMOS Engineer at NXP
Ashley Unitt
Advisor
• CTO and CSO of
NewVoiceMedia, a SaaS
contact centre tech provider
• Acquired by Vonage
Peter Collins
Advisor
• CEO of Permasense Ltd-
sensor systems; acquired by
Emerson
• Chairman Inflowmatix Ltd
• Chariman Guided Ultrasonics
Ltd
Dr Marlon Wijeyasinghe
Head of HLS
• PhD in Electronic
Engineering at Imperial
College London

20. Contribute back and grow the community
• Technical tutorials: Vitis Vision Library tutorial:
• https://beetlebox.org/getting-started-with-computer-vision-for-
vitis-embedded-systems/
• Explainer articles: What is Computer Vision and why are
Neural Networks so important:
• https://beetlebox.org/what-is-computer-vision-and-why-are-
neural-networks-important/
• Soon launching a tutorial on Vitis AI, where we explore
sign language recognition
• Open Source
• Soon launching our github.io page:
• beetlebox.github.io
Free resources

21. • Previously FPGAs have always been a powerful but
obscure chip that only a few hardware specialists can use
• With Xilinx’s new focus on SoCs and Vitis software
development environment, embedded FPGAs have never
been more accessible
• ClickCV Computer Vision library provides out-of-the-box
high performance functionality allowing the fast
development of systems without the need for hardware
expertise
• Currently running an Early Access Programme. Come talk
to us!
• Check out our website
Conclusion

22. Andrew Swirski,
Founder and Executive Director,
a.swirski@beetlebox.org
Any Questions?
Unit 1. 10,
Chester House,
Kennington Park,
1-3 Brixton Road,
London,
SW9 6DE