Next generation accelerated AIoT systems and applications. Pedro Trancoso. Special Session on EU Projects, co-located with Computing Frontiers 2023, Bologna, Italy, May 2023

1. K. Mika, R. Griessl, N. Kucza, F. Porrmann, M.
Kaiser, L. Tigges, J. Hagemeyer, P. Trancoso,
M.Waqar, F. Qararyah, S. Zouzoula, J. Ménétrey,
M. Pasin, V. Schiavoni, P. Felber, C. Marcus, O.
Brunnegard, O. Eriksson, H. Salomonsson, D.
Ödman, A. Ask, A. Casimiro, A. Bessani, T.
Carvalho, K. Gugala, P. Zierhoffer, G. Latosinski,
M. Tassemeier, M. Porrmann, H.-M. Heyn, E.
Knauss, Y. Mao, F. Meierhöfer
Next generation accelerated AIoT
systems and applications
10. May 2023

2. 2
Overview

3. 3
Overview
Applications
Requirements Security & Safety
Hardware
Plattforms
Microserver &
Accelerators
Middleware
Embedded/
Far Edge
Near Edge Cloud
Safety
&
Robustness
Modelling
&
Verification
Jetson AGX
NVIDIA Xavier
COM-HPC
Xilinx Zynq
UltraScale+
SMARC
Xilinx Zynq
UltraScale
Coral SoM
Xilinx
Kria
RPi CM4
ARVSOM
Smart Home Industrial IoT Automotive AI
Open
Call
Monitoring
Trusted
Execution
&
Communication
RISC-V
extensions
Toolchain Emulation Benchmarking & Deployment
uRECS t.RECS RECS|Box

4. 4
Overview
Applications
Requirements Security & Safety
Hardware
Plattforms
Microserver &
Accelerators
Middleware
Embedded/
Far Edge
Near Edge Cloud
Safety
&
Robustness
Modelling
&
Verification
Jetson AGX
NVIDIA Xavier
COM-HPC
Xilinx Zynq
UltraScale+
SMARC
Xilinx Zynq
UltraScale
Coral SoM
Xilinx
Kria
RPi CM4
ARVSOM
Smart Home Industrial IoT Automotive AI
Open
Call
Monitoring
Trusted
Execution
&
Communication
RISC-V
extensions
Toolchain Emulation Benchmarking & Deployment
uRECS t.RECS RECS|Box

5. 5
Overview
Applications
Requirements Security & Safety
Hardware
Plattforms
Microserver &
Accelerators
Middleware
Embedded/
Far Edge
Near Edge Cloud
Safety
&
Robustness
Modelling
&
Verification
Jetson AGX
NVIDIA Xavier
COM-HPC
Xilinx Zynq
UltraScale+
SMARC
Xilinx Zynq
UltraScale
Coral SoM
Xilinx
Kria
RPi CM4
ARVSOM
Smart Home Industrial IoT Automotive AI
Open
Call
Monitoring
Trusted
Execution
&
Communication
RISC-V
extensions
Toolchain Emulation Benchmarking & Deployment
uRECS t.RECS RECS|Box

6. 6
Overview
Applications
Requirements Security & Safety
Hardware
Plattforms
Microserver &
Accelerators
Middleware
Embedded/
Far Edge
Near Edge Cloud
Safety
&
Robustness
Modelling
&
Verification
Jetson AGX
NVIDIA Xavier
COM-HPC
Xilinx Zynq
UltraScale+
SMARC
Xilinx Zynq
UltraScale
Coral SoM
Xilinx
Kria
RPi CM4
ARVSOM
Smart Home Industrial IoT Automotive AI
Open
Call
Monitoring
Trusted
Execution
&
Communication
RISC-V
extensions
Toolchain Emulation Benchmarking & Deployment
uRECS t.RECS RECS|Box

7. 7
Overview
Applications
Requirements Security & Safety
Hardware
Plattforms
Microserver &
Accelerators
Middleware
Embedded/
Far Edge
Near Edge Cloud
Safety
&
Robustness
Modelling
&
Verification
Jetson AGX
NVIDIA Xavier
COM-HPC
Xilinx Zynq
UltraScale+
SMARC
Xilinx Zynq
UltraScale
Coral SoM
Xilinx
Kria
RPi CM4
ARVSOM
Smart Home Industrial IoT Automotive AI
Open
Call
Monitoring
Trusted
Execution
&
Communication
RISC-V
extensions
Toolchain Emulation Benchmarking & Deployment
uRECS t.RECS RECS|Box

8. 8
 Heterogenenous hardware platform
 Resource Efficient Cluster Server (RECS)
platform: cloud to edge
Accelerated AIoT Platform

9. 9
 Heterogenenous hardware platform
 Resource Efficient Cluster Server (RECS)
platform: cloud to edge
Accelerated AIoT Platform cloud
edge

10. 10
 Heterogenenous hardware platform
 Resource Efficient Cluster Server (RECS)
platform: cloud to edge
 u.RECS for far edge with 3 slots:
 NVIDIA NX – embeded GPU
 SMARC 2.1 – FPGA, CPU
 M.2 – dedicated accelerator
Accelerated AIoT Platform
USB 3
power sensing
GPIO
CSI
USB 3
USB 3 Mux
SMARC 2.1
FPGA
x86
ARM
Nvidia
Xavier
Jetson NX
M.2 M-Key
Accelerator / Storage
mPCIe
Accelerator /
Communication
GigE
Switch
BMC
ESP32
With WiFi
and BLE
SpE
Phy
Single Pair
Ethernet
2x RJ45
with PoE
LoRa
USB-CPower
COM
Brick
HDMI
Barrel Plug
PCIe x4
PCIe x1
PCIe x4
GigE
GigE
PCIe x4
USB 3
GPIO
CSI
USB 2
HDMI
USB 3
u.RECS

11. 11
 Accelerators and microservers
 Evaluated a multitude of accelerators
(CPUs, GPUs, FPGAs, ASICs)
 Same model (YoloV4), different batch sizes
(1, 4, 8)
 Efficiency from 100GOPS/W
to 1250 GOPS/W
Accelerated AIoT Platform
Xavier AGX (B8)
Hailo-8 (B8)
Xavier NX (B1)
Xavier NX (B8) Xavier AGX (B1)
GTX1660 (B1)
GTX1660 (B8)
ZU15 2xB4096
ZU3 B2304
Orin AGX (B1)
Orin AGX (B8)
Hailo-8 (B1)
V100 (B1)
V100 (B8)
A100 (B1)
A100 (B8)
VC1902 C32B6
Xavier AGX (B8)
Xavier NX (B8)
Xavier AGX (B1)
TX2 (B1)
TX2 (B8)
Nano (B8)
GTX1660 (B1)
GTX1660 (B8)
D1577 (B1)
D1577 (B8)
Epyc3451 (B1)
Epyc3451 (B4)
Myriad (B1)
Myriad (B4)
Orin AGX (B1)
Orin AGX (B8)
V100 (B1)
V100 (B8)
A100 (B1)
A100 (B8)
Xavier AGX (B8)
Xavier NX (B8)
Xavier AGX (B1)
TX2 (B1)
TX2 (B8)
GTX1660 (B1)
GTX1660 (B8)
D1577 (B1)
D1577 (B8)
Epyc3451 (B1)
Epyc3451 (B4)
Myriad (B1)
Myriad (B4)
Orin AGX (B1)
Orin AGX (B8)
V100 (B1)
V100 (B8)
A100 (B1)
A100 (B8)
100
1000
10000
6 12 24 48 96 192 384
Performance
[GOPS]
Power [Watt]
INT8
FP16
FP32

12. 12
Xavier AGX (B8)
Hailo-8 (B8)
Xavier NX (B1)
Xavier NX (B8) Xavier AGX (B1)
GTX1660 (B1)
GTX1660 (B8)
ZU15 2xB4096
ZU3 B2304
Orin AGX (B1)
Orin AGX (B8)
Hailo-8 (B1)
V100 (B1)
V100 (B8)
A100 (B1)
A100 (B8)
VC1902 C32B6
Xavier AGX (B8)
Xavier NX (B8)
Xavier AGX (B1)
TX2 (B1)
TX2 (B8)
Nano (B8)
GTX1660 (B1)
GTX1660 (B8)
D1577 (B1)
D1577 (B8)
Epyc3451 (B1)
Epyc3451 (B4)
Myriad (B1)
Myriad (B4)
Orin AGX (B1)
Orin AGX (B8)
V100 (B1)
V100 (B8)
A100 (B1)
A100 (B8)
Xavier AGX (B8)
Xavier NX (B8)
Xavier AGX (B1)
TX2 (B1)
TX2 (B8)
GTX1660 (B1)
GTX1660 (B8)
D1577 (B1)
D1577 (B8)
Epyc3451 (B1)
Epyc3451 (B4)
Myriad (B1)
Myriad (B4)
Orin AGX (B1)
Orin AGX (B8)
V100 (B1)
V100 (B8)
A100 (B1)
A100 (B8)
100
1000
10000
6 12 24 48 96 192 384
Performance
[GOPS]
Power [Watt]
INT8
FP16
FP32
 Accelerators and microservers
 Evaluated a multitude of accelerators
(CPUs, GPUs, FPGAs, ASICs)
 Same model (YoloV4), different batch sizes
(1, 4, 8)
 Efficiency from 100GOPS/W
to 1250 GOPS/W
Accelerated AIoT Platform
High-performance
Low power
Efficiency

13. 13
 Optimizing DL models
 Harware-aware optimizations
 Model compression without loss of accuracy
Optimizing Toolchain for Heterogenenous
Hardware

14. 14
 Optimizing DL models
 Harware-aware optimizations
 Model compression without loss of accuracy
 Hardware software co-design
 Reconfigurable (FPGA) accelerators
 Template-based description
 Heterogenenous engines
Optimizing Toolchain for Heterogenenous
Hardware
Co-design

15. 15
 Optimizing DL models
 Harware-aware optimizations
 Model compression without loss of accuracy
 Hardware software co-design
 Reconfigurable (FPGA) accelerators
 Template-based description
 Heterogenenous engines
 Model verification
 Load, optimize, deploy, and evaluate deep
neural networks on target hardware in a
traceable and reproducible manner.
Optimizing Toolchain for Heterogenenous
Hardware
Co-design

16. 16
 Optimizing DL models
 Harware-aware optimizations
 Model compression without loss of accuracy
 Hardware software co-design
 Reconfigurable (FPGA) accelerators
 Template-based description
 Heterogenenous engines
 Model verification
 Load, optimize, deploy, and evaluate deep
neural networks on target hardware in a
traceable and reproducible manner
 Simulation
 Simulation framework for implementation,
testing, and debugging
Optimizing Toolchain for Heterogenenous
Hardware
Co-design

17. 17
 Requirements concepts for AIoT
 Conceptual model of the compositional architecture
Framework
 Abstraction levels: Knowledge and Analytical, Conceptual,
Design, and Runtime level
 Clusters of concerns for quality aspects of an AI system in the IoT
are Safety, Security, Privacy and Ethical Aspects
 Safety aspects
 Requirements Engineering facilitates constructive design of
safety critical systems
 Trusted execution for AIoT
 Trusted Execution Environments (TEEs): Intel SGX & ARM
TrustZone
 Trusted runtimes: TWINE for SGX and WATZ for TrustZone
 Inferring trust in AIoT
 Remote Attestation (RA) integrated into trusted
WebAssembly runtimes
 SIRE: a Byzantine fault-tolerant infrastructure supporting
remote attestation
Safety, Security and Requirements for Distributed AIoT
Systems
Group of
concerns
Cluster of
concern
Architecture
view
Architecture
viewpoint
Business Goal
or Use Case
Stakeholder
Relation /
Correspon-
dence
Level of
Abstraction
System-
of-interest
1..*
1..*
1..*
1..*
has
1..*
exists on
describes /
specifies
determines
addresses
governs
1..*

18. 18
 Automotive
 Pedestrian Automatic Emergency Breaking (P-AEB)
 The main challenge is the limited processing power available
in the vehicle, the communication resource limits and the
possible processing power available in the edge (base
station)
 Distributing the DL model over multiple processing nodes
VEDLIoT Applications

19. 19
 Industrial IoT
 Predictive maintenance: Motor Condition Classification
 Anomaly detection: Series Arc Fault Detection (AFD) in low-voltage direct current (LVDC) systems
 Challenges: limited data of fault condition under real scenarios due to finance and safety considerations,
and the lack of framework for software development with DL.
VEDLIoT Applications

20. 20
 Smart home
 Smart mirror - intuitive user interface, display of
personalized information and the status of the
smart home
 Focus on data protection through local
processing
 Tasks:
 user identification by facial recognition and
tracking using depth imaging cameras
 hand gestures recognized to control the mirror
 voice assistant - natural language processing
(NLP) - to control key features through voice
commands
 Currently deployed on one NVIDIA Orin AGX,
combined with an M.2-based Hailo-8 DL
accelerator for gesture detection
VEDLIoT Applications
RGB-D Camara
Background Removal
RetinaFace
for Face Detection
Gesture Detection
with YoloV7-tiny
Object Detection
with YoloV7
Siamese Network for
Identification
Speech to Text
with coqui
Intent Extraction
for Skill System
ROS2 Bridge
Webstream
Person Recognition Decision Maker
Hailo8
ROS2 Nodes
Text to Speech
with coqui
Micophone Input
Speaker
Trusted
Execution
Environment
MagicMirror²
Nvidia AGX Orin

21. 21
 Integration of Deep Learning into IoT devices with restricted computing capabilities and
minimal power consumption requirements - energy-efficient computing
 AIoT hardware platform with tailored hardware components and accelerators: from
embedded systems to edge computing and cloud platforms
 Efficient middleware simplifies neural network programming, testing, and deployment to
this diverse hardware ecosystem
 Innovative approaches for requirements engineering, combined with safety and security
principles
 Concepts validated through use cases in vital industry sectors: automotive, automation,
and smart home
Summary

22. 22
Thank you for your
attention.