For the full video of this presentation, please visit: https://www.embedded-vision.com/platinum-members/mediatek/embedded-vision-training/videos/pages/may-2019-embedded-vision-summit For more information about embedded vision, please visit: http://www.embedded-vision.com Bing Yu, Senior Technical Manager and Architect at MediaTek, presents the "MediaTek’s Approach for Edge Intelligence" tutorial at the May 2019 Embedded Vision Summit. MediaTek has incorporated an AI processing unit (APU) alongside the traditional CPU and GPU in its SoC designs for the next wave of smart client devices (smartphones, cameras, appliances, cars, etc.). Edge applications can harness the CPU, GPU and APU together to achieve significantly higher performance with excellent efficiency. In this talk, Yu presents MediaTek’s AI-enabled SoCs for smart client devices. He examines the features of the AI accelerator, which is the core building block of the APU. He also describes the accompanying toolkit, called NeuroPilot, which enables app developers to conveniently implement inference models using industry-standard frameworks.

1. © 2019 MediaTek
MediaTek’s Approach
for Edge Intelligence
Bing Yu
MediaTek
May 22, 2019

2. © 2019 MediaTek
Summary
• MediaTek’s Product Position
• MediaTek’s Approach (P90)
• MediaTek AI Accelerator’s Features and
Architecture
• NeuroPilot & Platform-aware MLKits

3. © 2019 MediaTek
MediaTek’s Product Position

4. © 2019 MediaTek
Leading Market Positioning
with cross-platform synergies
Source (ranking by 2018 market share): Strategy Analytics, Gartner, IDC, IC Insight, iSuppli and MediaTek company data.
*Combined with MStar.
Digital TV*
Feature
Phone
Android
Tablet
Optical Drive & BD
Player
Voice Assistant
DeviceSmartphone
Connectivity
Network
#1 #1#1 #1#1 #2 #1
Growth HarvestMobile Computing

5. © 2019 MediaTek
Devices around us are getting smarter everyday.
Smartphone
Digital TV
Automotive
AP / Router VAD
IoT
Performance /
Power Balance
Heterogeneous
Computing
Various Processing
Units in SoC
Booming Intelligent Devices

6. © 2019 MediaTek
Customer
Product
Software &
System
Connectivity
Modem
SoC Design
Compute
RF &
Analog
Multimedia
Technology / IP
Core Technologies Enable Intelligent
Devices with Leading Customers

7. © 2019 MediaTek
DNN Inference is Moving to the Edge
• The inherent limitations of the cloud are difficult to improve
• Rapid improvement in DNN efficiency is enabling more edge AI applications
• Cloud-Edge collaborative model will provide the best user experience
⎼Most training will remain in the cloud
Thousands
Billions
CLOUD
Data Centers
EDGE
Devices
Computing
Resource
Memory
Capacity
Thermal
Budget
Network
Latency
Availability Privacy
Energy
Efficiency
Cloud
Edge

8. © 2019 MediaTek
MediaTek’s Approach

9. © 2019 MediaTek
MediaTek Helio P90 SoC and System
12 GB/s1.1 TMACs 5 W
Modem
ISPAPU
Wi-Fi
Bluetooth
GPUCPU
Sensors
Codec
GPS
45 °C
No fan
12 nm
APU
(VPU + AIA)

10. © 2019 MediaTek
http://ai-benchmark.com/ranking_processors.html
ETH Zürich AI Benchmark 3.0 (latest release@Mar 27)
AI Score
MediaTek Helio P90 19496
2nd Place 18924
QUANT
Score
QUANT
Accuracy
FP16
Score
FP16
Accuracy
MediaTek
Helio P90
6212 98 9910 95
2nd Place 3695 55 7361 37

11. © 2019 MediaTek
Network Quantization and Pruning
• Network Quantization
• DNNs are generally represented in floating point 32-bit (FP32) format
• Human can do image classification without FP32 precision
• Quantizing FP32 to INT8 can reduce the complexity with negligible accuracy loss
• Recent research shows promising results with lower bit precision, such as 4/2/1 bit
• Network Pruning
• DNN architectures have many redundant weights to help the model to
converge faster during the training process.
• Unimportant weights can be removed to increase weight matrix sparsity
• Importance assessment of weights is the key
Accuracy Energy DRAM
Pruning +
Qantization
Int8
Within 1% ~93% reduction 80~90% reduction

12. © 2019 MediaTek
AIA Features and Architecture

13. © 2019 MediaTek
AIA Key Features
▪ Bandwidth reduction techniques
- TCM for data-exchange
- Sparsity compression
▪ High Performance Engine
AIA: 806 GMAC/s
AIAx2: 1.6 TMAC/s
@788MHz
▪ Flexible quantization scheme
- Asymmetric or symmetric quant.
- No extra performance overhead
▪ Power Efficient
>1 TMACs/W (2x better than VPU)
@12FFC
▪ Bandwidth-Aware Design
▪ Dual AXI Port for high BW
▪ High Throughput Load/Store
▪ Simultaneous execution
of OPs (CONV/ACT/POOL)
▪ Support INT8/INT16/FP16

14. © 2019 MediaTek
AIA Architecture
• DNN performance and efficiency driven architecture design
⎼Acceleration HW for operations used intensively (CONV, Pooling, ReLu, etc.)
⎼Specialized scheduler to maximize PE utilization rate
Pooling
…
…
ElementwiseConvolution
Activation
Convolution
Buffer
PE PE
PE PE
APU Data Flow Controller and Scheduler

15. © 2019 MediaTek
PE PE
PE PE
PE PE
PE PE
Convolution Engine
• 16 GCUs, 32 CUs per GCU, each CU has dual-MAC units
• INT8: 16x32x2 = 1024 MAC/cycle
• INT16:16x32= 512 MAC/cycle
• FP16:16x16 = 256 MAC/cycle
• Dual-MAC unit provides: two 8-bit MAC or one 16-bit MAC
• FP16 has a separate design not shown in the diagram.
16 GCUs
PE PE
PE PE
PE PE
PE PE

16. © 2019 MediaTek
(Sequential)
(Simultaneous execution of
conv., Relu and pooling)
Convolution ReLu Pooling
ReLu
Pooling
Convolution
Simultaneous Execution
• Engines working in a pipelined fashion
Different engines work in parallel to deliver high throughput.

17. © 2019 MediaTek
NeuroPilot & Platform-aware MLKits

18. © 2019 MediaTek
NeuroPilot Platform-aware MLKits
Super-Resolution Depth Estimation Segmentation
MediaTek Platform
Network
Reduction
Network Architecture
Search
Network Deep Fusion (Tiling + Fusion)
BW Req.: 2.0GB/s
HW Util.: 80%
FPS: 100 FPS
Power: < 40mW
MediaTek Platform-aware MLKits
Platform-friendly NN StructureUser-defined NN Structure
Conv0
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
Conv…
FC
Application
Developers
Network
Quantization

19. © 2019 MediaTek
NeuroPilot for Developer
ANN Runtime
ANN API
ANN HAL
Interpreter
.tflite format
Tensowflow
Model
CPU NN HAL impl. GPU NN HAL impl. VPU NN HAL impl.
Caffe / ONNX
Model
MTK Ext. API
1. Bind Op with HW 2.Profiler 3.Debugger (Log)
TOCO
Offline Tool
Quantization
NeuroPilot specified
On Device
CPU GPU VPU
Developers
AIA NN HAL impl.
AIA
Supports Tensorflow as well as Caffe and ONNX
Highly integrated with Android Neural Network
MediaTek additions
1. Binding Op with HW
2. Profiler
3. Debugger

20. © 2019 MediaTek
20
MediaTek NeuroPilot Toolkit- utility and
debug tool
NN Utility
Debugger
Profiling
NeuroPilot
Toolkit
• Model Convertor
(TensorFlow/Caffe/ONNX)
• Quantization
• Power API
• Performance
• Memory
• System Crash
• Mobilelog

21. © 2019 MediaTek
Conclusion
1. Provides flexible HW (CPU, GPC, VPU and AIA) for the ever changing AI
algorithm.
2. Improves the performance of AI applications by reducing the memory
bandwidth and increasing the compute efficiency with specialized AI
accelerator.
3. Toolchain is highly integrated with Android NN, and adding performance
monitor to give feedback to the algorithm developer for the optimization of
AI algorithm runs on MediaTek’s platform.
4. Adding AI capability to enhance the user experience and create new use
cases which mean more value for our customers.
5. Rapid advancement in edge AI technology will drive more AI applications
to the edge

22. © 2019 MediaTek
Our mission is to be a change catalyst, empowering our
partners with smart technology solutions that will inspire
them to connect with “next billion” people.
By building technologies that help connect individuals to
the world around them, we are enabling people to expand
their horizons and more easily achieve their goals.
We believe anyone can achieve something amazing. And
we believe they can do it every single day. We call this idea
Everyday Genius and everything we do is dedicated to
making it possible.
Enhance & Enrich Everyone’s Life
Everyday Genius
22

23. Copyright © MediaTek Inc. All rights reserved.
23

24. © 2019 MediaTek
Resource and Info
24
Product Description
MediaTek helio P90
https://www.mediatek.com/products/smart
phones/mediatek-helio-p90
Benchmark
ETH Zurich Benchmark 3.0
http://ai-
benchmark.com/ranking_processors.html
Embedded Vision Summit
MediaTek’s Approach for Edge Intelligent
10:45 AM – 11:15 AM on May 22, 2019

25. © 2019 MediaTek
Thank you