Производительность инференса - одна из самых серьезных проблем при внедрении DL приложений, так как она определяет, какое впечатление от сервиса останется у конечного пользователя, а также какова будет цена внедрения этого продукта. Таким образом, для инференса важно быть высокопроизводительным и энергоэффективным. TensorRT автоматически оптимизирует обученную нейронную сеть для максимальной производительности, обеспечивая существенное ускорение по сравнению с обычными часто используемыми фреймворками. Из презентации вы узнаете, какие оптимизации применяются в TensorRT, как его использовать и увидите, насколько он быстр в избранных задачах.

1. Высокопроизводительный
инференс глубоких сетей на
GPU с помощью TensorRT
Максим Милаков, NVIDIA
v2

2. • You will learn:
• How GPUs are used for DL now?
• Why do you want to use GPUs for inference?
• Why do you want to use TensorRT for inference on GPUs?
• This talk is NOT:
• An intro to DL
• A set of code samples

3. NVIDIA
The AI Computing Company

4. NVIDIA Powering the Deep Learning Ecosystem
DEEP LEARNING FRAMEWORKS
COMPUTER VISION SPEECH AND AUDIO NATURAL LANGUAGE PROCESSING
Object Detection Voice Recognition Language Translation
Recommendation
Engines
Sentiment Analysis
Mocha.jl
Image Classification
NVIDIA DEEP LEARNING SDK
NCCLcuDNN cuBLAS cuSPARSE TensorRT

5. ML development and deployment cycle

6. Training with SGD backpropagation

7. ImageNet: results for 2010-2014
15%
83%
95%28%
26%
15%
11%
7%
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0%
5%
10%
15%
20%
25%
30%
2010 2011 2012 2013 2014
% Teams using GPUs
Top-5 error

8. Deployment scenarios - Hyperscale
• Input generated and output
used at the client device
• Inference is running at data
center
• High throughput
• On-the-fly batching
TensorRT for Hyperscale
Image
Classification Object Detection
Image
Segmentation
---

9. Deployment scenarios - Embedded
• On-device inference
• Small-batch inference
• Low latency
GPU Inference Engine for Automotive
Pedestrian
Detection
Lane
Tracking
Traffic Sign
Recognition
---
NVIDIA DRIVE PX 2

10. TensorRT
High-performance deep learning
inference for production deployment
0
1
2
3
4
5
6
7
8
1 8 128
CPU-Only Tesla M4 + TensorRT
Up to 16x More Inference Efficiency
Img/sec/watt
Batch Sizes
GoogLenet, CPU-only vs Tesla M4 + TensorRT on
Single-socket Haswell E5-2698 v3@2.3GHz with HT
EMBEDDED
Jetson TX1
AUTOMOTIVE
Drive PX
DATA CENTER
Tesla M4

11. Comparing to DL frameworks
• Particularly effective at
small batch-sizes
• Improves perf for complex
networks the most
Jetson TX1 HALF2 column uses fp16
GoogleNet Performance

12. TensorRT
• Fuse network layers
• Eliminate concatenation layers
• Kernel specialization
• Auto-tuning for target platform
• Select optimal tensor layout
• Batch size tuningTRAINED
NEURAL NETWORK
OPTIMIZED
INFERENCE
RUNTIME

13. Layers supported
• v1 designed for 2D images
• Layers supported
• Convolution: 2D
• Activation: ReLU, tanh and sigmoid
• Pooling: max and average
• ElementWise: sum, product or max of two tensors
• LRN: cross-channel only
• Fully-connected: with or without bias
• SoftMax: cross-channel only
• Deconvolution
• Custom layers possible with sandwich approach now

14. Optimizations
• Eliminate unused layers
• Vertical layer fusion: Fuse convolution, bias, and ReLU layers to form
a single layer
• Horizontal layer fusion: Combine layers with the same source tensor
and the same parameters

15. Optimizations: Original network
concat
max pool
input
next input
3x3 conv.
relu
bias
1x1 conv.
relu
bias
1x1 conv.
relu
bias
1x1 conv.
relu
bias
concat
1x1 conv.
relu
bias
5x5 conv.
relu
bias

16. Optimizations: Vertical layer fusion
concat
max pool
input
next input
concat
1x1 CBR 3x3 CBR 5x5 CBR 1x1 CBR
1x1 CBR 1x1 CBR

17. Optimizations: Horizontal layer fusion
concat
max pool
input
next input
concat
3x3 CBR 5x5 CBR 1x1 CBR
1x1 CBR

18. Optimizations: Concat elision
max pool
input
next input
3x3 CBR 5x5 CBR 1x1 CBR
1x1 CBR

19. TensorRT – 2 phases deployment
• Build
• Apply optimizations on the network configuration
• Generate an optimized plan for computing the forward pass
• Deploy
• Forward and output the inference result
Build
Deploy
File
Model
File
Deploy
Plan
Output
I/O
Layers
Max
Batchsize
Inputs
Batch
size

20. Pascal GPUs for inference
Maximum Efficiency for Scale-out Servers
TESLA P4 TESLA P40
Highest Throughput for Scale-up Servers

21. P40/P4 – New “Int8” for Inference

22. TensorRT v2: int8 accuracy
• Almost the same accuracy for
major models
• Still working on the procedure to
make quantization optimal

23. TensorRT v2: int8 performance
• Up to 3x performance
• No FFT and Winograd yet
• Constant factors
• Beneficial on small batches even
on bigger P40

24. TensorRT v2: more performance
178
480
1,514
4,121
3,200
6,514
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
E5-2690v4
14 Core
M4
(FP32)
M40
(FP32)
P100
(FP16)
P4
(INT8)
P40
(INT8)
All results are measured, based on GoogLenet with batch size 128
Xeon uses MKL 2017 GOLD with FP32, GPU uses TensorRT internal development ver.
>35x
1.4
12.3 10.6
27.9
91.1
56.3
0
20
40
60
80
100
E5-2690v4
14 Core
M4
(FP32)
M40
(FP32)
P100
(FP16)
P4
(INT8)
P40
(INT8)
>60x
P40 For Max Inference Throughput,
img/sec
P4 For Max Inference Efficiency,
img/sec/watt

25. Deep Learning Everywhere
• developer.nvidia.com/tensorrt
• developer.nvidia.com/deep-learning
• developer.nvidia.com/cuda-zone
• mmilakov@nvidia.com

26. Backup slides

27. Tesla Products Decoder