Presentation from the 2017 LinuxCon China

1. www.huawei.com
HUAWEI TECHNOLOGIES CO., LTD.
Make Accelerator Pluggable for
Container Engine
Author/ Email: Jiuyue Ma / majiuyue@huawei.com
Version: V1.0 (20150504)

2. HUAWEI TECHNOLOGIES CO., LTD. 2
Today’s Topic
Container Engine
Hardware Accelerators
e.g. GPU、QAT、FPGA
OLTP/OLAP BigData Processing
Neural Network
CloudRAN
Pluggable
Accelerator Support
for Container Engine

3. HUAWEI TECHNOLOGIES CO., LTD. 3
Agenda
 Use accelerator in containers and How?
 Key problems to deal with
 Identify accelerator requirements
 Prepare accelerator runtimes
 Manage accelerator resources
 Further development

4. HUAWEI TECHNOLOGIES CO., LTD. 4
Why Container?

5. HUAWEI TECHNOLOGIES CO., LTD. 5
Why Container? Why NOT Container?
* Raphael Da Silva. OPEN SOURCE TECHNOLOGY: Docker Containers on IBM Bluemix. 2015.

6. HUAWEI TECHNOLOGIES CO., LTD. 6
 CPU is not fast/effective/… enough
 Heterogeneous is the future
 Nvidia GPU-compute, Intel Skylake Xeon+FPGA, AWS P2/F1 Instance
 use the right processor, in the right place, at the right time
Why Accelerator?
Container
a.k.a. Docker
[baidu, HotChips 2016]

7. HUAWEI TECHNOLOGIES CO., LTD. 7
Google it ! “Use XXX in docker”
NVIDIA’s GPU-Docker Solution
DPDK
FPGA
GPU

8. HUAWEI TECHNOLOGIES CO., LTD. 8
Use GPU in Docker (from nvidia-docker)
Wrapped Command
docker run
--volume-driver=nvidia-docker
--volume=nvidia_driver_361.48:/usr/local/nvidia:ro
③ nvidia-docker cli-wrapper
$ ldconfig -p | grep -E 'nvidia|cuda'
libnvidia-ml.so (libc6,x86-64) => /usr/lib/nvidia-361/libnvidia-ml.so
libnvidia-glcore.so.361.48 (libc6,x86-64) => /usr/lib/nvidia-361/lib…
libnvidia-compiler.so.361.48 (libc6,x86-64) => /usr/lib/nvidia-361/lib…
libcuda.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libcuda.so
...
$ lsmod | grep nvidia
nvidia_uvm 711531 2
nvidia_modeset 742329 0
nvidia 10058469 80 nvidia_modeset,nvidia_uvm
Key Problem：VERSION MISMATCH
$ nvidia-smi
Failed to initialize NVML: Driver/library version mismatch
② VOLUME: nvidia-docker/nvidia_driver_361.48
① Labeled Image
LABEL com.nvidia.volumes.needed="nvidia_driver"
LABEL com.nvidia.cuda.version="7.5"
• detect/check cuda version
• hook docker cli args
--volume-driver / --volume nvidia

9. HUAWEI TECHNOLOGIES CO., LTD. 9
Do we really need “nvidia-docker”?
If so, we also need:
 dpdk-docker, qat-docker, fpga-docker, …
But,
if we need both
GPU and DPDK?
nvidia DPDK QAT FPGA
DPDK
nvidia
Pluggable
Accelerator

10. HUAWEI TECHNOLOGIES CO., LTD. 10
Device Files
(-d device)
/dev/nvidia0
/dev/nvidia-smi
/dev/nvidia-uvm
/dev/uio0 /dev/fpga0
File Bindings
(-v src:dest)
/usr/lib/nvidia-361/libnvidia-ml.so
/usr/lib/nvidia-361/libnvidia-glcore.so.361.48
/usr/lib/nvidia-361/libnvidia-
compiler.so.361.48
/usr/lib/x86_64-linux-gnu/libcuda.so
...
/sys/bus/pci/devices/xxxx:xx:xx.x
/dev/hugepages
/usr/lib/libfpga.so
/sys/bus/pci/devices/xxxx:xx:xx.x
/dev/hugepages
Environments
(-e env)
PATH=$PATH:/usr/lib/nvidia-361/bin
LD_LIBRARY_PATH=…
Pluggable Accelerator, How?
 What’s the difference?
nvidia DPDK FPGA
Accelerator
Plugin Interface
in-house
FPGA Accel

11. HUAWEI TECHNOLOGIES CO., LTD. 11
Implementation (1/3): IDENTIFY
 Accelerator Requirements
 We are using the function provided by device, not the device itself
 Identified by runtime instead of device
 e.g. “A SHA256 accelerator” instead of “A FPGA with XXXX-bitstream”
 Identify user requirements
 Integrate into image
 Specify when start container
 Identify device capabilities
 Identified by “accelerator plugin”
 Report to engine through plugin api
LABEL runtime
gpu0=cuda:7.5
$ docker run --accel fpga:crypto
huawei/crypto-image
image cli / rest-api
 dockerd
Device Accel Runtime
/dev/nvidia0 cuda:7.5, cuda:8.0, OpenCL 2.0, …
/dev/qat0 md5, sha256, aes, …
/dev/fpga0 crypto, compress, decode, …
Accelerator
Plugin
Plugin API

12. HUAWEI TECHNOLOGIES CO., LTD. 12
Implementation (2/3): PREPARE
 Select plugin from accelerator requirements
 Device initialization by plugin
 Dynamic inject required libraries & devices
DPDK-plugin
- eal.so/…
- igb_uio.ko
- /dev/uioX
FPGA-plugin
- fpga.so
- fpga.ko
- /dev/fpgaX
 container
 Host
Application
fpga.ko
v1
 dockerd
/dev/fpga0
generate
bind
libacc
accel requirements
PaaS
fpga.so/dev/fpga0
fpga.so
v1
(2) mount userspace lib
(3) mount device
GPU-plugin
- gpu.so / cuda.so
- gpu.ko
- /dev/gpuX
QAT-plugin
- qat.so
- qat.ko
- /dev/qatX

13. HUAWEI TECHNOLOGIES CO., LTD. 13
Implementation (3/3): MANAGEMENT
Running
CreatedStopped
CreatingDeleting
INIT
Container
Life-Cycles
allocate
persistent
allocate
non-persistent
release
non-persistent
release
persistent
prepare
parse accel
requirements
Accelerator
Plugin
 Accelerator Life-Cycle
 Allocate  Prepare  Release
 Plugin hooks for each stage
 Integrate with Container Life-
Cycle
 Persistent：CreateDelete
 Non-persistent： StartStop

14. HUAWEI TECHNOLOGIES CO., LTD. 14
Example Flow

15. HUAWEI TECHNOLOGIES CO., LTD. 15
Further Developments
 Standardize “Accelerator Runtime”
 OCI Runtime/Image Spec should be good place
 Integrate with Docker Swarm / K8S / etc
 Cluster level accelerator schedule&management
 More Features
 Accelerator Sharing
 Accelerator Exception Interface
 Accelerator Hot-Plug
 …

16. Copyright© 2011 Huawei Technologies Co., Ltd. All Rights Reserved.
The information in this document may contain predictive statements including, without limitation, statements regarding the future
financial and operating results, future product portfolio, new technology, etc. There are a number of factors that could cause actual
results and developments to differ materially from those expressed or implied in the predictive statements. Therefore, such information
is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any
time without notice.
Thank you
www.huawei.com