39. Want to try Chainer + ChainerMN?
39
Cloud formation support is coming soon!
40. Optimization technique for non-IB environment:
Double buffering
• Each update uses the gradients from previous iteration (1-step stale grad.)
40
41. Computing time of ImageNet training with Double Buffering + FP16 communication
2.1 times faster !
41
• Local batchsize: 64
• 32 processes
• NCCL for Allreduce
43. 43
95% scalability up to 32 GPUs !!
model acc. 75%
model acc. 76%
ResNet-50 on ImageNet training
• 25Gbps Ethernet
• Double buffering
• FP16 communication (NCCL)
• V100 GPUs
• Batchsize: 64/GPU