Deep Dive on Deep Learning (June 2018)
•
3 gostaram•56,428 visualizações
2-hour session at AWS Summit Berlin, 6/6/18.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Deep Dive on Deep Learning (June 2018)
Semelhante a Deep Dive on Deep Learning (June 2018) (20)
Mais de Julien SIMON
Mais de Julien SIMON (20)
Último
Último (20)
Deep Dive on Deep Learning (June 2018)
- 1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Julien Simon Principal Technical Evangelist, AI and Machine Learning @julsimon Deep Dive on Deep Learning
- 2. • Deep Learning concepts • Common architectures and use cases • Apache MXNet • Infrastructure for Deep Learning • Demos along the way: MXNet, Gluon, Keras, TensorFlow, PyTorch ☺ Agenda
- 4. Activation functions i=1 l xi ∗ wi = u ”Multiply and Accumulate” Source: Wikipedia The neuron
- 5. x = x11, x12, …. x1I x21, x22, …. x2I … … … xm1, xm2, …. xmI I features m samples y = 2 0 … 4 m labels, N2 categories 0,0,1,0,0,…,0 1,0,0,0,0,…,0 … 0,0,0,0,1,…,0 One-hot encoding Neural networks
- 6. x = x11, x12, …. x1I x21, x22, …. x2I … … … xm1, xm2, …. xmI I features m samples y = 2 0 … 4 m labels, N2 categories Total number of predictions Accuracy = Number of correct predictions 0,0,1,0,0,…,0 1,0,0,0,0,…,0 … 0,0,0,0,1,…,0 One-hot encoding Neural networks
- 7. Initially, the network will not predict correctly f(X1) = Y’1 A loss function measures the difference between the real label Y1 and the predicted label Y’1 error = loss(Y1, Y’1) For a batch of samples: 𝑖=1 𝑏𝑎𝑡𝑐ℎ 𝑠𝑖𝑧𝑒 loss(Yi, Y’i) = batch error The purpose of the training process is to minimize error by gradually adjusting weights. Neural networks
- 8. Training data set Training Trained neural network Batch size Learning rate Number of epochs Hyper parameters Backpropagation Training
- 9. Stochastic Gradient Descent Imagine you stand on top of a mountain with skis strapped to your feet. You want to get down to the valley as quickly as possible, but there is fog and you can only see your immediate surroundings. How can you get down the mountain as quickly as possible? You look around and identify the steepest path down, go down that path for a bit, again look around and find the new steepest path, go down that path, and repeat—this is exactly what gradient descent does. Tim Dettmers University of Lugano 2015 https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-history-training/ The « step size » depends on the learning rate z=f(x,y)
- 10. Local minima and saddle points « Do neural networks enter and escape a series of local minima? Do they move at varying speed as they approach and then pass a variety of saddle points? Answering these questions definitively is difficult, but we present evidence strongly suggesting that the answer to all of these questions is no. » « Qualitatively characterizing neural network optimization problems », Goodfellow et al, 2015 https://arxiv.org/abs/1412.6544
- 11. https://medium.com/@julsimon/tumbling-down-the-sgd-rabbit-hole-part-1-740fa402f0d7 SGD works remarkably well and is still widely used. Adaptative optimizers use a variable learning rate. Some even use a learning rate per dimension (Adam). Optimizers
- 12. Validation data set (also called dev set) Neural network in training Validation accuracy Prediction at the end of each epoch This data set must have the same distribution as real-life samples, or else validation accuracy won’t reflect real-life accuracy. Validation
- 13. Test data set Fully trained neural network Test accuracy Prediction at the end of experimentation This data set must have the same distribution as real-life samples, or else test accuracy won’t reflect real-life accuracy. Test
- 14. Training accuracy Loss function Accuracy 100% Epochs Validation accuracy Loss Best epoch OVERFITTING « Deep Learning ultimately is about finding a minimum that generalizes well, with bonus points for finding one fast and reliably », Sebastian Ruder Early stopping
- 15. Common architectures and use cases
- 16. Convolutional Neural Networks (CNN) Le Cun, 1998: handwritten digit recognition, 32x32 pixels https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-core-concepts/
- 17. Source: http://timdettmers.com Extracting features with convolution Convolution extracts features automatically. Kernel parameters are learned during the training process.
- 18. Downsampling images with pooling Source: Stanford University Pooling shrinks images while preserving significant information.
- 21. Text Detection and Recognition https://github.com/Bartzi/stn-ocr MXNet
- 22. Face Detection https://github.com/tornadomeet/mxnet-face https://github.com/deepinsight/insightface https://arxiv.org/abs/1801.07698 Face Recognition LFW 99.80%+ Megaface 98%+ with a single model MXNet MXNet
- 24. Long Short Term Memory Networks (LSTM) • A LSTM neuron computes the output based on the input and a previous state • LSTM networks have memory • They’re great at predicting sequences, e.g. machine translation
- 26. GAN: Welcome to the (un)real world, Neo Generating new ”celebrity” faces https://github.com/tkarras/progressive_growing_of_gans From semantic map to 2048x1024 picture https://tcwang0509.github.io/pix2pixHD/ TF PyTorch
- 27. Wait! There’s more! Models can also generate text from text, text from images, text from video, images from text, sound from video, 3D models from 2D images, etc.
- 28. Apache MXNet
- 29. Apache MXNet: Open Source library for Deep Learning Programmable Portable High Performance Near linear scaling across hundreds of GPUs Highly efficient models for mobile and IoT Simple syntax, multiple languages Most Open Best On AWS Optimized for Deep Learning on AWS Accepted into the Apache Incubator
- 30. Input Output 1 1 1 1 0 1 0 0 0 3 mx. sym. Convol ut i on( dat a, ker nel =( 5, 5) , num_f i l t er =20) mx. sym. Pool i ng( dat a, pool _t ype=" max" , ker nel =( 2, 2) , st r i de=( 2, 2) l st m. l st m_unr ol l ( num_l st m_l ayer , seq_l en, l en, num_hi dden, num_embed) 4 2 2 0 4=Max 1 3 ... 4 0.2 -0.1 ... 0.7 mx. sym. Ful l yConnect ed( dat a, num_hi dden=128) 2 mx. symbol . Embeddi ng( dat a, i nput _di m, out put _di m = k) 0.2 -0.1 ... 0.7 Queen 4 2 2 0 2=Avg Input Weights cos(w, queen ) = cos(w, k i n g) - cos(w, m an ) + cos(w, w om an ) mx. sym. Act i vat i on( dat a, act _t ype=" xxxx" ) " r el u" " t anh" " si gmoi d" " sof t r el u" Neural Art Face Search Image Segmentation Image Caption “ People Riding Bikes” Bicycle, People, Road, Sport Image Labels Image Video Speech Text “ People Riding Bikes” Machine Translation “ Οι άνθρωποι ιππασίας ποδήλατα” Events mx. model . FeedFor war d model . f i t mx. sym. Sof t maxOut put
- 31. • More chances for optimization • Language independent • E.g. TensorFlow, Theano, Caffe, MXNet Symbol API • Less flexible • ’Black box’ training PROS CONS C can share memory with D because C is deleted later A = Variable('A') B = Variable('B') C = B * A D = C + 1 f = compile(D) d = f(A=np.ones(10), B=np.ones(10)*2) A B 1 + x Declarative Programming ‘define then run’
- 32. DEMO: Symbol API 1 – Fully Connected Neural Network (MNIST) 2 – Convolution Neural Network (MNIST)
- 33. import numpy as np a = np.ones(10) b = np.ones(10) * 2 c = b * a d = c + 1 • Straightforward and flexible. • Take advantage of language native features (loop, condition, debugger). • E.g. Numpy, PyTorch, MXNet Gluon API • Harder to optimize PROS CONS Imperative Programming ‘define by run’
- 34. DEMO: Gluon API Fully Connected Network (MNIST)
- 35. Gluon CV: classification, detection, segmentation https://github.com/dmlc/gluon-cv [electric_guitar], with probability 0.671
- 36. DEMO: Gluon CV
- 40. DEMO: Keras-MXNet Convolutional Neural Network (MNIST)
- 41. Infrastructure for Deep Learning
- 42. Amazon EC2 C5 instances AVX 512 72 vCPUs “Skylake” 144 GiB memory C5 12 Gbps to EBS 2X vCPUs 2X performance 3X throughput 2.4X memory C4 36 vCPUs “Haswell” 4 Gbps to EBS 60 GiB memory C5: Nex t Ge ne rat ion Compute - Opt imize d Insta nc e s wit h Inte l® Xe on® Sca la ble Proc e ssor AWS Compute opt imize d insta nc e s support t he new Inte l® AV X - 512 a dva nc e d inst ruc t ion set , e na bling you to more effic ie ntly run ve c tor proc e ssing work loa ds wit h single a nd double f loat ing point pre c ision, suc h a s AI/ma c hine le a rning or v ide o proc e ssing. 25% improvement in price/performance over C4
- 43. Faster TensorFlow training on C5 https://aws.amazon.com/blogs/machine-learning/faster-training-with-optimized-tensorflow-1-6-on- amazon-ec2-c5-and-p3-instances/
- 44. Amazon EC2 P3 Instances • P3.2xlarge, P3.8xlarge, P3.16xlarge • Up to eight NVIDIA Tesla V100 GPUs in a single instance • 40,960 CUDA cores, 5120 Tensor cores • 128GB of GPU memory • 1 PetaFLOPs of computational performance – 14x better than P2 • 300 GB/s GPU-to-GPU communication (NVLink) – 9x better than P2 T h e f a s t e s t , m o s t p o w e r f u l G P U i n s t a n c e s i n t h e c l o u d
- 45. Preconfigured environments to quickly build Deep Learning applications Conda AMI For developers who want pre- installed pip packages of DL frameworks in separate virtual environments. Base AMI For developers who want a clean slate to set up private DL engine repositories or custom builds of DL engines. AMI with source code For developers who want preinstalled DL frameworks and their source code in a shared Python environment. https://aws.amazon.com/machine-learning/amis/ AWS Deep Learning AMI
- 46. Amazon SageMaker Pre-built notebooks for common problems K-Means Clustering Principal Component Analysis Neural Topic Modelling Factorization Machines Linear Learner XGBoost Latent Dirichlet Allocation Image Classification Seq2Seq, And more! ALGORITHMS Apache MXNet TensorFlow Caffe2, CNTK, PyTorch, Torch FRAMEWORKS Set up and manage environments for training Train and tune model (trial and error) Deploy model in production Scale and manage the production environment Built-in, high- performance algorithms Build
- 47. Amazon SageMaker Pre-built notebooks for common problems Built-in, high- performance algorithms One-click training Hyperparameter optimization Build Train Deploy model in production Scale and manage the production environment
- 48. Amazon SageMaker Fully managed hosting with auto-scaling One-click deployment Pre-built notebooks for common problems Built-in, high- performance algorithms One-click training Hyperparameter optimization Build Train Deploy
- 49. Amazon ECR Model Training (on EC2) Model Hosting (on EC2) Trainingdata Modelartifacts Training code Helper code Helper codeInference code GroundTruth Client application Inference code Training code Inference requestInference response Inference Endpoint Amazon SageMaker
- 50. Open Source Containers for TF and MXNet https://github.com/aws/sagemaker-tensorflow-containers https://github.com/aws/sagemaker-mxnet-containers • Build them and run them on your own machine • Run them directly on a notebook instance (aka local mode) • Customize them and push them to ECR • Run them on SageMaker for training and prediction at scale
- 51. DEMO: SageMaker 1 – Use the built-in algorithm for image classification (CIFAR-10) 2– Bring your own Tensorflow script for image classification (MNIST) 3– Bring your own Gluon script for sentiment analysis (Stanford Sentiment Tree Bank 2) 4 – Build your own Keras-MXNet container (CNN + MNIST) 5 – Build your own PyTorch container (CNN + MNIST)
- 52. Danke schön! Julien Simon Principal Technical Evangelist, AI and Machine Learning @julsimon https://aws.amazon.com/machine-learning https://aws.amazon.com/blogs/ai https://mxnet.incubator.apache.org | https://github.com/apache/incubator-mxnet https://gluon.mxnet.io | https://github.com/gluon-api https://aws.amazon.com/sagemaker https://github.com/awslabs/amazon-sagemaker-examples https://github.com/aws/sagemaker-python-sdk https://github.com/aws/sagemaker-spark https://medium.com/@julsimon https://youtube.com/juliensimonfr https://gitlab.com/juliensimon/dlnotebooks