This document provides an introduction to deep learning using Intel Nervana. It discusses machine learning basics like supervised and unsupervised learning. It then introduces deep learning concepts like neural network layers and architectures. An example of classifying handwritten digits with a multi-layer perceptron is presented to demonstrate basic deep learning concepts like forward propagation, backpropagation, and updating weights. The document also discusses model ingredients like data, parameters, optimization techniques, and frameworks like Neon that can be used for deep learning.
2. Nervana Systems Proprietary
2
• Intel Nervana overview
• Machine learning basics
• What is deep learning?
• Basic deep learning concepts
• Example: recognition of handwritten digits
• Model ingredients in-depth
• Deep learning with neon
4. Nervana Systems Proprietary
Deep Dream
Autoencoders
Deep Speech 2
Skip-thought
SegNet
Fast-RCNN Object Localization
Deep Reinforcement Learning
imdb Sentiment Analysis
Video Activity Detection
Deep Residual Net
bAbI Q&A
AIICNN AlexNet GoogLeNet
VGG
https://github.com/NervanaSystems/ModelZoo
8. Nervana Systems Proprietary
8
• Intel Nervana overview
• Machine learning basics
• What is deep learning?
• Basic deep learning concepts
• Example: recognition of handwritten digits
• Model ingredients in-depth
• Deep learning with neon
9. Nervana Systems Proprietary
9
• SUPERVISED LEARNING
• DATA -> LABELS
• UNSUPERVISED LEARNING
• NO LABELS; CLUSTERING
• REDUCING DIMENSIONALITY
• REINFORCEMENT LEARNING
• REWARD ACTIONS (E.G., ROBOTICS)
10. Nervana Systems Proprietary
10
• SUPERVISED LEARNING
• DATA -> LABELS
• UNSUPERVISED LEARNING
• NO LABELS; CLUSTERING
• REDUCING DIMENSIONALITY
• REINFORCEMENT LEARNING
• REWARD ACTIONS (E.G., ROBOTICS)
11. Nervana Systems Proprietary
11
(𝑓#, 𝑓%, … , 𝑓')
SVM
Random Forest
Naïve Bayes
Decision Trees
Logistic Regression
Ensemble methods
𝑁×𝑁
𝐾 ≪ 𝑁
Arjun
16. Nervana Systems Proprietary
16
• Intel Nervana overview
• Machine learning basics
• What is deep learning?
• Basic deep learning concepts
• Example: recognition of handwritten digits
• Model ingredients in-depth
• Deep learning with neon
17. Nervana Systems Proprietary
17
~60 million parameters
Arjun
But old practices apply:
Data Cleaning, Underfit/Overfit, Data exploration, right cost function, hyperparameters, etc.
𝑁×𝑁
18. Nervana Systems Proprietary
18
Bigger Data Better Hardware Smarter Algorithms
Image: 1000 KB / picture
Audio: 5000 KB / song
Video: 5,000,000 KB / movie
Transistor density doubles
every 18 months
Cost / GB in 1995: $1000.00
Cost / GB in 2015: $0.03
Advances in algorithm
innovation, including neural
networks, leading to better
accuracy in training models
20. Nervana Systems Proprietary
20
• Intel Nervana overview
• Machine learning basics
• What is deep learning?
• Basic deep learning concepts
• Model ingredients in-depth
• Deep learning with neon
23. Nervana Systems Proprietary
MNIST dataset
70,000 images (28x28 pixels)
Goal: classify images into a digit 0-9
N = 28 x 28 pixels
= 784 input units
N = 10 output units
(one for each digit)
Each unit i encodes
the probability of the
input image of being of
the digit i
N = 100 hidden units
(user-defined
parameter)
Input
Hidden
Output
50. Nervana Systems Proprietary
50
• Intel Nervana overview
• Machine learning basics
• What is deep learning?
• Basic deep learning concepts
• Model ingredients in-depth
• Deep learning with neon
61. Nervana Systems Proprietary
61
• Intel Nervana overview
• Machine learning basics
• What is deep learning?
• Basic deep learning concepts
• Model ingredients in-depth
• Deep learning with neon
64. Nervana Systems Proprietary
•Popular, well established, developer familiarity
•Fast to prototype
•Rich ecosystem of existing packages.
•Data Science: pandas, pycuda, ipython, matplotlib, h5py, …
•Good “glue” language: scriptable plus functional and OO support,
plays well with other languages
66. Nervana Systems Proprietary
1. Generate backend
2. Load data
3. Specify model architecture
4. Define training parameters
5. Train model
6. Evaluate