Brief Introduction on Deep Learning

1. Introduc)on
to
Deep
Learning
Massimiliano
Ruocco

2. Outline
• Introduction and Motivation for DL
• From NN to Deep Learning
• Deep Learning Models
• Deep Learning in the Real World
• Conclusion

3. Introduction and
Motivation for DL

4. Introduction
Deep Learning - WHAT
Class of ML training algorithm

5. Introduction
Deep Learning - Motivations
• ML Algorithms:
– Supervised
– Unsupervised
– Semi-supervised
– Reinforcement Learning

6. • ML Algorithms: unsupervised learning
Data
Representa)on
Input
Clustering
Output
Example (Marketing/Customer segmentation):
• Input : Customers of a specific product
• Output: Customer subgroups
Introduction
Deep Learning - Motivations

7. • ML Algorithms: supervised learning
Data
Representa)on
Input
Classifica)on/
Regression
Output
Training
Labeled
DataSet
Data
Representa)on
Example (spam detection):
• Input : Email
• Output: Spam/NotSpam
• Training Set: Data set of mail labeled as Spam/Not Spam
Introduction
Deep Learning in ML and AI

8. • ML Algorithms: supervised learning
Data
Representa)on
Input
Classifica)on/
Regression
Output
Training
Labeled
DataSet
Data
Representa)on
Example (spam detection):
• Input : Email
• Output: Spam/NotSpam
• Training Set: Data set of mail labeled as Spam/Not Spam
Introduction
Deep Learning – Representation Problem

9. • Data Representation:
– feature set selection
– #features
• Main Issues:
– Course of dimensionality
– Overfitting
– Handcrafted features
• How to tackle: Representation Learning
Introduction
Deep Learning – Representation Problem

10. • Deep learning methods:
– Representations are expressed in terms of
other, simpler representations
Introduction
Deep Learning - WHAT

11. • Deep Learning algorithm as application of Machine
Learning to Artificial intelligence
Ar#ficial
Intelligence
(i.e.
knowledge
bases)
Machine
Learning
(i.e.
Support
Vector
Machine)
Representa#on
Learning
(i.e.
Autoencoders)
Deep
Learning
(i.e.
Mul=layer
Perceptron)
Introduction
Deep Learning in ML and AI

12. From Neural Network to
Deep Learning

13. • Neural Network: Basic
– Different layers of neurons/perceptrons
– Human brain analysis
– Input, Hidden Layer, Output
• Neural Network: Applications
– Classification (Spam Detection)
– Pattern Recognition (Character recognition)
Introduction
From Neural Network to Deep Learning

14. • The core: Neuron
Introduction
From Neural Network to Deep Learning
W1
W2
W3
x1
x2
xn
Sigmoid
func)on
1/(1+e-­‐z)
Output
hw(x)
x
=
[x0…xn]T
w
=
[w0…wn]T
z
=
wTx

15. • Neural Network – Single layer
Introduction
From Neural Network to Deep Learning

16. • Forward Propagation:
– process of computing the output
Introduction
From Neural Network to Deep Learning
x1
x2
x3
a1
2
a2
2
W(1)
W(2)
a(2)
z(2)
z(3)
X
z(2)
=
XW(1)
a(2)
=
f(z(2))
z(3)
=
a(2)W(2)
y
=
f(z(3))

17. • Training a Neural Network:
– Learning the parameters (weights)
• Supervised
• Unsupervised
• Reinforcement Learning
• Employing a Neural Network:
– Selecting the Architecture
– # Layers
– # Units per layer
– Kind of learning algorithm
Introduction
From Neural Network to Deep Learning

18. • Training a Neural Network:
– Backward Propagation
• Gradient descent
• Objective: Minimize the cost function J
Introduction
From Neural Network to Deep Learning
x1
x2
x3
a1
2
a2
2
W(1)
W(2)
a(2)
z(2)
z(3)
X

19. • DNN à Typically artificial neural netwok
with 3 or more levels of non-linear
operations
Introduction
From Neural Network to Deep Learning

20. • Using Back propagation for Deep NN
– Does not scale
– Bad performance for random initialization
– Local Optima
– Vanishing gradient problem
Introduction
Issues in Training DNN

21. Introduction
The Breakthrough
2006*+
Backward
Propaga#on
Greedy-­‐layer
wise
training
+
Supervised
fine
tuning
* Hinton et al. A fast learning algorithm for deep belief nets.
Neural Computation, 18:1527–1554, 2006
+ Ranzato et al. Efficient learning of sparse representations with an energy-based model.
Advances in Neural Information Processing Systems 19 (NIPS’06),

22. • Deep learning methods:
– Class of ML algorithm
– Use cascade of many levels of non linear
processing units for feature extraction
– Hierarchy of concepts
– Multiple-layered model
– NN with high number of hidden layers
– NEW LEARNING ALGORITHM Overcoming previous
training problems
Introduction
Deep Learning - Summary

23. Deep
Learning
Models

24. Deep Learning Models
Introduction
• Two main classes:
– Generative
• Deep Network for supervised Learning
– Discriminative
• Deep Network for unsupervised learning
– Hybrid

25. Deep Learning Models
Generative – Deep Belief Network
• Generative graphic model
• Mix directed and undirected between vars
• Learn to reconstruct the input

26. Deep Learning Models
Generative – Deep Belief Network
• Training algorithm
– Iteratively apply RBM training to each pair of
layers

27. Deep Learning Models
Discriminative – Convolutional NN
• CNN in Computer Vision: Image Recognition
– Feed-forward multilayer network
– Kind of back propagation for learning
– Receptive fields
– Learn suitable representation of the image

28. Deep Learning Models
Discriminative – Convolutional NN
• CNN in Computer Vision: Image Recognition
– Key concepts:
• Max pooling
• Sparse Connectivity
• Convolution

29. Deep
Learning
in
the
Real
World

30. • NLP
• Image Classification/Computer Vision
• Speech Recognition
Introduction
Deep Learning – Application Field

31. • [Google] 2013
acquired DNNresearch of professor Geoff
Hinton to improve the state of the art in
image recognition in photos
• [Facebook] 2013
hired deep learning expert Yann to head up
the company’s new artificial intelligence lab
specialized in deep learning for computer
vision and image recognition
• [Pinterest] 2014
announced it has acquired Visual Graph
• [Google + Baidu]:
20G13 - Deep Learning Visual Search Engine
Deep Learning in the Real World
Facts

32. • [Baidu] 2013:
Deep Learning Visual Search Engine
• [Google] 2013
Photo Search Engine
• [Microsoft] 2013
Search by voice on Xbox console
• [Google] 2014
word2vec for word tagging or text messaging
suggestion
Deep Learning in the Real World
Products

33. Thanks
for
the
aUen)on