Introduction to deep learning

1. Introduction to Deep
Learning
Vishwas Lele (@vlele)
Applied Information Sciences

2. Agenda
•Introduction to Deep Learning
• From ML to Deep Learning
•Neural Network Basics
• Dense, CNN, RNN
• Key Concepts
• CNN Basics
•X-ray Image Analysis
• Optimization (using pre-trained models)
• Visualizing the learning

4. Demo
• Why do I need to learn neural networks when I have Cognitive and
other APIs?

5. What is Deep Learning?

6. Deep Learning Examples
• Near-human-level image classification
• Near-human-level speech recognition
• Near-human-level handwriting transcription
• Digital assistants Google Now, Cortana
• Near-human-level autonomous driving

7. Why now? What about past AI winters?
• Hardware
• Datasets and benchmarks
• Algorithmic advances

8. What about other ML techniques?
• Probabilistic modeling (Naïve Bayes)
• Kernel Methods
• Support Vector Machines
• Decision Trees
• Random Forests
• Gradient Boosting Machines

9. “Deep” in deep learning

10. How can a network help us?
0
1
2
3
4
7
5
6
9
8

11. Digit Classification

12. Network of layers
• Layers are like LEGO bricks of deep learning
• A data-processing module that takes as input one or more tensors
and that outputs one or more tensors

13. Tensor
shape (128, 256, 256, 1)

14. How layers transform data?
output = relu(dot(W, input) + b)
W- Weight
b - Bias

15. Rectified Linear Unit (RELU)

16. Power of parallelization
def naive_relu(x):
i in range(x.shape[0]): for j in range(x.shape[1]): x[i, j] = max(x[i, j], 0)
return x
z = np.maximum(z, 0.)

17. Sigmoid function

18. How network learns?

19. Training Loop
• Draw a batch of training samples x and corresponding targets y.
• Run the network on x
• Compute the loss of the network
• Update all weights of the network in a way that slightly reduces the
loss on this batch.

20. Key Concepts
• Derivative
• Gradient
• Stochastic Gradient

21. Stochastic gradient descent
• Draw a batch of training samples x and corresponding targets y.
• Run the network on x to obtain predictions y_pred.
• Compute the loss of the network on the batch, a measure of the
mismatch between y_pred and y.

22. Stochastic gradient descent
• Compute the gradient of the loss with regard to the network’s
parameters (a backward pass).
• Move the parameters a little in the opposite direction from the
gradient—for example W = step * gradient—thus reducing the loss on
the batch a bit.

23. Back propagation
0
1
2
3
4
7
5
6
9
8

24. Demo
• Digit Classification
• Keras Library

25. Training and validation loss

26. Overfitting

27. Weight Regularization
• L1 regularization— The cost added is proportional to the absolute
value of the weight coefficients
• L2 regularization— The cost added is proportional to the square of
the value of the weight coefficients

28. Dropout
• Randomly dropping out (setting to zero) a number of output features
of the layer during training.

29. Feature Engineering
• process of using your own knowledge to make the algorithm work
better by applying hardcoded (nonlearned) transformations to the
data before it goes into the model

30. Feature engineering

31. How network learns?

32. Enter CNN or (Convolution Neural Networks)

33. Convolution Operation

34. Spatial Hierarchy

35. How Convolution works?

36. Sliding

37. Padding

38. Stride

39. Max-Pooling
• Extracting windows from the input feature maps and outputting the
max value of each channel.
• It’s conceptually similar to convolution.
• Except that instead of transforming local patches via a learned linear
transformation

40. Demo
• Digit Classification (using CNN)
• Keras Library

41. Demo
• X-ray Image Classification
• Dealing with overfitting
• Visualizing the learning

42. Visualizing the learning

43. Overfitting

44. With dropout regularization

45. With data augmentation

46. Demo
• Activation heatmap