Understanding Deep Learning
•
0 gostou•182 visualizações
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2QcNaOA. Jessica Yung talks about the foundational concepts about neural networks. She highlights key things to pay attention to: learning rates, how to initialize a network, how the networks are constructed and trained, and understanding why these parameters are so important. She ends the talk with practical takeaways used by state-of-the-art models to help us start building neural networks. Filmed at qconlondon.com. Jessica Yung is a research masters student in ML at University College London. She was previously at the University of Cambridge and was an NVIDIA Self-Driving Car Engineer Scholar. She applied machine learning to finance at Jump Trading and consults on machine learning. She is keen on sharing knowledge and writes about ML and how to learn effectively on her blog at JessicaYung.com.
Recomendados
Mais conteúdo relacionado
Semelhante a Understanding Deep Learning
Semelhante a Understanding Deep Learning (20)
Mais de C4Media
Mais de C4Media (20)
Último
Último (20)
Understanding Deep Learning
- 2. InfoQ.com: News & Community Site Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ understanding-deep-learning • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon London www.qconlondon.com
- 4. Time Price
- 5. ? Time Price
- 6. ? Time Price
- 11. learning rate
- 12. learning rate
- 15. def debug(): return 0 + “no”
- 16. 567 563m8dep j6lmj? _ g “?@”
- 26. Outline 1. Single layer and putting layers together 2. How to train models 3. Deeper is better 4. Practical tips
- 27. Understanding is not something you can Google.
- 29. I. Single Layer
- 35. Fully connected layer + Linear layer Non-linearity
- 36. A single layer Linear layer Non-linearity
- 37. Linear layer
- 49. Output layer
- 52. Sigmoid tanh
- 56. Sigmoid
- 57. MNIST dataset
- 58. [0.0003 0.047 0.95] Softmax(3) [-1, 4, 7]
- 59. Wider Deeper
- 60. Wider Deeper
- 63. Time of day Coffee needed Keynote Break Lunch Talk!
- 64. In
- 65. In Sigmoid
- 66. In
- 76. In
- 77. In
- 79. + =
- 80. + =
- 81. + = - =
- 82. + = - =
- 83. In
- 84. In
- 85. Time of day Coffee needed Keynote Break Lunch Talk!
- 86. Time of day Coffee needed Keynote Break Lunch Talk!
- 87. Time of day Coffee needed Keynote Break Lunch Talk!
- 88. Time of day Coffee needed Keynote Break Lunch Talk!
- 89. Wider Deeper
- 90. Foundational for CNNs and RNNs
- 91. Convolutional layers 1. Translation invariance 2. Locality
- 92. W1 W1 W1 Output
- 96. Recurrent layers
- 99. Recurrent layers +Linearlayer Non-linearity Input xt +Linearlayer Non-linearity +Linearlayer Non-linearity previous state/memory
- 100. Recurrent layers +Linearlayer Non-linearity Input xt Output +Linearlayer Non-linearity +Linearlayer Non-linearity previous state/memory
- 101. II. How train?
- 104. Parameters
- 105. Parameters Reward
- 106. Parameters Reward
- 107. Parameters Reward
- 122. step size x gradient
- 123. Gradient descent
- 126. Gradient descent Learning rate (step size) Gradient Parameters
- 128. Loss Iterations
- 129. Loss Iterations
- 130. gradient x step size Loss Iterations
- 131. gradient x step size Loss Iterations
- 132. gradient x step size Loss Iterations
- 133. Loss Iterations
- 134. Loss Iterations
- 135. gradient x step size Loss Iterations
- 136. gradient x step size Loss Iterations
- 137. gradient x step size Loss Iterations
- 138. Loss Iterations
- 139. Loss Iterations
- 140. Loss Iterations gradient x step size
- 141. Loss Iterations gradient x step size
- 142. Loss Iterations gradient x step size
- 143. Loss Iterations
- 144. Loss Iterations
- 145. gradient x step size Loss Iterations
- 146. gradient x step size Loss Iterations
- 147. gradient x step size Loss Iterations
- 148. Dense 1 Dense 2 Linear Output Input Softmax(3) Iris dataset
- 159. Dense 1 Dense 2 Dense 10 … Output Input
- 160. Dense 1 Dense 2 Dense 10 … Output Input
- 161. Dense 1 Dense 2 Dense 10 … Output Input
- 162. Dense 1 Dense 2 Dense 10 … Output Input
- 163. Dense 1 Dense 2 Dense 10 … Output Input
- 164. Dense 1 Dense 2 Dense 10 … Output Input
- 165. Dense 1 Dense 2 Dense 10 … Output Input
- 166. Dense 1 Dense 2 Dense 10 … Output Input Backpropagation
- 167. Dense 1 Dense 2 Dense 10 … Output Input Backpropagation
- 168. Dense 1 Dense 2 Dense 3 Output Input
- 169. Dense 1 Dense 2 Dense 3 Output Input
- 171. A single layer
- 172. A single layer Linear layer Non-linearity
- 173. A single layer Linear layer Non-linearity x
- 174. A single layer Linear layer Non-linearity x
- 178. Initialise weights with SMALL variance!
- 179. Initialise weights with SMALL variance! e.g. Glorot initialisation
- 180. A single layer Linear layer Non-linearity x
- 181. A single layer Linear layer Non-linearity x
- 182. A single layer Linear layer Non-linearity x
- 183. Initialise weights with SMALL variance!
- 184. Initialise weights with SMALL variance! e.g. Glorot initialisation
- 185. III. Deeper is better
- 186. ImageNet
- 187. ImageNet leopard
- 190. sabre-toothed tiger snow leopardcat ImageNet leopard tree shrew radiator
- 191. Data from Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. ‘Deep Residual Learning for Image Recognition’, CVPR 2016. Revolution of Depth: Top models on ImageNet Numberoflayers 0 40 80 120 160 2010 2011 2012 AlexNet 2013 2014 VGG 2014 GoogLeNet 2015 ResNet
- 192. Learning rate (step size) Gradient Parameters
- 202. Vanishing gradients
- 209. Exploding gradients
- 212. Vanishing gradients
- 213. Vanishing gradients
- 214. Vanishing gradients
- 215. Skip connections Output Dense 1 Dense 2 Input ResNets (He et. al., 2015)
- 216. Skip connections Output Dense 1 Dense 2 Input + ResNets (He et. al., 2015)
- 217. Data from Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. ‘Deep Residual Learning for Image Recognition’, CVPR 2016. Revolution of Depth: Top models on ImageNet Numberoflayers 0 40 80 120 160 2010 2011 2012 AlexNet 2013 2014 VGG 2014 GoogLeNet 2015 ResNet
- 218. IV. Practical Tips
- 219. Overfit First
- 220. Model a few examples
- 223. Batchnorm
- 224. Output Dense 1 Dense 2 Input Batchnorm Batchnorm Output Dense 1 Dense 2 Input Ioffe and Szegedy, 2015
- 225. Output Dense 1 Dense 2 Input Batchnorm Batchnorm Ioffe and Szegedy, 2015
- 226. Transfer Learning
- 227. Based on Zeiler and Fergus, 2013 § Low Mid High
- 231. Trained network Last layer(s) New Mid/high level features e.g. ResNet
- 232. Trained network New output Last layer(s) New Mid/high level features e.g. ResNet
- 233. Simplify
- 234. VGG-19 2 x conv-64 pool/2 2 x conv-128 pool/2 4 x conv-256 pool/2 4 x conv-512 pool/2 4 x conv-512 pool/2 3 x dense softmax Simonyan and Zisserman, 2014
- 235. Dense 1
- 238. Time and Effort Model performance
- 239. Time and Effort Model performance
- 240. Time and Effort Model performance
- 241. Thank You!
- 242. Appendix
- 243. Resources on Coding Neural Networks • Jason Brownlee’s Machine Learning Mastery (lots of code) • My blog posts on training models in TensorFlow (training a model, MLP, CNN)
- 244. Resources on Deep Learning • THE Deep Learning book (Ian Goodfellow, Yoshua Bengio, Aaron Courville) • Stanford’s course on Convolutional Neural Networks • fast.ai courses on neural networks (haven’t tried much myself but I’ve heard they are good)
- 245. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ understanding-deep-learning