Latest developments including hardware and algorithm updates presented at the London Deep Learning Lab meetup https://www.meetup.com/Deep-Learning-Lab/
7. What is Intelligence?
• Intelligence is an Agent’s ability to adapt to and to
achieve goals within its Environment
• Human vs machine intelligence – ultimately the same
• “The term artificial intelligence is somewhat
nonsensical. Something is either intelligent or it isn’t.
Just as something either flies or it doesn’t. We don’t
talk about artificial flying” - Zoubin Ghahramani,
Cambridge University
• Information processing, computation, physics,
hardware
• Exploration vs exploitation
• Biological (any species) versus machine (any type)
•Does substrate matter - carbon vs silicon?
7AI Developments v0.10 Peter Morgan August 2017
8. What is Learning?
• Learning algorithms – system gets better
with more data until no further
improvement
• Train the system – just like animals learn
• Supervised, unsupervised and
reinforcement learning
• Physically, it is the strengthening of
connections (synapses) between nodes
(neurons)
• Memory (short and long term) is involved
• Deep learning is a step towards the goal of
artificial general intelligence (AGI)
• Ensemble of techniques
8AI Developments v0.10 Peter Morgan August 2017
12. Deep Learning Is Eating the World
• What about the “AI winters”? 1974–80 and 1987–93, where AI companies
over-promised and under-delivered https://en.wikipedia.org/wiki/AI_winter
• Due to more labeled data, more compute power, better optimization
algorithms, and better neural net models and architectures, deep learning
has started to supersede humans when it comes to image recognition and
classification
• Work is being done to obtain similar levels of performance in natural
language processing and understanding
• According to Jeff Dean in a recent interview, Google have implemented DL in
over one hundred of their products and services including search and photos
• AI is enjoying a renaissance now, not simply because of the promise it holds
for the future but because of the impact it is having on businesses today
AI Developments v0.10 Peter Morgan August 2017 12
16. AI Developments v0.10 Peter Morgan August 2017 16
• Hardware (compute) – Nvidia GPU, Intel (Nervana), AMD Radeon
• Data available - structured and unstructured
• Research activity
• Conference attendance (e.g., NIPS)
• Meetup groups
• PhD enrolments in CS and machine learning
• Performance measures - chess, Jeopardy, Go, computer vision,
language processing, ...
• Number of papers being published in AI/ML
AI Trends 1
19. AI Developments v0.10 Peter Morgan August 2017 19
• Venture capital investment in AI startups
• Number of press/news articles
• Announcements from AI experts with 30 years experience confirming that
this time is for real - there will be no more AI winters
• Number of professors being hired way from academia to join AI
companies , e.g. Uber and CMU, Google and Oxford, Facebook, Apple, etc.
• Government level panels on the development and impact of AI on jobs,
society and policy, e.g., Whitehouse and U.K. parliament
• AI Safety consortium announced last month between Google, Microsoft,
IBM and Amazon to track developments in AI
• Recent books published by professors and engineers on AI development
AI Trends 4
30. Where does the data come from?
• Science – particle, astrophysics
• Industry – oil, finance, telecom (all verticals)
• Social – Facebook, LinkedIn, Twitter
• Medicine – genome, neuroscience
• Government – census, education, police
• Sports – statistics
• Environment – weather, sensors
30AI Developments v0.10 Peter Morgan August 2017
37. LSTM and NTM
• Long Short Term Memory (LSTM)
• LSTM (Schmidhuber, 1997) is an RNN architecture that contains blocks that can
remember a value for an arbitrary length of time
• It solves the vanishing or exploding gradient problem when calculating back
propagation
• An LSTM network is universal in the sense that given enough network units it can
compute anything a conventional computer can compute, provided it has the proper
weight matrix
• LSTM outperforms alternative RNNs and Hidden Markov Models and other sequence
learning methods in numerous applications, e.g., in handwriting recognition, speech
recognition and music composition
• Neural Turing Machines (NTM)
• NTMs are a method of extending the capabilities of recurrent neural networks by
coupling them to external memory resources
37AI Developments v0.10 Peter Morgan August 2017
38. Optimizations
• Initializers
Constant, Uniform, Gaussian, Glorot Uniform, Xavier, Kaiming, IdentityInit, Orthonormal
• Optimizers
Gradient Descent with Momentum, RMSProp, Adadelta, Adam, Adagrad, MultiOptimizer
• Activations Rectified Linear, Softmax, Tanh, Logistic, Identity, ExpLin
• Layers Linear, Convolution, Pooling, Deconvolution, Dropout, Recurrent, Long Short-
Term Memory, Gated Recurrent Unit, BatchNorm, LookupTable, Local Response Normalizat
ion, Bidirectional-RNN, Bidirectional-LSTM
• Costs Binary Cross Entropy, Multiclass Cross Entropy, Sum of Squares Error
• Metrics, Misclassification
(Top1, TopK), LogLoss, Accuracy, PrecisionRecall, ObjectDetection
AI Developments v0.10 Peter Morgan August 2017 38
50. Cognitive Toolkit
• Microsoft open source deep learning framework (Jan 25, 2016)
• Version 2.0 released Oct 25, major upgrade
• Renamed CNTK to Microsoft Cognitive Toolkit
• Announced partnership with Nvidia and OpenAI (Elon Musk backed
AI startup), Nov 16
• Languages are Python, C++ or BrainScript
• Can run on Azure GPU’s
• https://www.microsoft.com/en-us/research
/product/cognitive-toolkit/
AI Developments v0.10 Peter Morgan August 2017 50
53. Types of Hardware
• Sensors, processors, storage, memory, network
• Processors - CPU, GPU, FPGA, ASIC, NPU, QPU
• GPU - Graphics Processing Units were first brought to market by Nvidia in 2007 to
meet the demands of the gaming market
• Massively parallel processing (MPP)
• 100 x speedup compared with CPU’s
• Widespread application – science, industry, government
• Nvidia www.nvidia.com
• Intel Xeon Phi http://www.intel.com/content/www/us/en/processors/xeon/xeon-
phi-detail.html
• AMD Radeon www.amd.com
53AI Developments v0.10 Peter Morgan August 2017
75. DLaaS - Cloud Services
• DL/ML as a Service is offered by all the major cloud providers
• AWS https://aws.amazon.com/machine-learning/
• Azure https://azure.microsoft.com/en-us/services/machine-learning/
• GCP https://cloud.google.com/products/machine-learning/
• Bluemix https://www.ibm.com/cloud-computing/bluemix/watson
AI Developments v0.10 Peter Morgan August 2017 75
78. AI Conferences – Research Focussed
• NIPS = Neural Information Processing Systems https://nips.cc/
• IJCNN = International Joint Conference on Neural Networks http://www.ijcnn.org/
• IJCAI = International Joint Conference on Artificial Intelligence http://ijcai.org/
• ICANN = International Conference on Artificial Neural Networks http://www.icann2017.org/
• IWANN = International Work-Conference on Artificial Neural Networks http://iwann.uma.es/
• ICONIP = International Conference on Neural Information Processing http://www.iconip2017.org/papers.html
• ICAART = International Conference on Agents and Artificial Intelligence http://www.icaart.org/
• ISIS = International Symposium on Advanced Intelligent Systems http://isis2017.org/
• AAAI = Association of Advancement of Artificial Intelligence
http://www.aaai.org/Conferences/conferences.php
• ACM = Association of Computing Machinery https://www.acm.org/conferences
• AGI = Artificial General Intelligence Conference http://agi-conf.org/
• TensorCon = TensorFlow Conference https://ti.to/TensorCon/
78AI Developments v0.10 Peter Morgan August 2017
80. New Developments
• Multi-modal learning, Transfer learning, One-shot learning, GANs
• Better reinforcement learning / integration of deep learning and
reinforcement learning
• Better generative models. Algorithms that can reliably learn how to
generate images, speech, text that humans can’t tell apart from the
real thing
• Learning to learn and ubiquitous deep learning: algorithms that
redesign their own architecture, tune their own hyperparameters,
etc. Right now it still takes a human expert to run the learning-to-
learn algorithm, but in the future it will be easier to deploy, and all
kinds of business that don’t specialize in AI will be able to leverage
deep learning
AI Developments v0.10 Peter Morgan August 2017 80
82. Where are we headed?
AI Developments v0.10 Peter Morgan August 2017 82
World Economic Forum (WEF) Report, 2016:
Today, we are at the beginning of a Fourth Industrial Revolution.
Developments in genetics, artificial intelligence, robotics, nanotechnology, 3D
printing and biotechnology, to name just a few, are all building on and
amplifying one another. This will lay the foundation for a revolution more
comprehensive and all-encompassing than anything we have ever seen
Deepmind Mission:
Solve intelligence then use it to solve everything
else
83. AI Related Books
• Bengio, Yoshua et al, Deep Learning, MIT Press, 2016
• Brynjolfsson, Erik and Andrew McAfee, The Second Machine Age, W.W. Norton &
Co., 2014
• Chollet, Francois, Deep Learning with Python, Manning, Oct 2017
• Domingos, Pedro, The Master Algorithm, Basic Books, 2015
• Ford, Martin, Rise of the Robots: Technology and the Threat of a Jobless Future,
Basic Books, 2015
• Kaku, Michio, The Future of the Mind, Doubleday, 2014
• Kurzweil, Ray, How to Create a Mind, Penguin Books, 2013
• Russell and Norvig, Artificial Intelligence, A Modern Approach, Pearson, 2009
• Shanahan, Murray, The Technological Singularity, MIT Press, 2015
• Yampolskiy, Roman, Artificial Superintelligence, A Futuristic Approach, CRC, 2015
83AI Developments v0.10 Peter Morgan August 2017
85. References
• Brtiz, D. et al, Massive Exploration of Neural Machine Translation Architectures,
Mar 2017 https://arxiv.org/abs/1703.03906
• Johnson, M. et al, Google's Multilingual Neural Machine Translation System:
Enabling Zero-Shot Translation, Nov 2016 https://arxiv.org/abs/1611.04558
• Gehring, J. et al, Convolutional Sequence to Sequence Learning, May 2017
https://arxiv.org/abs/1705.03122
• Feedback Networks
http://feedbacknet.stanford.edu/feedback_networks_2016.pdf
• AI safety discussion https://www.facebook.com/groups/467062423469736/
• Google ICML 2017 Publications
https://research.googleblog.com/2017/08/google-at-icml-2017.html
AI Developments v0.10 Peter Morgan August 2017 85