Artificial agents interacting in highly dynamic environments are required to continually acquire and fine-tune their knowledge overtime. In contrast to conventional deep neural networks that typically rely on a large batch of annotated training samples, lifelong learning systems must account for situations in which the number of tasks is not known a priori and the data samples become incrementally available over time. Despite recent advances in deep learning, lifelong machine learning has remained a long-standing challenge due to neural networks being prone to catastrophic forgetting, i.e., the learning of new tasks interferes with previously learned ones and leads to abrupt disruptions of performance. Recently proposed deep supervised and reinforcement learning models for addressing catastrophic forgetting suffer from flexibility, robustness, and scalability issues with respect to biological systems. In this tutorial, we will present and discuss well-established and emerging neural network approaches motivated by lifelong learning factors in biological systems such as neurosynaptic plasticity, complementary memory systems, multi-task transfer learning, and intrinsically motivated exploration.

1. Continual Lifelong Learning
with Neural Networks
April 16, 2019 -Tutorial @ INNSBDDL2019

2. A Practical Example
• 50GB/s streaming
data.
• ~30240TB of data
after only a week.
• Impossible to re-train
the mini-spot brain
from scratch and to
adapt fast.
Mini-spot Robot from Boston Dynamics, 2018

3. Continual Learning (CL)
• Ability to continually acquire, fine-tune, and transfer
new knowledge and skills
• Higher and realistic time-scale where data (and tasks)
become available only during time.
• No access to previously encountered data.
• Constrained computational and memory resources.

4. paper

5. Catastrophic forgetting
• Training a model with new information interferes with
previously learned knowledge
• Abrupt performance decrease or old knowledge
completely overwritten by the new one.

6. Catastrophic forgetting
• Training a model with new information interferes with
previously learned knowledge
• Abrupt performance decrease or old knowledge
completely overwritten by the new one.

7. The Stability-Plasticity Dilemma
Stability-Plasticity Dilemma:
• Remember past concepts
• Learn new concepts
• Generalize
Biggest Problem in Deep Learning:
• Catastrophic Forgetting

8. Biological factors of CL
• Structural Plasticity
• Neurosynaptic adaptation to changes in the environment
• Change of physical structure as the result of learning
• Stability-plasticity balance
• Complementary Learning Systems
• Retaining episodic memories (memorization)
• Extracting statistical structure (generalization)
• Memory replay

9. Biological factors of CL
•

10. Neural Network Architectures

11. ElasticWeights Consolidation (EWC)
Fisher Information
...

12. Growing Networks
Parisi,Tani,Weber,Wermter. Lifelong Learning of Spatio-temporal Representations with Dual-Memory
Recurrent Self-Organization. Frontiers in Neurorobotics 2019.

13. Neurogenesis

14. CL Strategies
Architectural
Regularization Rehearsal
CWR PNN
EWC
SI
LWF
ICARL
AR1
GEM
Pure
Rehearsal
GDM
Exstream

15. Continual Learning
Benchmarks

16. Supervised CL benchmarks
Dataset Strategy
Permuted MNIST EWC, GEM, SI, ...
Rotated MNIST GEM
MNIST Split SI
CIFAR10/100 Split GEM, iCARL, SI, AR1, ...
ILSVRC2012 iCARL
CUB-200 GMD, ...
LomonacoV. and Maltoni D. CORe50: a New Dataset and Benchmark for Continuous Object Recognition. CoRL2017.

17. Sequential CL benchmarks
LomonacoV. and Maltoni D. CORe50: a New Dataset and Benchmark for Continuous Object Recognition. CoRL2017.

18. CRL Environments
Environments Scenarios
Atari Multiple 2D games
DeepMind Lab
Maze Exploration,
Object Picking
Malmo Multiple tasks
OpenAI Gym Multiple 3D tasks
MuJoCo Multiple Joint Stiffness
VizDoom -
Unity 3D -
StarCraft II Curriculum learning

19. Some References for CRL
• Al-Shedivat, Maruan, et al. "Continuous adaptation via meta-learning in
nonstationary and competitive environments." arXiv preprint arXiv:1710.03641
(2017).
• Tessler, Chen, et al. "A deep hierarchical approach to lifelong learning in
minecraft."Thirty-First AAAI Conference on Artificial Intelligence. 2017.
• Kirkpatrick, James, et al. "Overcoming catastrophic forgetting in neural
networks." Proceedings of the national academy of sciences 114.13 (2017):
3521-3526.
• Schwarz, Jonathan, et al. "Progress & compress: A scalable framework for
continual learning." arXiv preprint arXiv:1805.06370 (2018).
• Kaplanis, Christos, Murray Shanahan, and Claudia Clopath. "Continual
reinforcement learning with complex synapses." arXiv preprint arXiv:1802.07239
(2018).

20. LomonacoV. and Maltoni D. CORe50: a New Dataset and Benchmark for Continuous Object Recognition. CoRL2017.
CORe50: aVideo Benchmark for CL and Object
Recognition, Detection and Segmentation

21. # Images 164,866
Format RGB-D
Image size 350x350
128x128
# Categories 10
# Obj. x Cat. 5
# Sessions 11
# img. x Sess. ~300
# Outdoor Sess. 3
Acquisition Sett. Hand held
LomonacoV. and Maltoni D. CORe50: a New Dataset and Benchmark for Continuous Object Recognition. CoRL2017.
CORe50: aVideo Benchmark for CL and Object
Recognition, Detection and Segmentation

22. CORe50 Benchmark
(NI)
(NC)
(NIC)

23. CORe50Website
Dataset, Benchmark, code and additional
information freely available at:
vlomonaco.github.io/core50

24. CRL in 3D non-stationary environment
LomonacoV., Desai K., Maltoni D. and Culurciello, E. Continual Reinforcement Learning in 3D non-stationary
environments. Submitted to ECML-PKDD, 2019.
VIDEO!

25. ConsideredVariations

26. Lifelong Learning
Metrics

27. CL Framework and Metrics
CL Algorithm
N. Díaz-Rodríguez,V. Lomonaco et al. Don't forget, there is more than forgetting: new metrics for Continual Learning.
CLWorkshop, NeurIPS 2018.

28. Continual Learning:
Where to start?

29. ContinualAI non-profit Research Organization
http://continualai.org https://continualai.herokuapp.com/

30. A Gentle Introduction to CL in PyTorch
https://github.com/ContinualAI/colab

31. Limitations and FutureWorks
Limitations
• Young line of research
• Theoretical foundations
• Real-world applications
What’s next?
• Towards Biological Synaptic Plasticity, learning and
memory.
• Robustness, flexibility, and efficiency.

32. CL in autonomous agents & robots
• Progressively acquire, fine-tune, and transfer
knowledge and skills through the interaction with the
environment
• Data are temporally correlated and increasingly more
complex
• Active exploration through intrinsic motivation

33. Rethinking CL for autonomous agents

34. Rethinking CL for autonomous agents

35. Rethinking CL for autonomous agents

36. Rethinking CL for autonomous agents