Tensors are higher order extensions of matrices that can incorporate multiple modalities and encode higher order relationships in data. Tensors play a significant role in machine learning through (1) tensor contractions, (2) tensor sketches, and (3) tensor decompositions. Tensor contractions are extensions of matrix products to higher dimensions. Tensor sketches efficiently compress tensors while preserving information. Tensor decompositions compute low rank components that constitute a tensor.

1. AWS re:INVENTRole of Tensors in Large-Scale
Machine Learning
Anima Anandk umar
Pr inc ipal Sc ientis t, AW S
Br en Pr ofes s or, C altec h

2. Tensors are Multi-Dimensional

3. Tensors are Multi-Dimensional
Modern data is inherently multi-dimensional

4. Tensors to encode multi-dimensional data
Images: 3 dimensions

5. Tensors to encode multi-dimensional data
Images: 3 dimensions Videos: 4 dimensions

6. Tensors to encode multi-dimensional data
Are there trees in
the image?
Images: 3 dimensions Videos: 4 dimensions Visual q&a: ? dimensions

7. Tensors to encode higher order moments
Pairwise correlations

8. Tensors to encode higher order moments
Pairwise correlations
Third order correlations

9. Operations on Tensors: Tensor
Contraction

10. Tensorized Neural Networks
h t t p s : / / g i t h u b . c o m / a w s l a b s / a m a z o n - s a g e m a k e r - e x a m p l e s

11. Deep Neural Nets: Transforming Tensors

12. Deep Tensorized Networks
Tensor Contraction & Regression Networks by J. Kossaifi, Z. C. Lipton,A. Khanna,T. Furlanello, A, 2017.

13. Space Saving in Deep Tensorized
Networks

14. Tensorly: Framework for Tensor Algebra
• Python programming
• User-friendly API
• Multiple backends:
flexible + scalable
• Example notebooks in
repository

15. Tensorly with Pytorch backend
import tensorly as tl
from tensorly.random import tucker_tensor
tl.set_backend(‘pytorch’)
core, factors = tucker_tensor((5, 5, 5),
rank=(3, 3, 3))
core = Variable(core, requires_grad=True)
factors = [Variable(f, requires_grad=True) for f in factors]
optimiser = torch.optim.Adam([core]+factors, lr=lr)
for i in range(1, n_iter):
optimiser.zero_grad()
rec = tucker_to_tensor(core, factors)
loss = (rec - tensor).pow(2).sum()
for f in factors:
loss = loss + 0.01*f.pow(2).sum()
loss.backward()
optimiser.step()
Set Pytorch backend
Attach gradients
Set optimizer
TuckerTensor form

16. Speeding up Tensor Contractions
h t t p s : / / g i t h u b . c o m / a w s l a b s / a m a z o n - s a g e m a k e r - e x a m p l e s

17. Speeding up Tensor Contractions

18. Speeding up Tensor Contractions

19. New Primitive for Tensor Contractions

20. Performance of StridedBatchedGEMM
Performance on par with pure GEMM (P100 and beyond).

21. Tensors in Time Series
h t t p s : / / g i t h u b . c o m / a w s l a b s / a m a z o n - s a g e m a k e r - e x a m p l e s

22. Tensors for long-term forecasting
Difficulties in long term forecasting:
• Long-term dependencies
• High-order correlations
• Error propagation
22

23. RNNs: First-Order Markov Models
Input state 𝑥 𝑡, hidden state ℎ 𝑡, output 𝑦𝑡,
ℎ 𝑡= 𝑓 𝑥 𝑡, ℎ 𝑡−1 ; 𝜃 ; 𝑦𝑡 = 𝑔( ℎ 𝑡; 𝜃)
23

24. Tensorized RNNs
L-order Markov
ℎ 𝑡= 𝑓 𝑥𝑡, ℎ 𝑡−1 , ℎ 𝑡−2 , … ℎ 𝑡−𝐿 ; 𝜃 ; 𝑦𝑡 = 𝑔( ℎ 𝑡; 𝜃)
Polynomial Interactions
𝑠𝑡−1 = [1, ℎ 𝑡−1, ℎ 𝑡−2 , … ℎ 𝑡−𝐿 ]
ℎ 𝑡,𝛼 = 𝑓 𝑊ℎ𝑥
𝑥𝑡 + 𝒲 𝛼,𝑖1,…𝑖 𝑃
𝑠𝑡−1;𝑖1
⨂ … ⨂𝑠𝑡−1;𝑖 𝑃
; 𝜃 ;
24

25. Tensor-Train Decomposition
25
Reduce # of parameters
Linear tensor network
Dimension-free decomposition
𝒲𝑖1…𝑖 𝑃
=
𝛼0…𝛼 𝑃
𝒜 𝛼0 𝑖1 𝛼1
… 𝒜 𝛼 𝑃−1 𝑖 𝑃 𝛼 𝑃

26. Tensor-Train RNNs and LSTMs
26
Seq2seq architecture
TT-LSTM cells

27. C l i m a t e d a t a s e tTr a ff i c d a t a s e t
Tensor LSTM for Long-term Forecasting

28. Approximation Guarantees
Theorem 1 [Yu et al 2017]: Let the state transition function 𝑓 ∈ ℋ𝜇
𝑘
be a
Hölder continuous, with bounded derivatives up to order k and finite
Fourier magnitude distribution 𝐶𝑓. Then a single layer Tensor Train RNN
can approximate 𝑓 with an estimation error of 𝜀 using with ℎ hidden units:
ℎ ≤
𝐶𝑓
2
𝜀
𝑑 − 1
𝑟 + 1 − 𝑘−1
𝑘 − 1
+
𝐶𝑓
2
𝜀
𝐶 𝑘 𝑝−𝑘
Where 𝑑 is the size of the state space, 𝑟 is the tensor-train rank and 𝑝 is
the degree of high-order polynomials i.e., the order of tensor.
• Easier to approximate if function is smooth
• Polynomial interactions are more efficient
Long-term Forecasting usingTensor-Train RNNs by R.Yu, S. Zheng, A,Y.Yue, 2017.

29. Visual Question & Answering
Tensors for multiple modalities

30. Visual Question & Answering
Tensors for multiple modalities

31. Visual Question & Answering
Tensor Sketching Algorithms

32. Tensors for Topic Modelingh t t p s

33. Tensors for Topic Detection

34. Tensors for Topic Detection
Government
Information
Technology
Politics
Topics

35. LDA Topic Model
brai
n
comput
data
evolve
gene
neuron
Justice
Education
Sports
Topics

36. Topic-word matrix [word = i|topic = j ]
Topic proportions P[topic = j|document]
Moment Tensor: Co-occurrence of Word Triplets
= + +
crim
e
Sports
Educa
on
Learning LDA Model

37. Topic-word matrix [word = i|topic = j ]
Topic proportions P[topic = j|document]
Moment Tensor: Co-occurrence of Word Triplets
= + +
crim
e
Sports
Educa
on
Learning LDA Model

38. Spectral Decomposition

39. Why Tensors?
Statistical reasons:
• Incorporate higher order relationships in data
• Discover hidden topics (not possible with matrix methods)
A. Anandkumar etal,Tensor Decompositions for Learning Latent Variable Models, JMLR 2014.

40. Why Tensors?
Statistical reasons:
• Incorporate higher order relationships in data
• Discover hidden topics (not possible with matrix methods)
Computational reasons:
• Tensor algebra is parallelizable like linear algebra.
• Faster than other algorithms for LDA
• Flexible: Training and inference decoupled
• Guaranteed in theory to converge to global optimum
A. Anandkumar etal,Tensor Decompositions for Learning Latent Variable Models, JMLR 2014.

41. Spectral LDA on AWS SageMaker

42. SageMaker LDA training is faster
0.00
20.00
40.00
60.00
80.00
100.00
5 10 15 20 25 30 50 75 100
Timeinminutes
Number of Topics
Training time for NYTimes
Spectral Time(minutes) Mallet Time (minutes)
0.00
50.00
100.00
150.00
200.00
250.00
5 10 15 20 25 50 100
Timeinminutes
Number of Topics
Training time for PubMed
Spectral Time (minutes) Mallet Time (minutes)
8 million documents
22x faster on average 12x faster on average
• Mallet is an open-source framework for topic modeling
• Mallet does training and inference together
• Benchmarks on AWS SageMaker Platform
300000 documents

43. End-to-end
Machine Learning
Platform
Zero setup Flexible model
training
Pay by the
second
Introducing Amazon SageMaker
The quickest and easiest way to get ML models from idea to production

44. XGBoost, FM,
and Linear for
classification and
regression
Kmeans and PCA
for clustering and
dimensionality
reduction
Image
classification with
convolutional
neural networks
LDA and NTM for
topic modeling,
seq2seq for
translation
Many optimized algorithms on SageMaker

45. AWS ML Stack
Frameworks &
Infrastructure
AWS Deep Learning AMI
GPU
(P3 Instances)
Mobile
CPU
(C5 Instances)
IoT
(Greengrass)
Vision:
Rekognition Image
Rekognition Video
Speech:
Polly
Transcribe
Language:
Lex Translate
Comprehend
Apache
MXNet
PyTorch
Cognitive
Toolkit
Keras
Caffe2
& Caffe
TensorFlow Gluon
Application
Services
Platform
Services
Amazon Machine
Learning
Mechanical
Turk
Spark &
EMR
Amazon
SageMaker
AWS
DeepLens

46. CONCLUSION

47. Conclusion
• Tensors are higher extensions of matrices.
• Encode data dimensions, modalities and higher order relationships.
• Tensor algebra is richer than matrix algebra. Richer neural network
architectures. More compact networks/better accuracy.
• Tensor operations are embarrassingly parallel. New primitives for tensor
operations.
= + ..

48. THANK YOU!