Because of deep learning we now talk a lot about tensors, yet tensors remain relatively unknown objects. In this presentation I will introduce tensors and the basics of multilinear algebra, then describe tensor decompositions and give some examples of how they are used in representation learning for understanding/compressing data. I will also briefly describe how tensor decompositions are used in 1) the method of moments for training latent variable models, and 2) deep learning for understanding why deep convolutional networks are such excellent classifiers.

1. a brief survey of tensors
multilinear algebra, decompositions, method of moments
and deep learning
Berton Earnshaw
Senior Scientist, Savvysherpa
bertonearnshaw@savvysherpa.com

2. tensors, tensors everywhere…

3. tensor = multidimensional array
vector matrix tensor
v ∊ ℝ64 X ∊ ℝ8x8 𝓧 ∊ ℝ4x4x4

4. third-order tensors
𝓧 ∊ ℝ7x5x8

5. color image is 3rd-order tensor

6. color video is 4th-order tensor

7. MNIST is third-order tensor

8. facial images database is 6th-order tensor

9. tensors are just bookkeeping devices!
easy right?

10. multilinear algebra
wrong!

11. multilinear algebra basics
mode-n product
𝓧 ∊ ℝ3x4x2
U ∊ ℝ2x3
𝓨 = 𝓧 x1 U
∊ ℝ2x4x2

12. multilinear algebra basics
mode-n matricization (unfolding)
𝓧 ∊ ℝ3x4x2
mode-1
mode-2
mode-3

13. multilinear algebra basics
Kronecker product
A ∊ ℝI x J, B ∊ ℝK x L
A ⨂ B ∊ ℝIK x JL
mode-n matricization of multilinear product

14. tensor decompositions

15. decomposition = compression
X ∊ ℝI x J, A ∊ ℝI x K, B ∊ ℝJ x K (K ≪ I,J)
X ≈ ABT
(I+J)K
IJ
matrix # entries
X
ABT

16. compression = understanding
X ∊ ℝI x J, A ∊ ℝI x K, B ∊ ℝJ x K (K ≪ I,J)
X ≈ ABT
columns of B = basis of low-dim. subspace
rows of A = low-dim. representation of X
A,B encode X using fewer bits!

17. X = abT = a○b
=
rank-1 matrices

18. X ≈ ABT
ABT = ∑K
k=1 akbk
T = ∑K
k=1 ak○bk
decomposition = sum rank-1 matrices

19. 𝓧 = a○b○c
rank-1 tensors

20. 𝓧 ≈ [[A,B,C]]
[[A,B,C]] = ∑K
k=1 ak○bk○ck
When K is minimal, called the Canonical Polyadic Decomposition
decomposition = sum rank-1 tensors

21. MNIST decomposition

22. MNIST as matrix

23. MNIST matrix decomposition
X ≈ ABT = ∑K
k=1 ak○bk (K = 10)

24. MNIST matrix reconstruction

25. original 70,000 * (28 * 28) = 54,880,000
matrix (70,000 + (28 * 28)) * 10 = 707,840 1.29%
matrix decomposition requires ~1% original memory!
representation size

26. MNIST as tensor

27. MNIST tensor decomposition
𝓧 ≈ [[A,B,C]] = ∑K
k=1 ak○bk○ck (K = 10)

28. MNIST tensor reconstruction

29. original 70,000 * (28 * 28) = 54,880,000
matrix (70,000 + (28 * 28)) * 10 = 707,840 1.29%
tensor (70,000 + 28 + 28) * 10 = 700,560 1.28%
tensor decomposition requires less memory than matrix
representation size

30. 𝓧 = [[A,B,C]] = ∑K
k=1 ak○bk○ck
Kruskal’s rank theorem: CPD is unique (up to unavoidable scaling
and permutation of columns of A, B and C) if
rA + rB + rC ≥ 2K + 2
where rM is the Kruskal rank of the matrix M (the largest number r
such that any choice of r columns of M is linearly independent).
Note: rank(M) ≥ rM
uniqueness of decomposition

31. learning via method of moments

32. learning via method of moments

33. learning via method of moments

34. 𝓧 = [[𝓖;A,B,C]] = 𝓖 x1 A x2 B x3 C
Tucker decomposition

35. classification via convolutional networks
Task: learn Y score functions hy such that the prediction rule
is as accurate as possible.
coefficient tensor
tensor decomposition

36. hierarchical Tucker decomposition

37. universality and depth efficiency

38. references
https://github.com/bearnshaw/mnist-factorization
• Kolda, Tamara G., and Brett W. Bader. "Tensor decompositions and applications."
SIAM review 51.3 (2009): 455-500.
• Cichocki, Andrzej, et al. "Tensor decompositions for signal processing applications:
From two-way to multiway component analysis." IEEE Signal Processing Magazine
32.2 (2015): 145-163.
• Anandkumar, Animashree, et al. "Tensor decompositions for learning latent variable
models." Journal of Machine Learning Research 15.1 (2014): 2773-2832.
• Sedghi, Hanie, and Anima Anandkumar. "Training Input-Output Recurrent Neural
Networks through Spectral Methods." arXiv preprint arXiv:1603.00954 (2016).
• Cohen, Nadav, Or Sharir, and Amnon Shashua. "On the expressive power of deep
learning: A tensor analysis." arXiv preprint arXiv:1509.05009 554 (2015).
• Cohen, Nadav, and Amnon Shashua. "Convolutional rectifier networks as generalized
tensor decompositions." International Conference on Machine Learning (ICML).
2016.