Brief survey on application of modern harmonic analysis to deep CNN(convolutional neural networks). Made by Sungbin Lim @ Kakao Brain

1. Harmonic Analysis
&
Deep Learning
Sungbin Lim

2. In this talk…
Mathematical theory about filter, activation,
pooling through multi-layers based on DCNN
Encompass general ingredients
Lipschitz continuity & Deformation sensitivity
WARNING : Very tough mathematics
…without non-Euclidean geometry (e.g. Geometric DL)

3. What is Harmonic Analysis?
f(x)=
X
n2N
an n(x), an := hf, niH
How to represent a function efficiently in the
sense of Hilbert space?
Number theory
Signal processing
Quantum mechanics
Neuroscience, Statistics, Finance, etc…
Includes PDE theory, Stochastic Analysis

4. What is Harmonic Analysis?
f(x)=
X
n2N
an n(x), an := hf, niH
How to represent a function efficiently in the
sense of Hilbert space?
Number theory
Signal processing
Quantum mechanics
Neuroscience, Statistics, Finance, etc…
Includes PDE theory, Stochastic Analysis

5. Hilbert space & Inner product
Banach space :
Hilbert space :
© Kyung-Min Rho

6. Hilbert space & Inner product
© Kyung-Min Rho
Banach space :
Normed space + Completeness
Hilbert space :

7. Banach space :
Normed space + Completeness
Hilbert space :
Banach space + Inner product
Hilbert space & Inner product
© Kyung-Min Rho

8. Banach space :
Normed space + Completeness
Hilbert space :
Banach space + Inner product
Rd
, L2, Wn
2 , · · ·
Hilbert space & Inner product
Cn
, Lp, Wn
p · · ·
© Kyung-Min Rho

9. Banach space :
Normed space + Completeness
Hilbert space :
Banach space + Inner product
Rd
, L2, Wn
2 , · · ·
hu, vi =
dX
k=1
ukvk
hf, giL2
=
Z
f(x)g(x)dx
hf, giW n
2
= hf, giL2 +
nX
k=1
h@k
xf, @k
xgiL2
Hilbert space & Inner product
Cn
, Lp, Wn
p · · ·
© Kyung-Min Rho

10. Why Harmonic Analysis?
Pn(x) = anxn
+ an 1xn 1
+ · · · + a1x + a0

11. Why Harmonic Analysis?
Pn(x) = anxn
+ an 1xn 1
+ · · · + a1x + a0
(an, an 1, . . . , a1 , a0)
Encoding

12. Why Harmonic Analysis?
Pn(x) = anxn
+ an 1xn 1
+ · · · + a1x + a0
(an, an 1, . . . , a1 , a0)
Encoding
Pn(x) = anxn
+ an 1xn 1
+ · · · + a1x + a0
Decoding

13. Why Harmonic Analysis?
Pn(x) = anxn
+ an 1xn 1
+ · · · + a1x + a0
(an, an 1, . . . , a1 , a0)
Encoding
Pn(x) = anxn
+ an 1xn 1
+ · · · + a1x + a0
Decoding
Why we prefer polynomial?

14. Stone-Weierstrass theorem
Polynomial is Universal approximation!
8f 2 C(X), 8" > 0,
9Pn s.t. max
x2X
|f(x) Pn(x)| < "
© Wikipedia

15. 8f 2 C(X),
9Pn s.t. lim
n!1
kf Pnk1 = 0
Stone-Weierstrass theorem
Polynomial is Universal approximation!
© Wikipedia

16. Stone-Weierstrass theorem
Even we can approximate derivatives!
9Pn s.t. lim
n!1
kf PnkCn ! 0
Polynomial is Universal approximation!
8f 2 Ck
(X),
© Wikipedia

17. Stone-Weierstrass theorem
Even we can approximate derivatives!
Universal approximation = {DL, polynomials, Tree,…}
Polynomial is Universal approximation!
9Pn s.t. lim
n!1
kf PnkCn ! 0
8f 2 Ck
(X),
© Wikipedia

18. Stone-Weierstrass theorem
Even we can approximate derivatives!
Universal approximation = {DL, polynomials, Tree,…}
But why we do not use polynomial?
Polynomial is Universal approximation!
9Pn s.t. lim
n!1
kf PnkCn ! 0
8f 2 Ck
(X),
© Wikipedia

19. Local interpolation works well for low dimension
© S. Mallat

20. Local interpolation works well for low dimension
Need " d
points to cover [0, 1]d
at a distance "
© S. Mallat

21. Local interpolation works well for low dimension
Need " d
points to cover [0, 1]d
at a distance "
High dimension ⇢ Curse of dimension!
© H. Bölcskei

22. Universal approximator
= Good feature extractor
?

23. Universal approximator
= Good feature extractor
…in HIGH dimension!

24. Nonlinear Feature Extraction
© S. Mallat, © H. Bölcskei

25. Dimension Reduction ⇢ Invariants
© S. Mallat

26. Dimension Reduction ⇢ Invariants
How?
© S. Mallat

27. Main Topic in Harmonic Analysis
Linear operator ⇢ Convolution + Multiplier
Invariance vs Discriminability
L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!)

28. Main Topic in Harmonic Analysis
L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!)
Linear operator ⇢ Convolution + Multiplier
Invariance vs Discriminability

29. Main Topic in Harmonic Analysis
L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!)
Linear operator ⇢ Convolution + Multiplier
Discriminability vs Invariance
Littlewood-Paley Condition ⇢ Semi-discrete Frame
AkfkH  kL[f]kH  BkfkH

30. Main Topic in Harmonic Analysis
L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!)
AkfkH  kL[f]kH  BkfkH
Linear operator ⇢ Convolution + Multiplier
Discriminability vs Invariance
Littlewood-Paley Condition ⇢ Semi-discrete Frame
kL[f1] L[f2]kH = kL[f1 f2]kH Akf1 f2kH
i.e. f1 6= f2 ) L[f1] 6= L[f2]

31. Main Topic in Harmonic Analysis
L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!)
AkfkH  kL[f]kH  BkfkH
Linear operator ⇢ Convolution + Multiplier
Discriminability vs Invariance
Littlewood-Paley Condition ⇢ Semi-discrete Frame
k L · · · L| {z }
n-fold
[f]kH  Bk L · · · L| {z }
(n-1)-fold
[f]kH  · · ·  Bn
kfkH

32. Main Topic in Harmonic Analysis
L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!)
AkfkH  kL[f]kH  BkfkH
Linear operator ⇢ Convolution + Multiplier
Discriminability vs Invariance
Littlewood-Paley Condition ⇢ Semi-discrete Frame
k L · · · L| {z }
n-fold
[f]kH  Bk L · · · L| {z }
(n-1)-fold
[f]kH  · · ·  Bn
kfkH
Banach fixed-point theorem

33. Main Tasks in Deep CNN
Representation learning
Feature Extraction
Nonlinear transform

34. Main Tasks in Deep CNN
Representation learning
Feature Extraction
Nonlinear transform

35. Main Tasks in Deep CNN
Representation learning
Feature Extraction
Nonlinear transform
Lipschitz continuity
ex) ReLU, tanh, sigmoid …
|f(x) f(y)|  Ckx yk () krf(x)k  C

36. How to control Lipschitz ?
k⇢(L[f])kH  N(B, C)kfkH
Theorem
No change in Invariance!

37. k⇢(L[f])kH  N(B, C)kfkH
Proof)
No change in Invariance!
Let ⇢ = ReLU, H = W1
2 . Then
Theorem
How to control Lipschitz ?

38. k⇢(L[f])kH  N(B, C)kfkH
Proof)
No change in Invariance!
Let ⇢ = ReLU, H = W1
2 . Then
Theorem
k⇢(L[f])kW 1
2
= k max{L[f], 0}kL2
+ kr⇢(L[f])kL2
 kL[f]kL2 + k ⇢0
(L[f])
| {z }
=1 or 0
r(L[f])kL2
 kL[f]kL2
+ kr(L[f])kL2
= kL[f]kW 1
2
 BkfkW 1
2
How to control Lipschitz ?

39. k⇢(L[f])kH  N(B, C)kfkH
Proof)
No change in Invariance!
Let ⇢ = ReLU, H = W1
2 . Then
Theorem
k⇢(L[f])kW 1
2
= k max{L[f], 0}kL2
+ kr⇢(L[f])kL2
 kL[f]kL2 + k ⇢0
(L[f])
| {z }
=1 or 0
r(L[f])kL2
 kL[f]kL2
+ kr(L[f])kL2
= kL[f]kW 1
2
 BkfkW 1
2
How to control Lipschitz ?

40. k⇢(L[f])kH  N(B, C)kfkH
Proof)
No change in Invariance!
Let ⇢ = ReLU, H = W1
2 . Then
Theorem
k⇢(L[f])kW 1
2
= k max{L[f], 0}kL2
+ kr⇢(L[f])kL2
 kL[f]kL2 + k ⇢0
(L[f])
| {z }
=1 or 0
r(L[f])kL2
 kL[f]kL2
+ kr(L[f])kL2
= kL[f]kW 1
2
 BkfkW 1
2
How to control Lipschitz ?

41. k⇢(L[f])kH  N(B, C)kfkH
Proof)
No change in Invariance!
Let ⇢ = ReLU, H = W1
2 . Then
Theorem
k⇢(L[f])kW 1
2
= k max{L[f], 0}kL2
+ kr⇢(L[f])kL2
 kL[f]kL2 + k ⇢0
(L[f])
| {z }
=1 or 0
r(L[f])kL2
 kL[f]kL2
+ kr(L[f])kL2
= kL[f]kW 1
2
 BkfkW 1
2
How to control Lipschitz ?

42. k⇢(L[f])kH  N(B, C)kfkH
Proof)
No change in Invariance!
Let ⇢ = ReLU, H = W1
2 . Then
Theorem
k⇢(L[f])kW 1
2
= k max{L[f], 0}kL2
+ kr⇢(L[f])kL2
 kL[f]kL2 + k ⇢0
(L[f])
| {z }
=1 or 0
r(L[f])kL2
 kL[f]kL2
+ kr(L[f])kL2
= kL[f]kW 1
2
 BkfkW 1
2
How to control Lipschitz ?

43. k⇢(L[f])kH  N(B, C)kfkH
Proof)
No change in Invariance!
Let ⇢ = ReLU, H = W1
2 . Then
Theorem
k⇢(L[f])kW 1
2
= k max{L[f], 0}kL2
+ kr⇢(L[f])kL2
 kL[f]kL2 + k ⇢0
(L[f])
| {z }
=1 or 0
r(L[f])kL2
 kL[f]kL2
+ kr(L[f])kL2
= kL[f]kW 1
2
 BkfkW 1
2
How to control Lipschitz ?
What about Discriminability?

44. Scale Invariant Feature
Translation Invariant
Stable at Deformation
© S. Mallat

45. Scale Invariant Feature
Translation Invariant
Stable at Deformation

46. Scattering Network (Mallat, 2012)
(f) =
[
n
(
· · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p)
| {z }
n-fold convolution
⇤ n
)
(j),··· , (p)
© H. Bölcskei

47. Generalized Scattering Network (Wiatowski, 2015)
(f) =
[
n
(
· · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p)
| {z }
n-fold convolution
⇤ n
)
(j),··· , (p)
Gabor frame
Tensor wavelet Directional wavelet
Ridgelet frame Curvelet frame
© H. Bölcskei

48. Generalized Scattering Network (Wiatowski, 2015)
(f) =
[
n
(
· · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p)
| {z }
n-fold convolution
⇤ n
)
(j),··· , (p)
© S. Mallat

49. Generalized Scattering Network (Wiatowski, 2015)
(f) =
[
n
(
· · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p)
| {z }
n-fold convolution
⇤ n
)
(j),··· , (p)
Linearize symmetries
© S. Mallat

50. Generalized Scattering Network (Wiatowski, 2015)
(f) =
[
n
(
· · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p)
| {z }
n-fold convolution
⇤ n
)
(j),··· , (p)
Linearize symmetries
“Space folding”, Cho (2014)
© S. Mallat

51. (f) =
[
n
(
· · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p)
| {z }
n-fold convolution
⇤ n
)
(j),··· , (p)
f 7! Sd/2
n Pn(f)(Sn·)
|k n(Ttf) n(f)|k = O
ktk
Qn
j=1 Sj
!
Theorem
Generalized Scattering Network (Wiatowski, 2015)

52. f 7! Sd/2
n Pn(f)(Sn·)
|k n(Ttf) n(f)|k = O
ktk
Qn
j=1 Sj
!
Theorem
Features become more translation invariant
with increasing network depth
Generalized Scattering Network (Wiatowski, 2015)

53. Generalized Scattering Network (Wiatowski, 2015)
© Philip Scott Johnson
(f) =
[
n
(
· · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p)
| {z }
n-fold convolution
⇤ n
)
(j),··· , (p)
Theorem
F⌧,! = e2⇡i!(x)
f(x ⌧(x))
|k (F⌧,![f]) (f)k|  C(k⌧k1 + k!k1)kfkL2

54. Generalized Scattering Network (Wiatowski, 2015)
© Philip Scott Johnson
Theorem
F⌧,! = e2⇡i!(x)
f(x ⌧(x))
|k (F⌧,![f]) (f)k|  C(k⌧k1 + k!k1)kfkL2
Multi-layer convolution linearize Features
i.e. stable to deformations

55. Generalized Scattering Network (Wiatowski, 2015)
© Philip Scott Johnson

56. Ergodic Reconstructions
© Philip Scott Johnson
© S. Mallat

57. David Hilbert
Wir müssen wissen.
Wir werden wissen.

58. Q.A