Introduction to Neural Art

1. Everyone can be van Gogh
Introduction to Neural Art
物理三 王昱翔
2016. 11. 08

2. Reference
• Leon A. Gatys, Alexander S. Ecker, Matthias Bethge, A Neural
Algorithm of Artistic Style
• Mark Chang’s slide, http://www.slideshare.net/ckmarkohchang/a-
neural-algorithm-of-artistic-style?next_slideshow=1

3. Outline
• Few examples of Neural Art
• Introduction to CNN
• Implement Neural Art
• Application
• Conclusion

4. Paintings of van Gogh

5. Can we paint like van Gogh? (I’m not 9.2%)

6. Yes! (Use Convolutional Neural Network)
Ref: Mark Chang’s slide

7. Neural Network

8. Partial CNN Process
Image
(Input)
Convolution Max Pooling

9. CNN– Convolution
0 0 1 1 0 0
0 1 1 1 1 0
1 1 1 1 1 1
1 1 1 1 1 1
0 0 0.5 0.5 0 0
0 0 0.5 0.5 0 0
0 0 1
0 0 1
0 0 1
Filter_2
Number of filters depends on Users !
0 0 1
0 0.5 0
1 0 0
Filter_1
Find pattern
Input image

10. Filter -- Element wise matrix multiplication
0 0 1 1 0 0
0 1 1 1 1 0
1 1 1 1 1 1
1 1 1 1 1 1
0 0 0.5 0.5 0 0
0 0 0.5 0.5 0 0
0 0 1
0 0.5 0
1 0 0
0 0 1
0 0 1
0 0 1
2.5

11. CNN -- convolution 0
1
2
3
4
5
6
w0 = 𝐹11
0 1 2 3 4 5
6 7 8 9 10 11
0 0 1 1 0 0
0 1 1 1 1 0
Corresponding Index
2.5
w1 = 𝐹12
w2 = 𝐹13
w6 = 𝐹21

12. 0 0 1 1 0 0
0 1 1 1 1 0
1 1 1 1 1 1
1 1 1 1 1 1
0 0 0.5 0.5 0 0
0 0 0.5 0.5 0 0
0 0 1
0 0.5 0
1 0 0
0 0 1
0 0 1
0 0 1
2.5

13. CNN – Max Pooling
2.5 2.5 1.5 1.5
2.5 2.5 2.5 1.5
1.5 1.5 2 2
1 1.5 1.75 1.5
2.5 2.5
1.5 2
0 0 1 1 0 0
0 1 1 1 1 0
1 1 1 1 1 1
1 1 1 1 1 1
0 0 0.5 0.5 0 0
0 0 0.5 0.5 0 0
0 0 1
0 0.5 0
1 0 0
Max pooling
2.5

14. Repeat Convolution + Max pooling
Input Image
Convolution
Max pooling
Convolution
Max pooling

15. Whole CNN Process
Image Convolution
Max
Pooling
N times
Flatten
Fully
Connected
Neural
Network

16. Neural Art
Content
Style

17. Neural Art- Create Content
0.3 0.7 1 0 0.6
0.2 0.8 0.7 0.4 0.4
0.7 0.1 0.2 0.3 0.5
0.8 0 0.3 0.9 0.1
0.5 0.9 0.6 1 0.8
CNN
CNN
Canvas x
Output X
Output P
𝐿 𝑐𝑜𝑛𝑡𝑒𝑛𝑡 =
1
2 𝑖,𝑗(𝑋𝑖,𝑗 − 𝑃𝑖,𝑗)2
Gradient Descent (Back Propagation)
Photo p
Canvas x: Our paintings
CNN: Already trained by
other people

18. CNN
Neural Art – Create Style
0.3 0.7 1 0 0.6
0.2 0.8 0.7 0.4 0.4
0.7 0.1 0.2 0.3 0.5
0.8 0 0.3 0.9 0.1
0.5 0.9 0.6 1 0.8
CNN
Canvas x
Output
Output 𝐿 𝑠𝑡𝑦𝑙𝑒 =
1
2 𝑖,𝑗(𝑋𝑖,𝑗 − 𝑆𝑖,𝑗)2
Gradient Descent (Back Propagation)
Style s
Gram S
Gram X
G
G
∗ 𝐺𝑟𝑎𝑚𝑖,𝑗 = <𝑣𝑖, 𝑣𝑗>
Gram S_00
= <O_s row[0], O_s col[0]>

19. Update Canvas
• 𝐿 𝑡𝑜𝑡𝑎𝑙 = 𝛼𝐿 𝑐𝑜𝑛𝑡𝑒𝑛𝑡 + 𝛽𝐿 𝑠𝑡𝑦𝑙𝑒  x = x - 𝜂
𝜕𝐿 𝑡𝑜𝑡𝑎𝑙
𝜕𝑥

20. Application -- DeepDream
Ref: Deep Dream

21. Application -- Prisma
Ref: Prisma

22. Real time Neural Art
Ref: Fast Style transfer in Tensorflow

23. Conclusion
• Yeah! Computer can paint by themselves!
• Really? They still need a photo for painting.

24. Thank You For Your Listening !