End-to-end music classification model의 짧은 역사와 그들의 작동 방식을 이해하기 위한 노력들을 살펴봅니다. Techtalk @ Naver Green Factory - 2018. 05. 09.

1. End-to-end
Music Classification

2. • MIR
‣ 2017 DJ music
classification
•
•
‣ ...
‣ ...
• ^_^
‣
- ML for Music
- Automated feature engineer using RL
ICASSP 2018 😆

3. Music Classification, Why?
• classification “representation learning”
‣ Music streaming service
?
‣ content-based recommendation
• Music streaming service
‣ !
•
‣ “ ”
‣

4. :
End-to-end Music Classification
• History of E2E Music Classification Models
‣ E2E ?
• Interpretability of E2E Music Classification Models
‣

5. • Sample-level Deep Convolutional Neural Networks
for Music Auto-tagging using Raw Waveforms (2017)
Jongpil Lee, Jiyoung Park, Keunhyoung Luke Kim and Juhan Nam
Sound and Music Computing Conf. (SMC), 2017.
‣ Music classification end-to-end approach
‣ Frequency
• Sample-level CNN Architectures
for Music Auto-tagging using Raw Waveforms (2018)
Taejun Kim, Jongpil Lee and Juhan Nam,
IEEE Int. Conf. Acoustical, Speech Signal Processing (ICASSP), 2018.
‣ CNN architecture
‣ Loudness

7. Spectrogram to End-to-end
1 2 3
2014 2017 (SampleCNN)
• End-to-end
• STFT conv. layer
• (frame-level)
• End-to-end
• STFT conv. layer
• (sample-level)
• Handcrafted spectrogram
• 1D or 2D conv.

8. Frame- vs. Sample-level
• ( )
• Sample-level
‣ 1D convolution
‣ 7 ( 0.14ms )*
• Frame-level
‣ STFT window 1D convolution
‣ 256 ( 12ms )*
‣ Spectrogram frame-level
→ Trade-off between time- & frequency resolution!
• *22,050kHz

9. Frame-level Mel-spectrogram Model
(=Channel)
1
1) 2D Convolution 2) 1D Convolution
(=Channel)
(=Channel)
• Spectrogram
• (e.g. MNIST)
•
• Spectrogram 1 sequence
• Frequency dim. = dim.
• End-to-end

10. Spectrogram Model
•
‣ ( 3~7 )
‣
‣ Phase invariant
•
‣ Task mid-level representation
‣ Hyperparameter tuning (e.g. window length, hop size, etc.)
‣ Phase
‣ Time- & frequency-resolution trade-off

11. Audio time & frequency resolution

12. Convolutional Filters Decouple
Time & Frequency Resolution
• Conv. filter time & frequency resolution trade-off
• Convolution resolution:
‣ Time resolution stride
- Stride↓ time resolution↑
‣ Frequency resolution filter
- #filters↑ frequency resolution↑
‣ Stride time & frequency resolution

13. Frame-level Raw Waveform Model
• 2014 E2E music classification
‣ But, spectrogram model
- , E2E music classification
• STFT 1D strided conv. layer
‣ Spectrogram 1D conv. mid-level
representation
• Strided conv. output spectrogram
hyperparameter
‣ Filter size (=window size of STFT)
‣ Stride (=hop size of STFT)
2
Spectrogram
Waveform
with one 1D conv.
net.

14. 1D Conv. on
Spectrogram
Time
Channel
(=Frequency)
Channel
(=unsortedfrequency-like)
Time
1D conv. filter
STFT
1D conv. filter
1D conv. filter
1D Conv. on
Waveform
vs.
• 1D conv. 2D conv.
spectrogram vs.
waveform
• E2E music classification
Channel
frequency
→
!

15. Frame-level Raw Waveform Model
• :
‣ CNN (layer )
‣ Log-based amplitude compression
‣ conv. layer phase variation
• 2014 , :
‣ Batch Norm. ResNet ( )
‣ GPU ( ,
)
2

16. Sample-level Raw Waveform Model
•
‣ Log-scale amplitude compression
‣ Phase invariance
• :
‣
- Filter size = one of {2, 3, 4, 5, 7}
‣ STFT conv. layer
• net.
3
6551 × 128
19683 × 128
2187× 128
729 × 256
243 × 256
81 × 256
27 × 256
9 × 256
3 × 256
512
1 × 512
50
tag prediction
1Dconvolutionalblock×9
strided conv
FC×2
59049 × 1
raw waveform
1D conv. block
SampleCNN!

17. Sample-level Raw Waveform Model3
•E2E model spectrogram
•E2E music classification model
Comparison with frame-level models
Comparison with state-of-the-arts

18. Sample-level Raw Waveform Model3
...
Filter size 2
Filter size 3
Filter size 4
Filter size 5

19. 6551 × 128
19683 × 128
2187× 128
729 × 256
243 × 256
81 × 256
27 × 256
9 × 256
3 × 256
256 512 256
1 × 512
50
tag prediction
1Dconvolutionalblock×9
multi-levelfeatureaggregation
strided conv
FC×2
59049 × 1
raw waveform
globalmaxpooling
globalmaxpooling
Base Architecture
Advanced Sample-level Raw Waveform Model4
image classification
1) Convolutional blocks from ResNet & SENet
2) Multi-level feature aggregation
1D Convolutional Blocks
Conv1D
BatchNorm
MaxPool
relu
Basic block
relu
relu
Conv1D
BatchNorm
Conv1D
BatchNorm
MaxPool
Dropout
Res-n block
relu
relu
sigmoid
relu
T×C
1×C
1×αC
1×C
T×C
T×C
Conv1D
BatchNorm
Dropout
Conv1D
BatchNorm
GlobalAvgPool
FC
FC
Scale
MaxPool
ReSE-n block
relu
relu
sigmoid
T×C
1×C
1×αC
1×C
T×C
T×C
Conv1D
FC
FC
Scale
BatchNorm
MaxPool
GlobalAvgPool
SE block
Excitation
SampleCNN

20. Res-n Block
• From ResNet (2015 ImageNet challenges )
• Motivation:
‣ Skip-connection net.
• n: conv. layer (1 or 2)
‣ Conv. layer regularization
dropout (inspired by WideResNet)
relu
relu
Conv1D
BatchNorm
Conv1D
BatchNorm
MaxPool
Dropout
Skip-connection
Basic
Res-1
Res-2 0.9061
0.9048
0.9055
AUC on MagnaTagATune
1.8 ,
but

21. Basic
Res-2
SE 0.9083
0.9061
0.9055
SE Block
• From SENet (2017 ImageNet challenges )
• Motivation:
‣ Channel , channel recalibration
- channel (=frequency-like) ( ) ,
( )
- channel weight (0~1) rescale
(recalibration)
relu
relu
sigmoid
T×C
1×C
1×αC
1×C
T×C
T×C
Conv1D
FC
FC
Scale
BatchNorm
MaxPool
GlobalAvgPool
AUC on MagnaTagATune
basic block
1.08

22. SE Block for Image (2D Conv.)
Squeeze operation:
• Aggregate spatial dimensions
• Produce channel-wise statistics
Excitation operation:
• Using the statistics, learn channel relationships
• Produce weight for each channel
Global spatial information
for each channel
Excitations (range 0~1):
Weight for each channel
Reweight each channel
using the weights

23. SE Block for Audio (1D Conv.)
Time
Channel(orFrequency-like)
Time
Global temporal statistics
for each channel
Squeeze operation:
• Aggregate temporal dimensions
• Produce frequency-wise statistics
Excitation operation:
• Using the statistics, learn frequency relationships
• Produce weight for each frequency
Excitations (range 0~1):
Weight for each frequency
Reweight each frequency
using the weights

24. SE Block for Audio (1D Conv.)
relu
relu
sigmoid
T×C
1×C
1×αC
1×C
T×C
T×C
Conv1D
FC
FC
Scale
BatchNorm
MaxPool
GlobalAvgPool

25. Difference with Original SE Block
• Original SENet FC layer
‣ 𝑟: reduction ratio
•
‣ 𝜶: amplifying ratio
• Original SENet 16 ,
16
‣ Audio channel
?
relu
relu
sigmoid
T×C
1×C
1×αC
1×C
T×C
T×C
Conv1D
FC
FC
Scale
BatchNorm
MaxPool
GlobalAvgPool

26. Amplifying Ratio (alpha) Grid Search
AUC
Amplifying Ratio
OverfittingUnderfitting

27. ReSE-n block
• Res-n & SE blockrelu
relu
sigmoid
relu
T×C
1×C
1×αC
1×C
T×C
T×C
Conv1D
BatchNorm
Dropout
Conv1D
BatchNorm
GlobalAvgPool
FC
FC
Scale
MaxPool
Basic
Res-2
SE
ReSE-1
ReSE-2 0.9102
0.9066
0.9083
0.9061
0.9055
1.8
AUC on MagnaTagATune

28. Multi-level Feature Aggregation
• layer 3 output
‣ 3 output concatenate
‣ Simple, but powerful
• Motivation:
‣ music tag abstraction
‣ Example:
- “vocal”: low abstraction
- “metal”: high abstraction
• Global max pooling time dimension
1

29. Comparison of Architectures
Basic
SE
Res-1
Res-2
ReSE-1
ReSE-2 0.9102
0.9066
0.9061
0.9048
0.9083
0.9055
0.9113
0.9053
0.9098
0.9037
0.9111
0.9077
multi-feature aggregation no multi-feature aggregation
x1.7
x1.08SampleCNN

30. Comparison with SoTA
Ensemble of 3 models
Best among single models!

32. Interpretability of Deep Learning
•
• interpretability
‣ vision
‣ Audio
• Weapons of Math Destruction ( )
— , “ ”
• ,
‣ ( )
‣ “ ?” “ ?” ...

33. SampleCNN Filter Visualization
• channel frequency signal
• frequency ( )
• Layer filter
(e.g. mel-scale)
‣ i.e. piano key
• Layer Low-frequency
Sorted channel index
Frequency(0~11KHz)

34. Channel back propagate
Input
Filter Viz. Process Example:
layer channel
6551 × 128
19683 × 128
2187× 128
729 × 256
243 × 256
81 × 256
utionalblock×9
strided conv
59049 × 1
raw waveformInitialize input randomly (random noise)
Backprop.
1
2
STFT
3
4
Frequency
Channel 3 of Layer 1
Time
Channel
layer

35. Excitation
Visualization
relu
relu
sigmoid
T×C
1×C
1×αC
1×C
T×C
T×C
Conv1D
FC
FC
Scale
BatchNorm
MaxPool
GlobalAvgPool
Excitation
Sorted channel index
Excitation

36. Excitation
Visualization
• SE block channel
tag
• Mid block general , last
block discriminative signal
processing
• block tag
excitation!
‣ Loudness
Sorted channel index
Excitation

37. Excitation
Visualization
• Excitation loudness
tag
excitation layer
Sorted channel index
Excitation
Standard deviations of excitations
across tags

38. Analysis of the First Excitation
Sorted channel index
Excitation
tag 50 excitation

39. Analysis of the First Excitation
• audio segment
• linear regression line
• SE block loudness
normalize
• But ,
Average
of
128 Channels
Most
Positive
Channel
Most
Negative
Channel
Most
Neutral
Channel
Least
Regression
Error
Channel
Loudness
Excitation

40. Analysis of the First Excitation
• linear regression
• loudness normalize
‣ #negative = 109
‣ #positive = 19
• Loudness excitation
?
Loudness
Excitation

41. Variation of Excitations increases
according to Loudness
• audio segment excitation channel
• segment ( )
• Loudness excitation

42. Excitation Comparison
with Speech Dataset

43. TensorFlow Speech Commands Dataset
• 1 audio
•
‣ : “yes”, “no”, “right”, “go”
• SE block

44. Average of 128 Channels
MagnaTagATune
(Music dataset)
TensorFlow Speech Commands
(Speech dataset)

45. Most Positive Channel
MagnaTagATune
(Music dataset)
TensorFlow Speech Commands
(Speech dataset)

46. Linear Regression Lines
MagnaTagATune
(Music dataset)
TensorFlow Speech Commands
(Speech dataset)
#negative=109, #positive=19 !

47. MagnaTagATune
(Music dataset)
TensorFlow Speech Commands
(Speech dataset)
Variation of Excitations increases according to Loudness

48. • audio filter visualization ?
• SE block squeeze, excitation ?
• ROC-AUC policy gradient directly optimize ?
Image filter viz.

49. (Taejun Kim)
i2r.jun@gmail.com
^_^

50. References
• [Cover art] http://www.sqoop.co.ug/201805/four-one-one/nation-media-
group-launches-music-record-label-lit-music.html