안녕하세요 딥러닝 논문읽기 모임 입니다! 오늘 소개 드릴 논문은 Once-for-All: Train One Network and Specialize it for Efficient Deployment 라는 제목의 논문입니다. 모델을 실제로 하드웨어에 Deploy하는 그 상황을 보고 있는데 이 페이퍼에서 꼽고 있는 가장 큰 문제는 실제로 트레인한 모델을 Deploy할 하드웨어 환경이 너무나도 많다는 문제가 하나 있습니다 모든 디바이스가 갖고 있는 리소스가 다르기 때문에 모든 하드웨어에 맞는 모델을 찾기가 사실상 불가능하다는 문제를 꼽고 있고요 각 하드웨어에 맞는 옵티멀한 네트워크 아키텍처가 모두 다른 상황에서 어떻게 해야 될건지에 대한 고민이 일반적 입니다. 이제 할 수 있는 접근중에 하나는 각 하드웨어에 맞게 옵티멀한 아키텍처를 모두 다 찾는 건데 그게 사실상 너무나 많은 계산량을 요구하기 때문에 불가능하다라는 문제를 갖고 있습니다 삼성 노트 10을 예로 한 어플리케이션의 requirement가 20m/s로 그 모델을 돌려야 된다는 요구사항이 있으면은 그 20m/s 안에 돌 수 있는 모델이 뭔지 accuracy가 뭔지 이걸 찾기 위해서는 파란색 점들을 모두 찾아야 되고 각 점이 이제 트레이닝 한번을 의미하게 됩니다 그래서 사실상 다 수의 트레이닝을 다 해야지만 그 중에 뭐가 최적인지 또 찾아야 합니다. 실제 Deploy해야 되는 시나리오가 늘어나면 이게 리니어하게 증가하기 때문에 각 하드웨어에 맞는 그런 옵티멀 네트워크를 찾는게 사실상 불가능합니다. 그래서 이제 OFA에서 제안하는 어프로치는 하나의 네트워크를 한번 트레이닝 하고 나면 다시 하드웨어에 맞게 트레이닝할 필요 없이 그냥 각 환경에 맞게 가져다 쓸 수 있는 서브네트워크를 쓰면 된다 이게 주로 메인으로 사용하고 있는 어프로치입니다. 오늘 논문 리뷰를 위해 펀디멘탈팀 김동현님이 자세한 리뷰를 도와주셨습니다 많은 관심 미리 감사드립니다!

1. Once-for-All: Train One Network and
Specialize it for Efficient Deployment
[ICLR 2020]
2022. 03. 20. (Sun)
Presented by: 김동현
w/ Fundamental Team: 김채현, 박종익, 양현모, 이근배, 이재윤, 송헌
1

2. Contents
● Problem and Approach
● Key Challenge
● How to Train Once-for-all Network
● How to Deploy Once-for-all Network
● Evaluations
● Discussions
● Conclusion
2

3. Contents
● Problem and Approach
● Key Challenge
● How to Train Once-for-all Network
● How to Deploy Once-for-all Network
● Evaluations
● Discussions
● Conclusion
3

4. Main Problem to Solve
● There are various hardware platforms to deploy DNN models.
○ Survey says there are 23.14 billion IoT devices until 2018.
○ The devices have different resource constraints;
It is impossible to deploy the same model to all devices.
● The optimal neural network architecture varies by deployment environments
(e.g., #arithmetic units, application requirements).
4

5. Main Problem to Solve
● It is computationally prohibitive to find all the optimal architecture by training
on each environment.
● Then, how is it possible to cost-efficiently find the specialized model on
each platform?
5
target latency
= 20ms

6. Suggested Approach
● Train a Once-for-all(OFA) network, which enables serving on various
environment without additional training.
○ Various scales of sub-networks (about 1019
) are available from one OFA network.
○ Each hardware can find the specialized model for its requirements (e.g, latency).
6

7. Key Challenges for Once-for-All Network
Requirements
1. The sub-network architecture should be part of the largest network.
2. Sub-networks should share parameters with larger networks.
3. Optimal model architecture for specified hardwares should be easily found.
7

8. Key Challenges for Once-for-All Network
Requirements
1. The sub-network architecture should be part of the largest network.
2. Sub-networks should share parameters with larger networks.
3. Optimal model architecture for specified hardwares should be easily found.
Challenges
1. How to design sub-network architecture space based on a OFA network.
2. How to let sub-networks share parameters with larger networks.
3. How to select the optimal model for the hardware (in terms of latency,
accuracy).
8

9. Contents
● Problem and Approach
● Key Challenge
● How to Train Once-for-all Network : Challenges #1, #2
● How to Deploy Once-for-all Network: Challenges #3
● Evaluations
● Discussions
● Conclusion
9

10. Q&A
10

11. ● Assumption: Follow the common practice of CNN models (e.g., ResNet).
○ A model consists of groups of Layers (i.e., units).
● Architecture Search Space
○ # Layers(L): the depth of each unit is chosen from {2, 3, 4}
○ # Channels(C): expansion ratio in each layer is chosen from {3, 4, 6}
○ Kernel Size(Ks): {3, 5, 7}
○ Input Dimension: ranges from 128 to 224 with a stride
● Num available sub-networks: ((3 * 3)2
+ (3 * 3)3
+ (3 * 3)4
)5
= about 1019
Training OFA Network - Network Architecture
… … …
…
L1 L2 L3
C
…
Ks
# units
11

12. How sub-networks share parameters:
● Elastic Kernel Size
○ Merely sharing the parameters of larger kernel can affect the performance.
○ When changing kernel size, pass through Transform Matrix:
■ For each layer, hold parameters for elastic kernels.
● # 25*25 parameters for 7x7 -> 5x5.
● # 9*9 parameters for 5x5 -> 3x3.
● E.g., 5x5 kernel = (Center of 7x7) * Transform Matrix
Training OFA Network - Sharing Parameters
12

13. How sub-networks share parameters:
● Elastic Depth (= #Layers)
○ The first D layers are shared when L layers exist in a unit.
○ Simpler depth settings compared to selecting random layers from L layers.
Training OFA Network - Sharing Parameters
L D
13

14. How sub-networks share parameters:
● Elastic Width (= #Channels)
○ For the given expansion ratio, select channels through a channel sorting method:
1. Calculate L1 Norm for each channel’s weights.
2. Sort the channels by the L1 Norm order.
3. Choose the top-K channels.
Training OFA Network - Sharing Parameters
L1 Norm
14

15. Progressive Shrinking
1. Train a full model (i.e. max vaule for each configuration).
● With the trained full-size model, Knowledge-Distillation techniques are leveraged.
● Note: Full model != Best model
Training OFA Network - Training Process
… … …
…
L1 L2 L3
Note1: Input image size is randomly chosen for each training batch
15

16. Progressive Shrinking
1. Train a full model (i.e. max vaule for each configuration).
2. Sample sub-networks varying kernel sizes and fine-tune.
a. For each step, sample one sub-net with different kernel sizes.
b. Calculate Loss. Loss = Full model loss * KD_raio + sub-net loss
c. Update the weights (updating sub-net’s weight -> updating the full model’s weight)
Training OFA Network - Training Process
… … …
L1 L2 L3 16
Note1: Input image size is randomly chosen for each training batch

17. Progressive Shrinking
1. Train a full model (i.e. max vaule for each configuration).
2. Sample sub-networks varying kernel sizes and fine-tune.
3. Sample sub-networks varying depth and fine-tune.
4. Sample sub-networks varying channel expansion ratio and fine-tune.
Training OFA Network - Training Process
… … …
L1 L2 L3
Note2: Refer to Appendix B for impl. details of progressive shrinking
Note1: Input image size is randomly chosen for each training batch
17

18. Deploying Specialized Model w/ OFA Network
Problem:
● derive the specialized sub-network for a given deployment scenario (e.g.,
latency constraints).
Solution:
● Train an accuracy predictor (3-layer FFNN)
○ f(architecture, input image size) => accuracy
○ randomly sample 16K sub-networks, measure the accuracy on 10K validation images
● Latency Lookup Table (Details in the ProxylessNAS paper)
○ On each hardware platform, build a latency lookup table .
● Conduct an evolutionary search leveraging the above information.
○ Mutate from the known sub-network by sampling and predicting the performance.
○ add the mutated sub-network to the child pool if it satisfies the constraint (latency).
18

19. Q&A
19

20. Evaluation
● ImageNet Dataset
● Eval on Various Hardware Platforms:
○ Samsung S7 Edge, Note8, Note10, Google Pixel1, Pixel2, LG G8, NVIDIA 1080Ti, V100
GPUs, Jetson TX2, Intel Xeon CPU, Xilinx ZU9EG, and ZU3EG FPGAs
● Please refer to the paper for the detailed training configurations.
20

21. Evaluation
Performance of sub-networks on ImageNet
● top-1 accuracy under 224x224 resolution.
● Can achieve higher performance through Progressive Shrinking.
○ 74.8% top1 accuracy (D=4, W=3, K=3), which is on par with MobileNetV3-Large.
○ Without PS, it achieves 71.5%, which is 3.3% lower.
21
get the same architecture from
the full model w/o PS

22. Evaluation
Reduced Design Cost
● reports comparison between OFA and hardware-aware NAS methods
○ NAS: The design cost is linear to the number of deployment scenarios (N).
○ the total CO2 emissions of OFA is:
■ 16× fewer than ProxylessNAS
■ 19× fewer than FBNet
■ 1,300× fewer than MnasNet
22

23. Evaluation
OFA under Different Computational Resource Constraints
● Better accuracy under the same constraints:
○ (Left): MACs, (Right): Latency
○ Achieves higher accuracy, Requires lower computations
○ Better than “OFA - Train from scratch”, which is trained from the scratch without pretraining.
23

24. Discussions
● Would it work if the same approach is applied to other models, tasks (e.g.,
Transformer, NLP)?
● The architecture search space is limited to certain models.
○ e.g. How to apply the method to models such as HRNet?
24

25. Conclusion
● Once-for-all(OFA) Network allows training one large model and deploying
various sub-networks without additional training.
● OFA suggests Progressive Shrinking algorithm to share and find
sub-networks, which highly reduces the design cost.
● The paper shows OFA can achieve higher performance with ImageNet
dataset.
● With a trained OFA network, optimal sub-networks can be found on various
deployment environments.
25

26. Q&A
26