Empowered by today’s rich tools for media generation and collaborative production and convenient network access to the Internet, video streaming has become very popular. Dynamic adaptive video streaming is a technique used to deliver video content to users over the Internet, where the quality of the video adapts in real time based on the network conditions and the capabilities of the user’s device. HTTP Adaptive Streaming (HAS) has become the de-facto standard to provide a smooth and uninterrupted viewing experience, especially when network conditions frequently change. Improving the QoE of users concerning various applications‘ requirements presents several challenges, such as network variability, limited resources, and device heterogeneity. For example, the available network bandwidth can vary over time, leading to frequent changes in the video quality. In addition, different users have different preferences and viewing habits, which can further complicate live streaming optimization. Researchers and engineers have developed various approaches to optimize dynamic adaptive streaming, such as QoE-driven adaptation, machine learning-based approaches, and multi-objective optimization, to address these challenges. In this talk, we will give an introduction to the topic of video streaming and point out the significant challenges in the field. We will present a layered architecture for video streaming and then discuss a selection of approaches from our research addressing these challenges. For instance, we will present approaches to improve the QoE of clients in User-generated content applications in centralized and distributed fashions. Moreover, we will present a novel architecture for low-latency live streaming that is agnostic to the protocol and codecs that can work equally with existing HAS-based approaches.

1. How to Optimize Dynamic Adaptive Video
Streaming? Challenges and Solutions
Farzad Tashtarian
Department of Information Technology
Klagenfurt University
Feb. 27, 2023

2. Farzad Tashtarian (Ph.D.)
Postdoctoral Researcher (2021-Present)
University of Klagenfurt, Austria
Previous position:
Assistant Professor,
Azad University of Mashhad, Iran
About me
ATHENA Project
2
https://www.tashtarian.net/
farzad.tashtarian@aau.at
Adaptive Streaming over HTTP and Emerging Networked Multimedia Services

3. Agenda
➢ An Introduction on Dynamic Video Streaming
➢ Proposed Layered Architecture
➢ Where to be optimized? Input Space and Action Domain
➢ LALISA: Adaptive Bitrate Ladder Optimization in HTTP-based
Adaptive Live Streaming
➢ Future work
➢ Conclusion
3

4. Introduction
According to a report by Cisco, video streaming accounts for more than 80% of all
internet traffic.
As of 2021, YouTube has over 2 billion monthly active users who watch over a billion
hours of video every day.
Netflix has over 209 million subscribers globally.
Amazon Prime Video had over 175 million subscribers worldwide in 2021, with over 200
million hours of video content streamed every day.
In 2020, Twitch had an average of 30 million daily active users, who collectively watched
over 1 trillion minutes of content.
In 2021, the global video streaming market was valued at $50.11 billion and is expected to
reach $184.27 billion by 2027, growing at a CAGR of 20.4% from 2020 to 2027.
https://www.marketwatch.com/press-release/video-streaming-market-size-worth-18427-billion-by-2027-cagr-204-grand-view-research-inc-2021-07-27
4

5. Internet
Dynamic Adaptive Video Streaming In A Glance
Origin Server
Bandwidth
HTTP Response
5
HTTP Request
HTTP Request
HTTP Response
HTTP Request
HTTP Response
HTTP Response
HTTP Request
HTTP Request
HTTP Response
Client/Player
incoming buffer
outgoing buffer

6. Main Components In Video Streaming Path
Video Contribution Video Distribution
Internet
Network Core Network Edge
CDN Network
Cloud/Datacenter
Video on Demand
(VOD)
Live Video Source
HAS Players
HAS Players
ISP
Base station
Video Consumption
6

7. Layered Architecture of Dynamic Adaptive Streaming
Management and Control Layer
Resource Layer
Video Streaming Application Layer
7

8. Video Streaming Application Layer
Management and Control Layer
Resource Layer
Video Streaming Applications
Video on Demand (VOD) Live Video Streaming
Entertainment
Education
Events
Social Media
News, Gaming
Video Streaming
Application Layer
8

9. Video Streaming Application Layer
Management and Control Layer
Resource Layer
Video Streaming Applications
Application Requirements
Quality
Video Streaming
Application Layer
Quality in video streaming refers to the overall visual and audio experience of the content
being streamed. The quality of a video stream is determined by several factors, including
resolution, bit rate, frame rate, and audio quality.
9

10. Video Streaming Application Layer
Management and Control Layer
Resource Layer
Video Streaming Applications
Application Requirements
Quality Latency
Video Streaming
Application Layer
Latency is defined as the delay between the moment when the video signal is sent and the
moment when it is received and displayed by the viewer.
10

11. Video Streaming Application Layer
Management and Control Layer
Resource Layer
Video Streaming Applications
Application Requirements
Quality Latency Resource
Video Streaming
Application Layer
How much resources in terms of computation, storage and bandwidth are needed?
11

12. Video Streaming Application Layer
Management and Control Layer
Resource Layer
Video Streaming Applications
Application Requirements
Quality Latency Resource Reliability
Video Streaming
Application Layer
The ability of the streaming service to consistently deliver content to viewers without
interruption or failure. This includes:
● The stability and performance of the video player
● The availability and quality of the video content
● The responsiveness of the streaming service to user requests
12

13. Video Streaming Application Layer
Management and Control Layer
Resource Layer
Video Streaming Applications
Application Requirements
Quality Latency Resource Scalability
Reliability
Video Streaming
Application Layer
Scalability in video streaming refers to the ability of the streaming service to handle
increasing numbers of viewers without degradation of performance.
13

14. Management and Control Layer
Management and Control Layer
Resource Layer
Video Streaming
Application Layer
Video Streaming
Applications
Application
Requirements
Serving Policy
How to serve players’ requests:
● Fetch from an origin server
● Serve with using transcoding (using a segment with higher quality) function
● Serve with sending a lower quality
● ...
14

15. Management and Control Layer
Management and Control Layer
Resource Layer
Video Streaming
Application Layer
Video Streaming
Applications
Application
Requirements
QoS and QoE Controller
Serving Policy
To dynamically optimize the streaming, we need to control QoE and QoS metrics:
● QoE metrics (i.e, quality, bitrate switching, stall)
● QoS metrics (i.e, delay, jitter, packet loss )
How to calculate them?
● We need some feedback from clients, origin server, and network.
15

16. Management and Control Layer
Management and Control Layer
Resource Layer
Video Streaming
Application Layer
Video Streaming
Applications
Application
Requirements
App. Statistics Collector
QoS and QoE Controller
Serving Policy
There are some options:
● Deploying an Analytics Server
● Using CMCD* (i.e., adding some data into the HTTP request message)
Is it possible to have statistic from all parts of streaming path? Yes and No. It depends on the
type of information and time sensitiveness.
*CMCD: Common-Media-Client-Data 16

17. Management and Control Layer
Management and Control Layer
Resource Layer
Video Streaming
Application Layer
Video Streaming
Applications
Application
Requirements
Resource Monitoring App. Statistics Collector
QoS and QoE Controller
Serving Policy
To monitor available resources and active process using different tools:
● Network Performance Monitoring (NPM) tools
● Server Monitoring tools
● Bandwidth Monitoring tools
17

18. Management and Control Layer
Management and Control Layer
Resource Layer
Video Streaming
Application Layer
Video Streaming
Applications
Application
Requirements
Resource Monitoring
Task Assignment
Resource Allocation
App. Statistics Collector
QoS and QoE Controller
Serving Policy
Resource Allocation and Task Assignment are two main functions for launching (modifying) a
new (existing) streaming service.
18

19. Management and Control Layer
Management and Control Layer
Resource Layer
Video Streaming
Application Layer
Video Streaming
Applications
Application
Requirements
Resource Monitoring
Video Streaming
Controller and
Optimizer
Task Assignment
Resource Allocation
App. Statistics Collector
QoS and QoE Controller
Serving Policy
The core module that communicates with other functions.
19

20. Resource Layer
Management and Control Layer
Resource Layer
Video Streaming
Application Layer
Video Streaming
Applications
Application
Requirements
Resource Monitoring
Video Streaming
Controller and Optimizer
Task Assignment
Resource Allocation
App. Statistics Collector
QoS and QoE metrics
Computation
Serving Policy
For example, computation resources for encoding, transcoding, superresolution, running
algorithm, and models. 20

21. Resource Layer
Management and Control Layer
Resource Layer
Video Streaming
Application Layer
Video Streaming
Applications
Application
Requirements
Resource Monitoring
Video Streaming
Controller and Optimizer
Task Assignment
Resource Allocation
App. Statistics Collector
QoS and QoE metrics
Computation Storage
Serving Policy
For example, to store video data (i.e., segments) and metadata. 21

22. Resource Layer
Management and Control Layer
Resource Layer
Video Streaming
Application Layer
Video Streaming
Applications
Application
Requirements
Resource Monitoring
Video Streaming
Controller and Optimizer
Task Assignment
Resource Allocation
App. Statistics Collector
QoS and QoE metrics
Computation Storage Bandwidth
Serving Policy
For example, bandwidth resource for delivering (i) requested video data to the player and (ii)
video data from origin server to the CDN servers. 22

23. Where to be optimized?
23

24. Where to be optimized?
Internet
Network Core Network Edge
CDN Network
Cloud/Datacenter
VOD
Live Source
HAS Players
HAS Players
ISP
Base
Station
24
Client Side
(e.g., ABR)
Edge Network
(e.g., online transcoding,
serving strategy)
CDN Network
(e.g., caching policy)
Cloud Network
(e.g., resource allocation)
Origin Side
(e.g., encoding parameters)

25. ○ Advantages:
■ Easy deployment
● Access to required (local) inputs variable
● Apply actions on considered (local) elements
■ Low computation time compared with a global solution
○ Main challenge:
■ Local extrema
Local Optimization
25

26. Where to be optimized?
Internet
Network Core Network Edge
CDN Network
Cloud/Datacenter
VOD
Live Source
HAS Players
HAS Players
ISP
Base
Station
End-to-End Optimization

27. ○ Advantages:
■ Having a holistic view of the streaming path
■ End-to-end approach
○ Challenges:
■ High computation time
■ Difficult deployment
● Need to have inputs from various components
● Using CMCD
● Analytic server
● Need to employ SDN architecture
Global Optimization
27

28. Which parameters should be considered?
Which parts should be a ected?
28
Where to be optimized?

29. ● Input Space:
○ Which parameters should be taken into account by
the proposed solution?
● Action Domain:
○ Where should be affected by the output(s) of
solution?
29

30. Solutions
● Application Layer
○ Content complexity
○ Quality of Experience
○ User preference
○ Required latency
○ Required resources
○ Required scalability
○ Required availability
○ Security and privacy
○ Fairness
○ etc.
● Resource Layer
○ Available resources (i.e,
computation, storage,
bandwidth)
○ Performance
○ Network Quality of
Service (QoS)
○ Cost
○ etc.
● Video Contribution
○ Per-title encoding
○ Encoding parameters
○ Bitrate ladder
○ Segment size
○ etc.
● Video Distribution
○ Data path between
streamer and encoder
○ Multi-path data
transmission between
MEC server
○ Function placement (i.e.,
VNF (transcoding))
○ Cache management
○ Utilization
● Video Consumption
○ ABR algorithm design
○ Super resolution
○ etc.
Inputs Space Action Domain
30

31. ● Application Layer
○ Content Complexity
○ Quality of Experience
○ User preference
○ Required latency
○ Required resources
○ Required scalability
○ Required availability
○ Security and privacy
○ Fairness
○ etc.
● Resource Layer
○ Available resources (i.e,
computation, storage,
bandwidth)
○ Performance
○ Network Quality of
Service (QoS)
○ Cost
○ etc.
● Video Contribution
○ Per-title encoding
○ Encoding parameters
○ Bitrate ladder
○ Segment size
○ etc.
● Video Distribution
○ Data path between
streamer and encoder
○ Multi-path data
transmission between
MEC server
○ Function placement (i.e.,
VNF (transcoding))
○ Cache management
○ Utilization
● Video Consumption
○ ABR algorithm design
○ Super resolution
○ etc.
Solutions
Inputs Space Output Domain
31

32. Inputs Space Action Domain
● Video Contribution
○ Per-title encoding
○ Encoding parameters
○ Bitrate ladder
○ Segment size
○ etc.
● Video Distribution
○ Data path between
streamer and encoder
○ Multi-path data
transmission between
MEC server
○ Function placement (i.e.,
VNF (transcoding))
○ Cache management
○ Utilization
● Video Consumption
○ ABR algorithm design
○ Super resolution
○ etc.
Solutions
● Application Layer
○ Content complexity
○ Quality of Experience
○ User preference
○ Required latency
○ Required resources
○ Required scalability
○ Required availability
○ Security and privacy
○ Fairness
○ etc.
● Resource Layer
○ Available resources (i.e,
computation, storage,
bandwidth)
○ Performance
○ Network Quality of
Service (QoS)
○ Cost
○ etc.
32

33. Categories of Solutions
Meta-heuristic
● Local optimum solution
○ Simulated-annealing
○ Evolutionary algorithm
Mathematical
Optimization
● Time complexity issue
● Use some techniques to distributed solutions
○ ADMM: Alternating Direction Method of Multipliers
Machine Learning
& AI techniques
● Useful techniques for real-time applications
● Accurate in its predictions
● Suitable for distributed solutions
● Various techniques for different scenarios
○ Reinforcement learning
Solutions
33

34. LALISA: Adaptive Bitrate Ladder Optimization in
HTTP-based Adaptive Live Streaming
Tashtarian, F., Bentaleb, A., Amirpour, H., Taraghi, B, Timmerer, C., Hellwagner, H., and Zimmermann, R.
The 36th IEEE/IFIP Network Operations and Management Symposium (NOMS),
8-12 May- Miami, FL, USA, 2023.
34

35. Encoder
Chunk
Origin Server
CDN Server
CDN Server
Internet
4800 kbps
2400 kbps
1200 kbps
800 kbps
Fixed bitrate ladder
Segment Delivery In Live Streaming
Player
Player
35
Huang, Tianchi, Rui-Xiao Zhang, and Lifeng Sun. "Deep reinforced bitrate ladders for adaptive video streaming."
Proceedings of the 31st ACM Workshop on Network and Operating Systems Support for Digital Audio and Video. 2021.

36. Problem Definition
How to optimize the bitrate ladder in the live streaming applications?
How to consider the content , network condition (i.e., available
bandwidth) and users’ demands (i.e., desired bitrate)
36

37. Inputs Space Action Domain
● Video Contribution
○ Per-title encoding
○ Encoding parameters
○ Bitrate Ladder
○ Segment size
○ etc.
● Video Distribution
○ Data path between streamer
and encoder
○ Multi-path data transmission
between MEC server
○ Function placement (i.e., VNF
(transcoding))
○ Cache Management
○ Utilization
● Video Consumption
○ ABR Algorithm design
○ Super resolution
○ etc.
● Application Layer
○ Content complexity
○ Quality of Experience
○ User preference
○ Required Latency
○ Required Resources
○ Required Scalability
○ Required Availability
○ Security and Privacy
○ Fairness
○ etc.
● Resource Layer
○ Available resources (i.e,
computation, storage,
bandwidth)
○ Performance
○ Network Quality of
Service (QoS)
○ Cost
○ etc.
LALISA
37

38. Selected Bitrate vs. Desired Bitrate
Player
ABR Alg.
38
4.8 Mbps
2.4 Mbps
1.2 Mbps
0.8 Mbps
Manifest file
CDN/
Origin
Server
ABR Objective Function
Buffer
Bandwidth
4.8 Mbps
2.4 Mbps
1.2 Mbps
0.8 Mbps
MAX bps
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
min bps
X=1.8Mbps
1.8 Mbps
1.2 Mbps
Desired Bitrate
Selected Bitrate

39. Encoder
Chunk
Origin Server
Internet
Fixed bitrate ladder
Motivating Example
450 Players
39
4.3 Mbps
3.0 Mbps
1.5 Mbps
0.7 Mbps
0.15 Mbps
Base Station

40. Encoder
Origin Server
Motivating Example
450 Players
40
Selected bitrate Desired bitrate
50 players → 4.3 50 players → 4.3
80 players → 3.0
50 players → 3.2
30 players → 3.0
150 players → 1.5
50 players → 2.5
100 players → 2.0
0 players → 1.5
120 players → 0.7
100 players → 1.0
20 players → 0.7
50 players → 0.15 50 players →0.15
4.3 Mbps
3.0 Mbps
1.5 Mbps
0.7 Mbps
0.15 Mbps

41. Encoder
Origin Server
Motivating Example
450 Players
41
Selected bitrate Desired bitrate Quality (VMAF)
50 players → 4.3 50 players → 4.3 98
80 players → 3.0
50 players → 3.2 97
30 players → 3.0 96.5
150 players → 1.5
50 players → 2.5 96
100 players → 2.0 95
0 players → 1.5 80
120 players → 0.7
100 players → 1.0 75
20 players → 0.7 65
50 players → 0.15 50 players → 0.15 20
VMAF: Video Multimethod Assessment Fusion
(full-reference video quality metric)
4.3 Mbps
3.0 Mbps
1.5 Mbps
0.7 Mbps
0.15 Mbps

42. Encoder
Origin Server
Motivating Example
42
Selected bitrate Desired bitrate Quality
50 players → 4.3 50 players → 4.3 98
80 players → 3.0
50 players → 3.2 97
30 players → 3.0 96.5
150 players → 1.5
50 players → 2.5 96
100 players → 2.0 95
0 players → 1.5 80
120 players → 0.7
100 players → 1.0 75
20 players → 0.7 65
50 players → 0.15 50 players → 0.15 20
4.3 Mbps
2.0 Mbps
1.0 Mbps
0.7 Mbps
0.15 Mbps
2.0 Mbps
1.0 Mbps
0.7 Mbps
0.15 Mbps
2.0 Mbps
1.0 Mbps
0.15 Mbps
1.0 Mbps
0.15 Mbps
LALISA
L=5 LALISA
L=4
LALISA
L=3 LALISA
L=2
4.3 Mbps
3.0 Mbps
1.5 Mbps
0.7 Mbps
0.15 Mbps

43. Encoder
Origin Server
Motivating Example
43
Avg. VMAF Avg. BW (Mbps)
Fixed 74.2 1.7
LALISA (L=5) 81.5 2.0
LALISA (L=4) 80.8 1.5
LALISA (L=3) 78.8 1.4
LALISA (L=2) 66.4 0.8
VMAF and bandwidth comparison
Quality
Cost($)
4.3 Mbps
3.0 Mbps
1.5 Mbps
0.7 Mbps
0.15 Mbps
4.3 Mbps
2.0 Mbps
1.0 Mbps
0.7 Mbps
0.15 Mbps
2.0 Mbps
1.0 Mbps
0.7 Mbps
0.15 Mbps
2.0 Mbps
1.0 Mbps
0.15 Mbps
1.0 Mbps
0.15 Mbps
LALISA
L=5 LALISA
L=4
LALISA
L=3 LALISA
L=2
Fixed

44. CMCD: Common-Media-Client-Data
How does a player send its desired bitrate to
the server?
44
Player
ABR Alg.
Origin Server
HTTP Request
URL
https://videserver/seg1.mp4?db=x

45. LALISA Architecture
45

46. LALISA Player (LP) Agent
46
The LP agent is actually a plug-in that
can be deployed with a HAS client to
assist the ABR algorithm in extracting
the desired bitrates.

47. LALISA Analytics (LA) Server
47
Run a mathematical
optimization model
to determine optimal
bitrate ladder (BL).
(seg#, selected br, desired br)

48. Maximize a * Quality - (1-a)* Bitrate
Each request should be served
with one Bitrate
Maximum generated data
regarding selected bitrates
Amount of VMAF improvement
by serving desired bitrates
Amount of VMAF degradation
by serving with lower bitrates
Maximum length of the bitrate
ladder
Variables: 48

49. LALISA testbed
49
● Segment duration:
○ 1, 2, and 4 seconds
● Experiment time:
○ 120 seconds
● Network trace:
○ LTE (player), Fixed (origin)
● Codec:
○ H.264
● Content:
○ Animation
● Player#:
○ 10, 20, 40

50. Measured QoE score and VMAF values for different numbers of clients and
network traces with 1s segment duration
50

51. Impact of LALISA on Bandwidth and Computation Cost
51

52. Future Work
● Extend the idea of LALISA to cope with its limitations
● Consider energy in end-to-end live video streaming
● Leverage distributed ML/AI techniques for designing
end-to-end approaches
● End-to-end immersive multimedia streaming optimization
52

53. Conclusion
● Introduced the layered architecture for video streaming
application
○ Application layer
○ Management and control layer
○ Resource layer
● Local vs. global approaches
● Input space and action domain
● LALISA solution for determining the dynamic bitrate ladders
53

54. https://www.tashtarian.net/ farzad.tashtarian@aau.at
Published more than 20 journal and 30 conference papers

55. How to Optimize Dynamic Adaptive Video
Streaming? Challenges and Solutions
Farzad Tashtarian
Department of Information Technology
Klagenfurt University
Feb. 27, 2023