This document summarizes an upcoming presentation on HTTP Adaptive Streaming. The presentation will cover content provisioning, delivery, consumption, and end-to-end aspects of HAS, as well as quality of experience. It will introduce ATHENA, a research center focused on adaptive streaming over HTTP and emerging multimedia technologies. The agenda outlines sections on video encoding for HAS, edge computing, network assistance for clients, bitrate adaptation schemes, and quality of experience models. The presenters are Christian Timmerer and Hermann Hellwagner from Alpen-Adria-Universität Klagenfurt.

1. HTTP Adaptive Streaming – Quo Vadis?
Christian Timmerer, Assoc.-Prof. at AAU, Director at CD Lab ATHENA
Hermann Hellwagner, Professor at AAU
Klagenfurt, Austria
June 29, 2021
1

2. “By 2022, Internet video will represent
82% of all Internet traffic.”
Cisco Visual Networking Index: Forecast and Trends, 2017–2022 (White Paper), Cisco,
February 2019.
2

3. Presenters
Christian Timmerer
Assoc.-Prof at
Alpen-Adria-Universität Klagenfurt
Director CD Lab ATHENA
CIO | Head of Research and
Standardization at Bitmovin
Hermann Hellwagner
Professor at
Alpen-Adria-Universität Klagenfurt
3
3
2003: MSc CS (Dipl.-Ing.)
2006: PhD CS (Dr.-techn.)
2013: Co-founded Bitmovin
2014: Habilitation (Priv.-Doz.) & Assoc. Prof.
2016: Dep. Director @ ITEC/AAU
2019: Director @ ATHENA
Web: http://timmerer.com/

4. 4
4

5. ● Introduction
● ATHENA
○ Content Provisioning
○ Content Delivery
○ Content Consumption
○ End-to-End Aspects
○ Quality of Experience
● Conclusions: HAS – Quo Vadis?
Agenda
5

6. Video streaming is dominating today’s Internet traffic
● May 2020: 57.64%; YouTube is the undisputed king with
15.94% followed by Netflix with 11.42%*
● By 2022, video will account for 82% of global IP traffic and
live video will increase 15-fold and reach 17% of Internet
video traffic**
Motivation
Sources: * Sandvine Global
Internet Phenomena (May 2020).
** Cisco Visual Networking Index
(VNI), Complete Forecast Update,
2017–2022 (Feb. 2019)
6
6

7. HTTP Adaptive Streaming 101
Adaptation logic is within the
client, not normatively specified
by a standard, subject to
research and development
7
7
Client

8. Multimedia Systems Challenges and Tradeoffs
8
8
Basic figure by Klara Nahrstedt, University of Illinois at Urbana–Champaign, IEEE MIPR 2018

9. “Application-oriented basic research” to address current and future research
and deployment challenges of HAS and emerging streaming methods
ATHENA – Adaptive Streaming over HTTP and
Emerging Networked Multimedia Services
Content Provisioning Content Delivery Content Consumption
End-to-End Aspects
● Video encoding for HAS
● Quality-aware encoding
● Learning-based encoding
● Multi-codec HAS
● Edge computing
● Information CDN/SDN⇿clients
● Netw. assistance for/by clients
● Utility evaluation
● Bitrate adaptation schemes
● Playback improvements
● Context and user awareness
● Quality of Experience (QoE) studies
● Application/transport layer enhancements
● Quality of Experience (QoE) models
● Low-latency HAS
● Learning-based HAS
https://athena.itec.aau.at/
9
9
Funding:

10. Video coding for HTTP Adaptive Streaming
● Quality improvement
Per-Title Encoding (PTE) et al.: per-title/scene/shot/segment, content-/context
aware, content-adaptive, quality-aware encoding
● Runtime improvement
Hardware-/software-based (cloud), parallel/distributed, information reuse from
reference encodings (multi-rate/-resolution)
● Application scenarios
Video on Demand (VoD incl. diff. flavors AVoD, SVoD), live (incl. diff. flavors),
interactive, games, video conferencing
Content Provisioning
10
10

11. Video Encoding
Block
Partitioning
Motion
Compensation
Transformation
& Quantization
Entropy Coding
Entropy
Decoding
Inverse
Transformation &
Inverse
Quantization
Inter or Intra
Prediction
Picture Buffer In-loop Filtering
11

12. Video Encoding with Machine Learning
12
12
Block
Partitioning
Motion
Compensation
Transformation
& Quantization
Entropy
Coding
Entropy
Decoding
Inverse
Transformation &
Inverse Quantization
Inter or Intra
Prediction
Picture
Buffer
In-loop
Filtering
CTU
Decision
Prediction
Optical Flow
Detection
Mode
Prediction
Angular
Direction
Prediction
Deblocking
with ML
Denoising
with ML
Super-resolution

13. 1)
D. Schroeder et al. "Efficient multi-rate video encoding for HEVC-based adaptive HTTP streaming."
IEEE Transactions on Circuits and systems for Video Technology 28.1 (2016): 143-157.
2)
B. Guo, Y. Han, J. Wen, "Fast Block Structure Determination in AV1-based Multiple Resolutions Video
Encoding," 2018 IEEE Int’l. Conf. on Multimedia and Expo (ICME), San Diego, CA, USA, July 2018.
3)
H. Amirpour, E. Çetinkaya, C. Timmerer and M. Ghanbari, "Fast Multi-rate Encoding for Adaptive
HTTP Streaming," Data Compression Conference (DCC), Snowbird, UT, USA, 2020,
Encoding information can be used among different
quality representations
State-of-the-art:
● Encode the highest quality1)
or the lowest quality2)
as the
reference first, then use this information
Proposed method:
● Encode the highest quality first, then use its information to
encode the lowest quality and then use information from
both representations to encode the remaining
representations3)
● Double bound for CTU search ranges
Fast Multi-rate Encoding
13
13
QP1
QPN
QPN-1
QP3
QP2
...
Encoding
runtime
(norm.)

14. Parallel encoding is still problematic
State-of-the-art:
● Encode the highest quality1)
or the lowest quality2)
as the
reference first, then use this information
Proposed method:
● Try different quality levels as the reference representation to
determine the best starting point for parallel encoding
● Encode the middle quality first, then use its information to
reduce the time-complexity for higher qualities to eliminate
possible bottlenecks3)
● Upper or lower bound depending on the quality level
Fast Multi-rate Encoding
1)
Schroeder et al., 2016 2)
Guo et al., 2018
3)
H. Amirpour, E. Çetinkaya, C. Timmerer and M. Ghanbari, "Towards Optimal
Multirate Encoding for HTTP Adaptive Streaming," International MultiMedia
Modeling Conference (MMM), Prague, Czech Republic, 2021
QPN/2
QPN
QP2
QP1
...
14
14
Encoding
runtime
(norm.)

15. Combining multi-rate with multi-resolution approaches in x265
State-of-the-art:
● Encoder analysis sharing method in x2651)
Proposed method:
● Multi-encoding Algorithm-1: Scaled CU information from highest
bitrate representation of the previous resolution is used as lower
bound of CU depth estimation process
● Multi-encoding Algorithm-2: Scaled CU information from lowest
bitrate representation of the previous resolution is used as lower
bound of CU depth estimation process. More rigid bound
compared to Multi-encoding Algorithm-12)
Fast Multi-Encoding
1)
Aruna Mathesawaran et al. “Open source framework for reduced-complexity multi-rate
HEVC encoding”. In: Applications of Digital Image Processing XLIII. SPIE, 2020, pp. 461 –471.
2)
V. V Menon, H. Amirpour, C. Timmerer and M. Ghanbari, "Efficient Multi-Encoding
Algorithms for HTTP Adaptive Bitrate Streaming," Picture Coding Symposium (PCS) 2021,
Bristol (UK), 2021
15
15
Encoder analysis sharing method in
Proposed Multi-Encoding Algorithm-2
Results for the multi-rate algorithms
Results for the multi-encoding algorithms

16. Use ML to encode dependent representations
State-of-the-art:
● Use a CNN to predict CTU depth decisions1)
Proposed method:
● Train a CNN with encoding information obtained from
the reference quality (the lowest quality)
representation and use its decision to encode
dependent representations2)
● Focus on parallel encoding, thus only apply for
bottleneck situations
● Train different CNNs for different QP targets
Fast Multi-rate Encoding
with Machine Learning (FaME-ML)
1)
Kim, Kyungah, and Won Woo Ro. "Fast CU depth decision for HEVC using neural networks."
IEEE Transactions on Circuits and Systems for Video Technology 29.5 (2018): 1462-1473.
2)
E. Çetinkaya, H. Amirpour, C. Timmerer and M. Ghanbari, “FaME-ML: Fast Multirate
Encoding for HTTP Adaptive Streaming Using Machine Learning,” 2020 IEEE International
Conference on Visual Communications and Image Processing (VCIP), Macau, 2020
QPN
CNN
QPN-1
QP1
QP2
...
HEVC
HEVC
HEVC
CNN
HEVC HEVC
16
16
Encoding
runtime
(norm.)

17. Extending FaME-ML to Multi-Resolution scenario
State-of-the-art:
● Use the highest quality representation as reference1)
Proposed method:
● Train a CNN with encoding information obtained from
the reference representation (the highest quality from
the lowest resolution) and use its decision to encode
dependent representations2)
● Improves parallel encoding as well as serial encoding
● Train different CNNs for different QP and resolution
targets
Fast Multi-resolution Encoding
with Machine Learning (FaRes-ML)
1)
D. Schroeder et al. "Efficient multi-rate video encoding for HEVC-based adaptive HTTP streaming." IEEE
Transactions on Circuits and systems for Video Technology 28.1 (2016): 143-157.
2)
E. Çetinkaya, H. Amirpour, C. Timmerer and M. Ghanbari, “Fast Multi-Resolution and
Multi-Rate Encoding for HTTP Adaptive Streaming Using Machine Learning,” in IEEE Open
Journal of Signal Processing, doi: 10.1109/OJSP.2021.3078657
17
17
HEVC
QP1
HEVC
QP2
CNN
HEVC
QPN
CNN
HEVC
..
CNN
HEVC
QP2
CNN
HEVC
QPN
CNN
HEVC
..
CNN
HEVC
QP2
CNN
HEVC
QPN
CNN
HEVC
..
CNN
HEVC
QP1
CNN
HEVC
QP1
CNN
540p
540p
1080p
2160p
Encoding
runtime
(norm.)

18. End Game
ML
Encoder
ML
Decoder
18
18
See also:
● CLIC: Workshop and Challenge on Learned Image Compression, https://www.compression.cc/
● JPEG AI-based image coding, https://jpeg.org/
● JVET (MPEG/VCEG) AI-based video coding, http://mpeg.org/

19. Network assistance for HTTP Adaptive Streaming
● Edge computing support (at CDN / cellular network edge)
Functions at (or, assisted by) the edge: adaptation, analytics, (pre-)fetching,
caching, transcoding, repackaging of content, request aggregation
● Server/network/CDN ↔ HAS client information exchange and collaboration
IETF ALTO, MPEG SAND, MPEG NBMP, …; SDN-DASH, SDN-HAS, SABR, ...
● Use of modern network architecture features
SW Defined Networking (SDN); Network Function Virtual. (NFV); MC-ABR
● Low-latency live streaming
Use of MPEG CMAF, HTTP/1.1 Chunked Transfer Encoding (CTE), other protocol
features (e.g., HTTP/2 Push); LL-HLS; specific network functions; CDN support
Content Delivery
19
19

20. Approach:
● Deliver CMAF segments only over the core/CDN
● Repackage into requested format at the edge
Evaluation:
● Analytical model and simulation to assess bandwidth
savings as compared to all-formats delivery (~ 20%)
● Measurements to get segment repackaging times
(CMAF➝HLS: 45-67 ms, depending on seg. size)
● Real world-like testbed to assess “full” repackaging time
(avg. 136 ms, CAdViSE on AWS cloud, 4-sec. seg./1080p)
Dynamic Segment Repackaging at the Edge for HAS
20
20
Jesus Aguilar-Armijo, Babak Taraghi, Christian Timmerer, and Hermann Hellwagner.
”Dynamic Segment Repackaging at the Edge for HTTP Adaptive Streaming”.
IEEE Int’l. Symposium on Multimedia (ISM'20). Dec. 2020.

21. Approach:
● Employ SDN and NFV concepts to mitigate
Multicast ABR problems
● SDN: to set up and optimize multicast paths
● VRPs (Virtual Proxies): to aggregate clients’ requests
● VTFs (Virtual Transcoders): to transcode segments
to quality levels requested by clients
● MILP optimization model to jointly construct
optimal multicast tree and VTFs placement
● Heuristic algorithm (polynomial time)
On Optimizing Resource Utilization in Live Video Streaming
21
21
Alireza Erfanian, Farzad Tashtarian, Reza Farahani, Christian Timmerer, and Hermann Hellwagner.
”On Optimizing Resource Utilization in AVC-based Real-time Video Streaming”.
IEEE Conf. on Network Softwarization (NetSoft'20). June/July 2020.

22. Example: MC-ABR OSCAR OSCAR (VTFs closer to edge)
On Optimizing Resource Utilization in Live Video Streaming
22
22
37 Mbps
37 Mbps

23. Sample results:
On Optimizing Resource Utilization in Live Video Streaming
23
23
Bandwidth
/
#
OF
commands
(norm.)
Comparison of proposed algorithm (heuristic) and
other approaches in terms of bandwidth
consumption and # OpenFlow commands
generated for different network sizes
Comparison of proposed algorithm and other
approaches in terms of bandwidth consumption
and # OpenFlow commands generated for
different homogeneity levels of requests
(small-scale network)
Request sets (RS): homog. LQ / HQ
heterogeneous

24. Approach:
● Mechanism: Introduce a new server/segment selection approach at the edge of the network
● Main goal: Improve the users' QoE and network utilization
ES-HAS: An Edge- and SDN-Assisted Framework for HTTP
Adaptive Video Streaming
24
24
R. Farahani, F. Tashtarian, A. Erfanian, C. Timmerer, M. Ghanbari, and H. Hellwagner. ”ES-HAS: An
Edge- and SDN-Assisted Framework for HTTP Adaptive Video Streaming”.
The 31st edition of the Workshop on Network and Operating System Support for Digital Audio and
Video (NOSSDAV’21), Sept. 28-Oct. 1, 2021, Istanbul, Turkey.

25. Player Adaptation Logic and Quality of Experience
● Bitrate adaptation schemes
Client-based, server-based, network-assisted, hybrid, ML-based
● Application/transport layer enhancements
HTTP/2 (TCP) and HTTP/3 (QUIC), proprietary formats (SRT, RIST, …), WebRTC,
low-latency/delay
● Client playback improvements
User-aware playback, content-enhancement filters, super-resolution
● Quality of Experience
Objective and subjective quality assessment, models, analytics
Content Consumption and End-to-End Aspects
25
25

26. Bitrate Adaptation Schemes
Bitrate
Adaptation
Schemes
Client-based
Adaptation
Bandwidth
-based
Buffer-
based
Mixed
adaptation
Proprietary
solutions
MDP-based
Server-based
Adaptation
Network-
assisted
Adaptation
Hybrid
Adaptation
SDN-based
Server and
network-
assisted
A. Bentaleb, B. Taani, A.C. Begen, C. Timmerer and R. Zimmermann, "A Survey on Bitrate Adaptation Schemes for
Streaming Media Over HTTP," IEEE Communications Surveys & Tutorials, vol. 21, no. 1, pp. 562-585, First Quarter 2019.
26
26
Adaptive bitrate (ABR) algorithms, adaptation logics, ...

27. Adaptive bitrate (ABR) algorithms choose the lowest-quality segments in the
startup phase; quality switches due to throughput fluctuations.
H2BR: complementary to existing ABR utilizing HTTP/2 features
● Server push: piggyback retrans. segments
● Stream priority: for concurrent streams
● Stream termination: for retrans. segments
HTTP/2-Based Retransmission (H2BR)
27
27
Minh Nguyen, Christian Timmerer, and Hermann Hellwagner. “H2BR: An
HTTP/2-based retransmission technique to improve the QoE of adaptive
video streaming”. 25th ACM Workshop on Packet Video (PV'20), June 2020.

28. Client/server architecture with two computers
● HTTP/2 server and HTTP/2 client (both based on nghttp2)
● Dummynet emulates a state-of-the-art mobile network trace
Video content
● Big Buck Bunny: 596 seconds
● Segment duration: 1s, 2s, 4s, 6s
● Quality: 20 versions
● Resolutions: 320x240, 480x360, 854x480, 1280x720, 1920x1080
Compared method
● SQUAD1)
H2BR Evaluation Setup
28
28
Dummynet
Throughput-based AGG
Buffer-based BBA
Hybrid SARA
Last throughput
1)
Cong Wang, Divyashri Bhat, Amr Rizk, Michael Zink..
“Design and Analysis of QoE-Aware Quality Adaptation
for DASH: A Spectrum-Based Approach”. ACM Trans.
Multimedia Comput. Commun. Appl. 13, 3s, Article 45,
August 2017.

29. H2BR Experimental Results
29
29
Overall QoE score based on the ITU-T P.1203 QoE model mode 0
CDF of video quality in an experimental run (segment duration = 4s)
H2BR can decrease lowest-quality
playback by up to more than 70%
QoE increase by up to 13%

30. Quality of Experience (QoE) ...
● “... is the degree of delight or annoyance of the user of an application or service.
It results from the fulfillment of his or her expectations with respect to the utility and / or
enjoyment of the application or service in the light of the user’s personality and current state.”1)
● … can be easily extended to various domains, e.g., immersive media experiences.2)
30
30
1)
P. Le Callet, S. Möller, A. Perkis, et al. QUALINET White Paper
on Definitions of Quality of Experience. European Network on
Quality of Experience in Multimedia Systems and Services
(COST Action IC 1003). 2012.
2)
A. Perkis, C. Timmerer, et al. 2020. QUALINET White Paper on
Definitions of Immersive Media Experience (IMEx).
arXiv:2007.07032 [cs.MM]
3)
Jeroen van der Hooft, Tim Wauters, Filip De Turck, Christian
Timmerer, and Hermann Hellwagner. “Towards 6DoF HTTP
Adaptive Streaming Through Point Cloud Compression”.
27th ACM Int’l. Conf. on Multimedia (MM'19). Oct. 2019.
3)

31. Understanding Quality of Experience of
Heuristic-based HTTP Adaptive Bitrate Algorithms
● Perform (large-scale) objective/subjective evaluations of ABRs1)
using a cloud-based adaptive
video streaming evaluation framework for the automated testing of media players2)
31
31
1)
B. Taraghi, A. Bentaleb, C. Timmerer, R. Zimmermann, and H. Hellwagner. Understanding Quality of Experience of
Heuristic-based HTTP Adaptive Bitrate Algorithms. The 31st edition of the Workshop on Network and Operating System
Support for Digital Audio and Video (NOSSDAV’21), Sept. 28-Oct. 1, 2021, Istanbul, Turkey.
2)
B. Taraghi, A. Zabrovskiy, C. Timmerer, and H. Hellwagner. CAdViSE: Cloud-based Adaptive Video Streaming Evaluation
Framework for the Automated Testing of Media Players. ACM Multimedia Systems Conference 2020 (MMSys 2020)
Avg. QoE of ABRs in Fluctuation Network Profile. Avg. QoE of ABRs with Stable Network Profile.

32. QoE Evaluation for Adaptive Point Cloud Streaming
Volumetric media delivery with six degrees of freedom (6DoF) experience,
but significant bandwidth consumption
32
32
Jeroen van der Hooft, Maria Torres Vega, Filip De Turck, Christian Timmerer, Raimund Schatz, and Ali C. Begen.
“Subjective and Objective QoE Evaluation for Adaptive Point Cloud Streaming”.
12th Int’l. Conf. on Multimedia Quality of Experience (QoMEX’20). May 2020.
Slides courtesy of Jeroen van der Hooft (adapted)
4.1 Gbps 3.8 Gbps 5.7 Gbps 5.6 Gbps

33. QoE Evaluation for Adaptive Point Cloud Streaming
Volumetric media scene
33
33

34. QoE Evaluation for Adaptive Point Cloud Streaming
Point cloud compression (PCC, with MPEG reference encoder)
34
34
5.7 Gbps 40.4 Mbps 4.5 Mbps

35. Research questions:
1. What is the impact of network and content characteristics on the perceived
video quality?
2. How do objective metrics correlate with
subjective ratings for perceived video quality?
Methodology:
● 3 different video sequences were created
● Each one in 8 different configurations:
● Participants were asked to rate video quality
● Objective metrics were computed
QoE Evaluation for Adaptive Point Cloud Streaming
35
35
Bandwidth [Mbps] Allocation Prediction
20 Visible objects 0
60 Visible objects 0
100 Visible objects 0
20 Visible objects 1
60 Visible objects 1
100 Visible objects 1
60 All objects 0
∞ N/A N/A

36. Sample results (for RQ 1):
QoE Evaluation for Adaptive Point Cloud Streaming
36
36
Subjects can distinguish between different
bitrates
Viewport prediction allows to improve the
observed quality

37. Sample results (for RQ 2):
QoE Evaluation for Adaptive Point Cloud Streaming
37
37
Clear correlation between objective
metrics and MOS scores
Subjective scores match best with SSIM

38. Content Delivery
● Edge computing support
● CDNs, SDN, NVF, … ⇿ clients
● Low-latency live streaming
HTTP Adaptive Streaming – Quo Vadis?
38
38
Content Consumption & E2E
● Client-side content improvement, SR
● Machine learning
● Appl. & transport layer enhancements
https://athena.itec.aau.at/
Content Provisioning
● Content and context awareness
● Multi-codec (AVC, HEVC, VVC, VP9, AV1)
● Machine learning
Quality of Experience
● Immersive content
● User-aware playback
● Better/new QoE models and analytics

39. ATHENA team
Hadi Amirpour, Jesús Aguilar Armijo, Ekrem Çetinkaya, Alireza Erfanian, Reza Farahani,
Mohammad Ghanbari, David Langmeier, Vignesh V Menon, Minh Nguyen, Babak
Taraghi, Farzad Tashtarian; Hermann Hellwagner, Christian Timmerer
(Inter-)National Collaborators
● Prof. Ali C. Begen, Ozyegin University, Turkey
● Prof. Filip De Turck, Ghent University – imec, Belgium
● Dr. Jeroen van der Hooft, Ghent University – imec, Belgium
● Dr. Maria Torres Vega, Ghent University – imec, Belgium
● Dr. Raimund Schatz, AIT Austrian Institute of Technology, Austria
● Prof. Roger Zimmermann, NUS, Singapore
● Dr. Abdelhak Bentaleb, NUS, Singapore
Acknowledgments
39
39
https://athena.itec.aau.at/

40. Thank you for your attention
40

41. 41
41