Real-time entertainment services such as streaming audio-visual content deployed over the open, unmanaged Internet account now for more than 70% during peak periods. More and more such bandwidth hungry applications and services are proposed like immersive media services such as virtual reality and, specifically omnidirectional/360-degree videos. The adaptive streaming of omnidirectional video over HTTP imposes an important challenge on today’s video delivery infrastructures which calls for dedicated, thoroughly designed techniques for content generation, delivery, and consumption. This paper describes the usage of tiles — as specified within modern video codecs such HEVC/H.265 and VP9 — enabling bandwidth efficient adaptive streaming of omnidirectional video over HTTP and we define various streaming strategies. Therefore, the parameters and characteristics of a dataset for omnidirectional video are proposed and exemplary instantiated to evaluate various aspects of such an ecosystem, namely bitrate overhead, bandwidth requirements, and quality aspects in terms of viewport PSNR. The results indicate bitrate savings from 40% (in a realistic scenario with recorded head movements from real users) up to 65% (in an ideal scenario with a centered/fixed viewport) and serve as a baseline and guidelines for advanced techniques including the outline of a research roadmap for the near future.

1. Towards Bandwidth Efficient Adaptive Streaming of
Omnidirectional Video over HTTP
Design, Implementation, and Evaluation
Priv.-Doz. Dr. Christian Timmerer
Alpen-Adria-Universität Klagenfurt (AAU) w Faculty of Technical Sciences (TEWI) w Department of Information
Technology (ITEC) w Multimedia Communication (MMC) w Sensory Experience Lab (SELab)
http://blog.timmerer.com w http://selab.itec.aau.at/ w http://dash.itec.aau.at w christian.timmerer@itec.aau.at
Chief Innovation Officer (CIO) at bitmovin GmbH
http://www.bitmovin.com w christian.timmerer@bitmovin.com
ACM MMSys 2017, Taipei, Taiwan, June 20, 2017
http://www.slideshare.net/christian.timmerer

2. Outline
• Introduction
• Omnidirectional Video (ODV) Streaming
• Tile-based Adaptive Streaming
• Evaluations
• Conclusions and Future Work
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 2

3. Importance of Multimedia Delivery
• Real-time entertainment: Streaming video and audio;
>70% of Internet traffic at peak periods (forecast: 2021 >80%)
• Popular services: YouTube (17.53%), Netflix (35.15%), Amazon Video (4.26%), Hulu (2.68%); all
delivered over-the-top (OTT); huge potential for mobile access!
• More people now subscribe to Netflix (50.85M) than cable TV (48.61M) in the US (Q1 2017)
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 3
Global Internet Phenomena Report: 2016
https://www.businessinsider.com/netflix-more-popular-than-cable-us-chart-2017-6/

4. Over-The-Top – Adaptive Media Streaming
• In a nutshell…
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 4
Adaptation logic is within the
client, not normatively specified
by the standard, subject to
research and development

5. June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 5
Example Platform/Infrastructure
https://bitmovin.com/

6. What is Omnidirectional Video (ODV)?
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 6
Capturing Devices
• Stitching, Projection Formats
• Encoding, Encryption, Encapsulation
• Storage, Content Distribution, Delivery
• Processing, Decoding, Rendering, …
Consumer Devices

7. Adaptive Streaming of ODV Content
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 7
https://bitmovin.com/mpeg-vr-one-ring-rule
Projection format:
Equirectangular
projection
Coding format: AVC
File format: ISOBMFF, M2TS
Delivery format: DASH, HLS
Web clients: HTML5,
MSE, WebVR
https://multimediacommunication.blogspot.co.at/2017/03/360-degree-live-adaptive-streaming-with.html

8. Tile-based Adaptive Streaming
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 8
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Encoding &
Packaging
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Delivery
MPEG-HEVC/H.265
Tiles in ISOBMFF
Adaptive
Player
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile-based streaming of VR/360°
content with MPEG-DASH SRD
…
…
…
…
Adaptive Streaming using
MPEG-DASH SRD
Head Mounted Displays
Browsers, Smart (Mobile) Devices
(Stereo) 2D, (Stereo) 3D

9. Tile-based Streaming Options
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 9

10. ODV Quality Metrics: S-PSNR
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 10
M. Yu, H. Lakshman, and B. Girod. 2015. A Framework to Evaluate Omnidirectional Video Coding Schemes. In 2015 IEEE
International Symposium on Mixed and Augmented Reality (ISMAR). 31–36. DOI:http://dx.doi.org/10.1109/ISMAR.2015.12

11. ODV Quality Metrics: V-PSNR
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 11
M. Yu, H. Lakshman, and B. Girod. 2015. A Framework to Evaluate Omnidirectional Video Coding Schemes. In 2015 IEEE
International Symposium on Mixed and Augmented Reality (ISMAR). 31–36. DOI:http://dx.doi.org/10.1109/ISMAR.2015.12

12. Preliminary Evaluation Results
• Bitrate overhead due to various tiling patterns
• Bandwidth requirements
• Quality (viewport PSNR)
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 12

13. Dataset
• Segment length / Intra period
– 1s (tiled content) vs 1, 2, 4s (monolithic content)
• Tiling pattern (columns × rows): 1×1, (i.e., tiles
monolithic), 3×2, 5×3, 6×4, and 8×5
• Resolution: 1920×960, 3840×1920 and 7680×3840
• Map projection: equirectangular format
• Quantization parameter: QP={22,27,32,37,42}
• Head motion recordings for V-PSNR evaluation
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 13

14. Bitrate Overhead due to Tiling
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 14
500 1000 1500 2000 2500
384042444648
Bitrate [kbps]
Y−PSNR[dB]
●
●
●
●
●
●
● 1x1 tiles
3x2 tiles
5x3 tiles
6x4 tiles
8x5 tiles
Tile Overhead for Resolution: 1920x960
Sequence: AssassinsCreed
2000 4000 6000 8000
30323436384042
Bitrate [kbps]
Y−PSNR[dB]
●
●
●
●
●
●
● 1x1 tiles
3x2 tiles
5x3 tiles
6x4 tiles
8x5 tiles
Tile Overhead for Resolution: 1920x960
Sequence: ExploreTheWorld

15. Bandwidth Requirements (1)
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 15

16. Bandwidth Requirements (2)
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 16

17. Bandwidth Requirements (3)
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 17

18. Viewport PSNR (1)
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 18

19. Viewport PSNR (2)
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 19

20. Conclusions
• System architecture and encoding options for tile-
based streaming
• 6 × 4 provides the best tradeoff
– View-port selection flexibility
– Bitrate overhead
– Bandwidth requirements
• Tile-based streaming offers bitrate savings up to 65%
• Viewport PSNR: bitrate savings up to 40%
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 20

21. Roadmap for Future Work (1)
• Publicly available dataset enabling reproducible research in this
area
• Other codecs supporting tiles (incl. decoding options)
• Investigation of advanced adaptation/streaming strategies
• Incorporation of machine learning techniques to predict head
movements (possibly including saliency maps, eye tracking in
combination with HMDs)
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 21

22. Roadmap for Future Work (2)
• Usage of different projection formats reducing the
limitations of the equirectangular project format
• Utilization of HTTP/2 push mechanisms as requesting
each tile individually drastically increases the number
of HTTP requests which may impact the overall
streaming performance
• Subjective quality assessments as a prerequisite for
QoE models including various/different tilling patterns
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 22

23. Thank you for your attention
June 20, 2017 Dr. Timmerer [AAU/Bitmovin] 23
... questions, comments, etc. are welcome …
Priv.-Doz. Dipl.-Ing. Dr. Christian Timmerer
Associate Professor
Alpen-Adria-Universität Klagenfurt, Department of Information Technology (ITEC)
Universitätsstrasse 65-67, A-9020 Klagenfurt, AUSTRIA
christian.timmerer@itec.uni-klu.ac.at
http://blog.timmerer.com/
Tel: +43/463/2700 3621 Fax: +43/463/2700 3699
© Copyright: Christian Timmerer