The document summarizes key aspects of quality of experience (QoE) for video streaming services. It discusses that QoE is difficult to measure and manage due to human perception factors being non-linear and influenced by expectations. Both subjective and objective QoE measurement methods are explored, with their limitations noted. Machine learning is presented as a way to build models that can predict QoE and help optimize quality of service for video delivery in an adaptive way.

1. The human side of
video streaming services
Prof. Antonio Liotta
Eindhoven University of Technology
http://bit.ly/autonomic_networks
http://nl.linkedin.com/in/liotta
https://twitter.com/#!/a_liotta
www.slideshare.net/ucaclio
http://bit.ly/press_articles

2. Three questions about QoE
• What’s QoE?
• How can we measure QoE?
• Can we manage QoE?
Prof. A. Liotta 2

3. The video delivery chain
• An open loop system
• Over a best-effort network
• Operated via over-provisioning
Can we monitor the perceived quality?
Prof. A. Liotta 3

4. QoE is about perceived quality & satisfaction
“A person‟s individual perceptions and
responses that result from the use (or
anticipated use) of a system”
“An overall acceptability of a service, as
perceived subjectively by the end user”
“A measure of the overall end-to-end system
performance”
QoE is even more than this!
Prof. A. Liotta 4

5. Measuring QoE may seem straightforward
At which point does a video become unsatisfactory?
Prof. A. Liotta 5

6. Which factors affect QoE?
Very few are actually measurable !!
http://erasmus-ip-multimedia2012.ing.unimo.it/index.php/lectures-videos
Prof. A. Liotta Select: DAY 5 6

7. We can measure some technical factors but cannot
accurately correlate them with QoE perception
Prof. A. Liotta 7

8. We can measure some technical factors but cannot
accurately correlate them with QoE perception
Prof. A. Liotta 8

9. Particularly difficult to correlate network conditions with
video QoE perception
Technical factors If it weren’t for the ‘edge tricks’, packet
networks wouldn’t be able to stream
• best-effort vs. QoS audio/video
• Latency, jitter, packet loss, queue
size, …
• MPEG-4 video
• 1% packet loss
• no edge buffering
A. Liotta, L. Druda, G. Exarchakos, V. Menkovski, Quality of Experience management for
video streams: the case of Skype. In proc. of the 10th International Conference on Advances
Prof. A.in Mobile Computing and Multimedia, Bali, Indonesia, 3-5 December 2012 (ACM).
Liotta 9

10. The non-technical factors are even harder to measure and
correlate with QoE perception
The human visual system is non-linear
Prof. A. Liotta 10

11. Expectations affect QoE perception
Motivations, purpose, personal interest,
previous experience, boredom
Non-technical factors
expectations
Prof. A. Liotta 11

12. Same encoding, but the pedestrian video is perceived worse
Prof. A. Liotta 12

13. How can we measure QoE?
Prof. A. Liotta 13

14. Two main options
Subjective
QoE
Objective
QoE
http://erasmus-ip-multimedia2012.ing.unimo.it/index.php/lectures-videos
Select: DAY 5
Prof. A. Liotta 14

15. Subjective QoE
“The most reliable way of assessing the quality of
a video as perceived by a particular human
observer is to ask his [or her] opinion” (*)
But can we really measure human perception through questions?
• It’s hard to formulate the right questions
• Questionnaires are intrusive
• How many subjects give sufficient confidence?
(*) A. C. Bovik, The Essential Guide to Video Processing, Academic Press, 2009
Prof. A. Liotta 15

16. An engineering approach to subjective QoE
Single stimulus
Double stimulus
ITU-T Rec. P910 (*) provides guidelines for:
• Standard viewing conditions
• Criteria for the selection of observers
• Test material preparation
• Assessment procedures
• Data analysis methods
(*) ITU-T Rec. P.910 (09/99), Subjective video quality assessment methods for multimedia applications, 2008
Prof. A. Liotta 16

17. Single stimulus
• The subject watches an impaired video and rates its quality
without making any comparison with the original unimpaired
sequence
• The grading scale
is defined as Mean
Opinion Score (MOS)
Prof. A. Liotta 17

18. Single stimulus Huge variability and bias
35%
VQEG HD5
30%
25%
STDEV[%scale]
20% 17,94%
15%
10%
5%
0%
0% 50% 100%
MOS [%scale]
Typically, the STDEV of MOS is 15-20% in the midrange and decreases at the edges
“Report on the Validation of Video Quality Models for High Definition Video Content”
by the Video Quality Experts Group, Jun. 2010.
Prof. A. Liotta 18

19. We are better at spotting
Double stimulus
differences
Prof. A. Liotta 19

20. Double stimulus Better than single stimulus
but not good enough
Prof. A. Liotta 20

21. A broad range of objective video quality metrics
Full-reference metrics
compare the distorted video directly with its original
sequence. The most reliable but only applicable to off-line
assessment
No-reference metrics
Assess the distorted video without any reference video.
Measure image distortions, e.g. blockiness, blur, noise. Used
to assess the impact of transport errors but far less reliable
than FR
Reduced-reference metrics
Evaluate the distorted video based on a series of features that
have been extracted from the reference video. Used for QoE
prediction and management but less reliable than FR.
Prof. A. Liotta 21

22. PSNR: the most loved-hated metric
These two images have the same PSNR
Prof. A. Liotta 22

23. The problems with existing QoE assessment (both
subjective and objective)
• Not sufficiently accurate
• Meant for off-line study
• Not meant to correlate with QoS
Hard to close the loop, needed to manage video services !
Prof. A. Liotta 23

24. QoE management is a
machine learning problem
Prof. A. Liotta 24

25. Maximum Likelihood Different Scaling maps responses to a
psychometric function (the human perception curve)
DEVIATION OF RESPONSES
BETWEEN 1 AND 10%
DEPENDING ON VIDEO TYPE
V. Menkovski, G. Exarchakos, A. Liotta,
The Value of Relative Quality in Video Delivery,
Journal of Mobile Multimedia. Vol.7(3), pp. 151-162 (Sept. 2011)
http://bit.ly/JMM-2011
Prof. A. Liotta 25

26. MLDS works because we are much better at scoring
difference of differences
Which one of these two pairs has bigger difference?
Prof. A. Liotta 26

27. We can score ‘difference of differences’ even with video
(not just still pictures)
Prof. A. Liotta 27

28. MLDS provides a utility function to perform
QoE management
ZONE 1
QoS deltas
don’t produce
delta QoEs
364 Kbps
Prof. A. Liotta 512 Kbps 28

29. MLDS provides a utility function to perform
QoE management
ZONE 2
strong
non
linearity
64 Kbps
Prof. A. Liotta 256 Kbps 29

30. MLDS is more accurate than conventional QoE rating
but still unscalable
• Must consider all combinations of samples
• A full round of tests including 10 levels of stimuli requires
10
210 tests
4
• The test matrix explodes as we consider more parameters
Can we speed up the prediction-model
learning process?
Prof. A. Liotta 30

31. Active learning helps eliminating the redundant tests
• After the first few test we can start estimating
the answers of the remaining tests
• The estimation of the unanswered test uses
the characteristics of the psychometric curve
to reduce the problem domain
River bed
Tractor
Blue sky
Prof. A. Liotta 31

32. Learning convergence varies for different videos but
always leads to improved scalability
V. Menkovski, A. Liotta, Adaptive Psychometric Scaling for Video Quality Assessment
Journal of Signal Processing: Image Communication (Elsevier, 2012)
http://bit.ly/JSP-2012
Prof. A. Liotta 32

33. Closing the QoE control loop
QoS probe
actuators
Optimizing QoE QoE MLDS
QoS prediction models
Prof. A. Liotta 33

34. But we’ll also have to face NEW conditions !!
„Sport over mobile phone‟
QoS probe
actuators
Optimizing QoE QoE
QoS prediction models
Prof. A. Liotta 34

35. Reinforcement Learning to realize
‘trial & error’ network loops
„Sport over mobile phone‟
QoS probe
actuators
Optimizing QoE Machine QoE measure
QoS prediction Learning or inference
Prof. A. Liotta 35

36. Networks quickly learn how to deal with new conditions
(problem domain is constrained to psychometric function)
100
95
90
85
Accuracy
80
75
70
Old conditions New conditions
65
60
55
50 1030
1090
1150
1020
1080
1140
10
70
130
190
250
310
370
430
490
550
610
670
730
790
850
910
970
60
120
180
240
300
360
420
480
540
600
660
720
780
840
900
960
New ‘trial & error’ samples
V. Menkovski, G. Exarchakos, A. Liotta, Online Learning for Quality of Experience Management
The annual machine learning conference of Belgium and The Netherlands, Leuven, Belgium, 2010
http://bit.ly/BENELEARN-2010
Prof. A. Liotta 36

37. Take-home messages
• Existing QoE methods are
– annoying, expensive, inaccurate, ineffective
– can’t be used to control video services
• What is the ‘right’ question?
– we are good at spotting difference of differences
– off-line machine learning to build e2e models of video
services
• Service management is a ‘learning’ problem
– human perception is a moving target
– ML works with incomplete information, extrapolates non-
obvious patterns and handles the unknown via trial&error
Prof. A. Liotta 37

38. Thank you !
Check out my other Webinars at
www.slideshare.net/ucaclio
Want to author or
edit a book?
New Springer Series:
Internet of Things –
Technology,
Communications
and Computing
Get in touch!!
http://bit.ly/pervasive-networks liotta.antonio@gmail.com
Prof. A. Liotta 38