Video is by far the predominant consumer of network capacity. Yet, the Internet is a ‘video repellent’ machine, one that can transfer data but has no notion of deadlines. So what are we getting from modern video services? How can we measure quality of experience? And can we predict the quality perceived by the user, starting from simple network measurements? In this talk I give a critical perspective on conventional QoE assessment, ending up with a controversial proposition. This talk is based on the following material: V. Menkovski, A. Liotta 'Adaptive Psychometric Scaling for Video Quality Assessment' Journal of Signal Processing: Image Communication (Elsevier, 2012) http://bit.ly/JSP-2012 V. Menkovski, G. Exarchakos, A. Liotta 'The Value of Relative Quality in Video Delivery' Journal of Mobile Multimedia. Vol.7(3), pp. 151-162 (Sept. 2011) http://bit.ly/JMM-2011 V. Menkovski, G. Exarchakos, A. Liotta 'Online Learning for Quality of Experience Management' The annual machine learning conference of Belgium and The Netherlands, Leuven, Belgium, 2010 http://bit.ly/BENELEARN-2010 A. Liotta, G. Exarchakos 'Networks for Pervasive Services: Six Ways to Upgrade the Internet' Springer (2011) http://bit.ly/pervasive-networks

1. Can networks deliver
Quality of Experience?
Prof. Antonio Liotta
Eindhoven University of Technology, NL
http://bit.ly/autonomic_networks
Twitter: a_liotta

2. Four questions about Quality of Experience
• Why is QoE so important?
• How good are we at scoring QoE?
• How can we use Machine Learning to manage QoE?
• Can networks learn about QoE too?
Prof. A. Liotta 2

3. Why we should NEVER ignore QoE in streaming services
Sending rate: 2048 Kbps
HIGHER
39.48 dB
- 23.85 dB
Received quality
LOWER
15.65 dB
Prof. A. Liotta 3

4. Why we should NEVER ignore QoE in streaming services
Sending rate: 768 Kbps
LOWER
35.15 dB
- 16.03 dB
Received quality
HIGHER
19.12 dB
Prof. A. Liotta 4

5. QoE ≠ Σ QoS
Video
stream
Overprovisioning !!
Prof. A. Liotta 5

6. QoE results from a diversity of factors
Transport Environmental
Encoding
Prof. A. Liotta 6

7. Understanding QoE is essential for video delivery
(Internet not designed for QoS nor QoE)
4 3 2 1 2 1
Server A 1
4
3
4 3 2 4 3
Client B
No guarantees about:
• Delivery of all packets
• Delivery order
• Delivery time
Further details:
A.Liotta, G.Exarchakos, “The network as we know it”
Prof. A. Liotta (Springer, 2011) – http://bit.ly/pervasive-networks 7

8. Without ‘edge tricks’ the Internet wouldn’t operate
(MPEG-4 video, 1% packet loss, no buffering)
J. Okyere-Benya, M. Aldiabat, V. Menkovski, G. Exarchakos, A. Liotta
Video Quality Degradation on IPTV Networks, IEEE ICNC‟12, USA, Feb. 2012
http://bit.ly/ICNC-12
Prof. A. Liotta 8

9. Trends: 1. Traffic explosion
Anything on video – more producers than watchers?
(*)
Source: Cisco data
Estimated IP traffic in 2015: 1.3 Zettabytes (1021 bytes) (*)
forecast, Feb 2011
Prof. A. Liotta 9

10. Trends: 2. extremely mobile video
(mobile video responsible for majority of mobile traffic)
Prof. A. Liotta 10

11. Trends: 3. M2M video
(new communication patterns and ‘volumes’)
Building
ICT
autom.
Security
Energy
& Safety
Consu-
mer & Retail
Home
Health- Transpor
care Indust. tation
Autom.
(*)
Source: Cisco data
forecast, Feb 2011
Prof. A. Liotta 11

12. Trends: 4. opportunistic communications
(even more ‘video repellent’ than ordinary IP network)
A.Liotta, G.Exarchakos, “Spontaneous networks”
Prof. A. Liotta (Springer, 2011) – http://bit.ly/pervasive-networks 12

13. Trends: 4. opportunistic communications
(even more ‘video repellent’ than ordinary IP network)
A.Liotta, G.Exarchakos, “Spontaneous networks”
Prof. A. Liotta (Springer, 2011) – http://bit.ly/pervasive-networks 13

14. QoE: the missing link towards closed-loop delivery
Video
stream
N-QoS
A-QoS Objective Subjective
inference
QoE QoE
Some Some control
Expensive process
actuators techniques
Prof. A. Liotta 14

15. Subjective QoE: so far a helpless effort
Annoying
Expensive
Inaccurate
Impossible to capture
complexity and
variability of the
delivery context
Prof. A. Liotta 15

16. Absolute rating comes with huge variability and bias
(mean opinion score)
35%
VQEG HD5
30%
25%
STDEV[%scale]
20% 17,94% 1
15%
10%
5%
0%
0% 50% 100% 0
MOS [%scale]
Further details:
“Report on the Validation of Video Quality Models for High Definition Video Content”
by the Video Quality Experts Group, Jun. 2010.
Prof. A. Liotta 16

17. Human perception is biased
(the same video format gives different perceptions)
K. Seshadrinathan et al. “Study of subjective and objective quality assessment of video”
Prof. A. Liotta 17
IEEE Trans. Image Processing, Vol.29(6). June 2010.

18. Both videos are encoded exactly in the same way but
the “pedestrian” one is “perceived” as worse
• Difference from unimpaired reference is greater in ‘pedestrian’ video
• But the overlapping Gaussians indicate that many subjects had opposite
perceptions
Differential scoring is more
accurate than absolute MOS
BUT IT’S CLEAR THAT
Prof. A. Liotta
humans are not good at ranking 18

19. Can we use ‘machine learning’ to model QoE on a
QoE domain continuous scale?
Cumulative
Gaussian
(2 parameters)
QoS domain
Psychophysics quantitatively investigates the relationship
Prof. A. Liotta between physical stimuli and the sensation of perception 19

20. A more effective question
(Maximum Likelihood Different Scaling)
Which one of these two pairs has bigger difference?
Prof. A. Liotta 20

21. Responses fit a psychometric curve
(human perception)
DEVIATION OF RESPONSES
BETWEEN 1 AND 10%
DEPENDING ON VIDEO TYPE
V. Menkovski, G. Exarchakos, A. Liotta,
The Value of Relative Quality in Video Delivery,
Journal of Mobile Multimedia. Vol.7(3), pp. 151-162 (Sept. 2011)
http://bit.ly/JMM-2011
Prof. A. Liotta 21

22. Use of psychometric curves for QoE management
(resource optimization zone)
ZONE 1
QoS deltas
don’t produce
delta QoEs
364 Kbps
Prof. A. Liotta 512 Kbps 22

23. Use of psychometric curves for QoE management
(quality optimization zone)
ZONE 2
strong
non
linearity
64 Kbps
Prof. A. Liotta 256 Kbps 23

24. MLDS is more accurate than conventional QoE rating
but still unscalable
• Must consider all combinations of samples
• A full round of tests including 10 levels of stimuli requires
10
210 tests
4
• The test matrix explodes as we consider more parameters
Can we speed up the prediction-model
learning process?
Prof. A. Liotta 24

25. Active learning helps eliminating the redundant tests
• After the first few test we can start estimating
the answers of the remaining tests
• The estimation of the unanswered test uses
the characteristics of the psychometric curve
to reduce the problem domain
River bed
Tractor
Blue sky
Prof. A. Liotta 25

26. Learning convergence varies for different videos but
always leads to improved scalability
V. Menkovski, A. Liotta, Adaptive Psychometric Scaling for Video Quality Assessment
Journal of Signal Processing: Image Communication (Elsevier, 2012)
http://bit.ly/JSP-2012
Prof. A. Liotta 26

27. Reinforcement Learning to realize
‘trial & error’ network loops
„Sport over mobile phone‟
QoS probe
actuators
Optimizing QoE Machine QoE measure
QoS prediction Learning or inference
Prof. A. Liotta 27

28. Networks quickly learn how to deal with new conditions
(problem domain is constrained to psychometric function)
100
95
90
85
Accuracy
80
75
70
Old conditions New conditions
65
60
55
50 1030
1090
1150
1020
1080
1140
10
70
130
190
250
310
370
430
490
550
610
670
730
790
850
910
970
60
120
180
240
300
360
420
480
540
600
660
720
780
840
900
960
New ‘trial & error’ samples
V. Menkovski, G. Exarchakos, A. Liotta, Online Learning for Quality of Experience Management
The annual machine learning conference of Belgium and The Netherlands, Leuven, Belgium, 2010
http://bit.ly/BENELEARN-2010
Prof. A. Liotta 28

29. Take-home messages
• Existing QoE methods are
– annoying, expensive, inaccurate, ineffective
• What is the ‘right’ question?
– we are good at spotting difference of differences
• Learning how to deal with new perturbations is more
promising than brute-force subjective studies
– human perception is a moving target
– ML works with incomplete information, extrapolates non-
obvious patterns and handles the unknown via trial&error
Prof. A. Liotta 29

30. Thank you !
More about my work http://bit.ly/autonomic_networks
In the press http://bit.ly/press_articles
“All of YouTube through “Cognitive “Networks for
a 40-year-old funnel” Interconnections” pervasive services”
http://bit.ly/Volkskrant-EN http://bit.ly/booklet-antonio http://bit.ly/pervasive-networks
Prof. A. Liotta 30