*** QoE Metrics *** While Quality of Experience (QoE) has advanced very significantly as a field in recent years, the methods used for analyzing it have not always kept pace. When QoE is studied, measured or estimated, practically all the literature deals with Mean Opinion Scores (MOS). The MOS provides a simple scalar value for QoE, but it has several limitations, some of which are made clear in its name: for many applications, just having a mean value is not sufficient. For service and content providers in particular, it is more interesting to have an idea of how the scores are distributed, so as to ensure that a certain portion of the user population is experiencing satisfactory levels of quality, thus reducing churn. We put forward the limitations of MOS, present other statistical tools that provide a much more comprehensive view of how quality is perceived by the users, and illustrate it all by analyzing the results of several subjective studies with these tools. *** QoE Fairness *** User-centric service and application management focuses on the Quality of Experience (QoE) as perceived by the end user. Thereby, the goal is to maximize QoE while ensuring fairness among users, e.g., for resource allocation and scheduling in shared systems. Although the literature suggests to consider consequently QoE fairness, there is currently no accepted definition of QoE fairness. The contribution of this paper is the definition of a generic QoE fairness index F which has desirable key properties as well as the rationale behind it. By using examples and a measurement study involving multiple users downloading web content over a bottleneck link, we differentiate the proposed index from QoS fairness and the widely used Jain’s fairness index. Based on results, we argue that neither QoS fairness nor Jain’s fairness index meet all of the desirable QoE-relevant properties which are met by F. Consequently, the proposed index F may be used to compare QoE fairness across systems and applications, thus serving as a benchmark for QoE management mechanisms and system optimization.

1. Prof. Dr. Tobias Hoßfeld
Chair of Modeling of
Adaptive Systems (MAS)
Institute for Computer
Science and Business
Information Systems (ICB)
University of Duisburg-Essen
www.mas.wiwi.uni-due.de
On QoE Metrics and QoE
Fairness for Network & Traffic
Management
Tobias Hossfeld
Lea Skorin-Kapov
Poul Heegaard

2. QoE Metrics and QoE Fairness
10/14/2016 2
System
Fairness of system is
evaluated by considering all
QoE values f(QoS).
Subjects evaluate test conditions
e.g. on a 5-point scale.
QoE Model
e.g. f(QoS) = MOS
QoE Fairness
QoS Measurement
QoE Metrics
How to define and
calculate QoE fairness?
Which QoE metrics are of
interest for providers?

3. MOS AND QOE
Hoßfeld, Tobias, Poul E. Heegaard, Martín Varela, and Sebastian Möller. "QoE
beyond the MOS: an in-depth look at QoE via better metrics and their relation to
MOS." Quality and User Experience, no. 1(2) (2016).
http://link.springer.com/journal/41233

4. Quality of Experience
• From Quality of service (QoS) to Quality of Experience
(QoE)
– QoS: packet loss, delay, jitter, …
– QoE: subjective experience/satisfaction of users of a service
• Example: video user interested in video quality and
smooth video playout without interruptions
• QoE model required for evaluation, improving QoE by
proper monitoring and management…
Input video
(known reference)
10/14/2016 4

5. Mean Opinion Score (MOS)
• Mean Opinion Score (MOS): numerical indication of the perceived
quality of received media after compression and/or transmission
Excellent
Good
Fair
Poor
Bad
5
4
3
2
1
Imperceptible
Perceptible
Slightly annoying
Annoying
Very annoying
MOS Quality Impairment
Excellent!
Bad!
Fair!Good!
Poor!

Fair = 3
10/14/2016 5

6. Same MOS but Different Distributions
10/14/2016 6

7. 1 2 3 4 5
0
0.2
0.4
0.6
0.8
1
MOS
probabilityofdissatisfieduser
q =3
QoE Beyond the MOS
-20° 80°
On average, it’s
fine!
But still in pain!
On average, it’s
fine!
Still, some suffer!
MOS: Fair = 3
MOS > 3:
75% dissatisfied !
 Other metrics!
10/14/2016 7

8. QoE Metrics
• MOS: average user rating for one test condition
• SOS: user diversity for that test condition
• Theta-Acceptability: prob. that opinion score is above certain threshold
• %GoB: the percentage of users rating Good-or-Better (%GoB)
• %PoW: the percentage of users rating Poor-or-Worse (%PoW)
• Quantile: user rating of fraction of (satisfied, dissatfied)
• Probability distribution: complete
information
• SOS parameter a
– quantifies user diversity for one application
– relates SOS and MOS
𝑆 𝑥 = 𝛼 −𝑥2
+ 𝑁𝑥 .
– SOS parameter is scale independent
10/14/2016 8
Hoßfeld, Tobias, Poul E. Heegaard, Martín Varela, and Sebastian Möller. "QoE beyond the MOS:
an in-depth look at QoE via better metrics and their relation to MOS." Quality and User
Experience, no. 1(2) (2016). http://link.springer.com/journal/41233

9. QOE FAIRNESS
Tobias Hoßfeld, Lea Skorin-Kapov, Poul E. Heegaard, Martin Varela. Definition of
QoE Fairness in Shared Systems. IEEE Comm. Letters, accepted Oct. 2016

10. QoE Metrics and QoE Fairness
10/14/2016 10
System
Fairness of system is
evaluated by considering all
QoE values f(QoS).
Subjects evaluate test conditions
e.g. on a 5-point scale.
QoE Model
e.g. f(QoS) = MOS
QoE Fairness
QoS Measurement
QoE Metrics
How to define and
calculate QoE fairness?
Which QoE metrics are of
interest for providers?

11. QoE Fairness
• How to define and calculate QoE fairness?
System
Fairness of system is
evaluated by considering all
QoE values f(QoS).
QoE Model
e.g. f(QoS) = MOS
QoE Fairness
QoS Measurement
10/14/2016 11

12. Which system is better?
• A) 10% experience best QoE; 90% worst QoE
• B) 90% experience best QoE; 10% worst QoE
10/14/2016 12
QoE on 5-point scale
• L=1: worst QoE
• H=5: best QoE
Normalized QoE (linear transformation)
• L=0: worst QoE
• H=1: best QoE

13. Which system is fairer?
• A) 10% experience best QoE; 90% worst QoE
• B) 90% experience best QoE; 10% worst QoE
10/14/2016 13
QoE on 5-point scale
• L=1: worst QoE
• H=5: best QoE
Normalized QoE (linear transformation)
• L=0: worst QoE
• H=1: best QoE

14. For which x is the system maximal unfair?
• In the system
– x% of users experience maximum (best) QoE: 𝑯 = 𝟓
– 100-x% of users experience minimum (worst) QoE: 𝑳 = 𝟏
10/14/2016 14
Jain’s fairness
index
𝐽 =
1
1+𝑐2 =
𝐸 𝑌 2
𝐸[𝑌2]

15. Desirable Properties of a QoE Fairness Index
• (a) Population size independence
• (b) Scale and metric independence
• (c) Boundedness [0;1]
• (d) Continuity
• (e) Intuitive
10/14/2016 15
Jain‘s fairness index
designed for those
properties

16. For which x is the system maximal unfair?
• In the system
– x% of users experience maximum (best) QoE: 𝑯 = 𝟓
– 100-x% of users experience minimum (worst) QoE: 𝑳 = 𝟏
10/14/2016 16

17. Desirable Properties of a QoE Fairness Index
• (a) Population size independence
• (b) Scale and metric independence
• (c) Boundedness [0;1]
• (d) Continuity
• (e) Intuitive
• (f) Deviation symmetric
• (g) QoE level independence
• (h) Valid for multi-applications
10/14/2016 17
Jain‘s fairness index
designed for those
properties
Specific to QoE
Fairness

18. Definition of QoE Fairness Index
• QoE model maps QoS parameters x to QoE in [𝐿; 𝐻]
𝑄: 𝑥 ↦ 𝑦 = 𝑄 𝑥 ∈ [𝐿; 𝐻]
– E.g. 𝑄(𝑥) is the MOS value on a 5-point scale, 𝐿 = 1, 𝐻 = 5
• In a system with 𝑛 users, QoE values are random variable 𝑌
• Maximum standard deviation of 𝑌
𝜎 𝑚𝑎𝑥 =
1
2
(𝐻 − 𝐿).
• Fairness index
10/14/2016 18
𝐹 = 1 −
𝜎
𝜎 𝑚𝑎𝑥
= 1 −
2𝜎
𝐻 − 𝐿

19. Illustration
10/14/2016 19
𝐹 = 1 −
𝜎
𝜎 𝑚𝑎𝑥
= 1 −
2𝜎
𝐻 − 𝐿
1
=L
5
=H
2 3 4
5-point
scale
x x x xx xxx xx xx x xx xx xx x
Avg. QoE 𝝁
𝟐𝝈
x single user experience

20. Issues with Jain‘s Fairness Index
• Coefficient of variation: only useful for ratio scales
– Requires natural zero point
– No meaning for data on interval scale
• QoE is given on interval scales
– Coefficient of variation is not a valid measurement
10/14/2016 20
1
=L
5
=H
2 3 4
5-point
scale
x x x xx xxx xx xx x xx xx xx x

21. Comparison: Jain and QoE Fairness Index
• Jain’s fairness index
𝐽 =
1
1+𝑐2 =
𝐸 𝑌 2
𝐸[𝑌2]
• QoE fairness index
𝐹 = 1 −
2𝜎
𝐻−𝐿
10/14/2016 21
Jain’s J violates
(f) Deviation symmetric
(g) QoE level independence
Jain’s J is not very sensitive.
J for max. standard deviation:
𝑱 𝒎𝒊𝒏 = 𝟎. 𝟔𝟗 (for 5-point scale)

22. Some numbers
Scenario~ Description J F
1 All users experience 1. 1 1
2 50% experience 1 and 50% experience 2. 0.90 0.75
3 50% experience 1 and 50% experience 3. 0.80 0.50
4 50% experience 1 and 50% experience 4. 0.74 0.25
5 50% experience 1 and 50% experience 5. 0.69 0.00
6 50% experience 2 and 50% experience 4. 0.90 0.50
7 50% experience 2.9 and 50% experience 4.9. 0.94 0.50
8 Uniform distribution 𝑌 ~ 𝑈(𝐿; 𝐻). 0.75 0.42
10/14/2016 22

23. QoS Fairness != QoE Fairness
10/14/2016 23

24. References
• Hoßfeld, Tobias, Poul E. Heegaard, Martín Varela, and Sebastian Möller.
"QoE beyond the MOS: an in-depth look at QoE via better metrics and
their relation to MOS." Quality and User Experience 1, no. 1 (2016): 2.
– Open access: http://link.springer.com/journal/41233
– Scripts: https://github.com/hossfeld/QoE-Metrics/wiki
– Formal Definition of QoE Metrics: http://arxiv.org/abs/1607.00321
• Tobias Hoßfeld, Lea Skorin-Kapov, Poul E. Heegaard, Martin Varela.
Definition of QoE Fairness in Shared Systems. IEEE Comm. Letters,
accepted Oct. 2016
– http://ieeexplore.ieee.org/document/7588099/
10/14/2016 24