Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Global Sensitivity Analysis of Predictor Models in Software Engineering
1. Global Sensitivity Analysis of Predictor
Models in Software Engineering
Stefan Wagner
Software & Systems Engineering
Technische Universit¨t M¨nchen
a u
Germany
wagnerst@in.tum.de
May 20, 2007
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 1 / 27
3. Predictor Models in Software Engineering
Predictor models describe situations in software engineering
Introduction
s
Predictor Models in
Various types, aims, . . .
Software
s
Engineering
Examples
s
Problem
Example:
Fischer-Wagner
COCOMO (costs)
x
Model
Questions About the
Musa-Okumoto (reliability)
x
Model
Overview
Fault-proneness
x
Global Sensitivity
Analysis
Approach for SE
Summary
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 3 / 27
4. Problem
The models themselves can be complex
Introduction
s
Predictor Models in
Their use needs a lot of effort
Software
s
Engineering
How can I analyse those models themeselves?
s
Problem
Example:
How can I simplify them?
s
Fischer-Wagner
Model
How do I improve their predictive power?
s
Questions About the
Model
Overview
Global Sensitivity
Analysis
Approach for SE
Summary
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 4 / 27
5. Example: Fischer-Wagner Model
Reliability model used at Siemens COM
Introduction
s
Predictor Models in
Two parameters estimated by failure data
Software
s
Engineering
Failure probability of a fault: pa = p1 · d(a−1)
s
Problem
Example:
Geometrical distribution : Fa (t) = 1 − (1 − pa )t
s
Fischer-Wagner
Model
Cummulated failures up to t:
s
Questions About the
Model
µ(t) = ∞ 1 − (1 − p1 · d(a−1) )
Overview a=1
Input parameters
s
Global Sensitivity
Analysis
p1 : Highest failure probability of a fault
x
Approach for SE
d: Complexity of the system ]0; 1[
x
Summary
t: Execution time (incidents)
x
inf : approximation of ∞
x
Output: µ(t)
s
More details: S. Wagner, H. Fischer. Ada-Europe, 2006
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 5 / 27
6. Questions About the Model
How do the input parameters influence the output?
Introduction
s
Predictor Models in
Which parameter(s) do I have to estimate best to get a good
Software
s
Engineering
prediction?
Problem
Example:
Are there insignificant parameters that I can remove?
s
Fischer-Wagner
Model
Questions About the
Model
Overview
Global Sensitivity
Analysis
Approach for SE
Summary
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 6 / 27
7. Overview
Introduction
Introduction
Predictor Models in
Software
Global Sensitivity Analysis
Engineering
Problem
Example:
Approach for SE
Fischer-Wagner
Model
Summary
Questions About the
Model
Overview
Global Sensitivity
Analysis
Approach for SE
Summary
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 7 / 27
9. Definition
Introduction
Saltelli (2000): “Sensitivity analysis studies the relationships
s
Global Sensitivity
between information flowing in and out of the model.”
Analysis
Definition
How do the input parameters influence the output?
s
Method
How can this influence by quantified?
Variance-based
s
Methods
Global properties
s
Main Effect
Higher-Order Effects
Inclusion of influence of shape and scale
x
Total Effect
Approach for SE
Multidimensional averaging
x
Summary
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 9 / 27
10. Method
Introduction Input Model Output
Global Sensitivity
Analysis
Definition
x1
Method
Variance-based
Methods
Main Effect
Higher-Order Effects
x2
Total Effect
Approach for SE
Summary Y
f (x)
x3
xn
(Based on slide of S. Tarantola)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 10 / 27
11. Method
Introduction Input Model Output
Global Sensitivity
Analysis
Definition
x1
Method
Variance-based
Methods
Main Effect
Higher-Order Effects
x2
Total Effect
Approach for SE
Summary Y
f (x)
x3
xn
(Based on slide of S. Tarantola)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 10 / 27
12. Method
Introduction Input Model Output
Global Sensitivity
Analysis
Definition
x1
Method
Variance-based
Methods
Main Effect
Higher-Order Effects
x2
Total Effect
Approach for SE
Summary Y
f (x)
x3
xn
(Based on slide of S. Tarantola)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 10 / 27
13. Method
Introduction Input Model Output
Global Sensitivity
Analysis
Definition
x1
Method
Variance-based
Methods
Main Effect
Higher-Order Effects
x2
Total Effect
Approach for SE
Summary Y
f (x)
x3
xn
(Based on slide of S. Tarantola)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 10 / 27
14. Method
Introduction Input Model Output
Global Sensitivity
Analysis
Definition
x1
Method
Variance-based
Methods
Main Effect
Higher-Order Effects
x2
Total Effect
Approach for SE
Summary Y
f (x)
x3
xn
(Based on slide of S. Tarantola)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 10 / 27
15. Method
Introduction Input Model Output
Global Sensitivity
Analysis
Definition
x1
Method
Variance-based
Methods
Main Effect
Higher-Order Effects
x2
Total Effect
Approach for SE
Summary Y
f (x)
x3
xn
(Based on slide of S. Tarantola)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 10 / 27
16. Method
Introduction Input Model Output
Global Sensitivity
Analysis
Definition
x1
Method
Variance-based
Methods
Main Effect
Higher-Order Effects
x2
Total Effect
Approach for SE
Summary Y
f (x)
x3
xn
(Based on slide of S. Tarantola)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 10 / 27
17. Method
Introduction Input Model Output
Global Sensitivity
Analysis
Definition
x1
Method
Variance-based
Methods
Main Effect
Higher-Order Effects
x2
Total Effect
Approach for SE
Summary Y
f (x)
x3
xn
(Based on slide of S. Tarantola)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 10 / 27
18. Method
Introduction Input Model Output
Global Sensitivity
Analysis
Definition
x1
Method
Variance-based
Methods
Main Effect
Higher-Order Effects
x2
Total Effect
Approach for SE
Summary Y
f (x)
x3
xn
(Based on slide of S. Tarantola)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 10 / 27
19. Method
Introduction Input Model Output
Global Sensitivity
Analysis
Definition
x1
Method
Variance-based
Methods
Main Effect
Higher-Order Effects
x2
Total Effect
Approach for SE
Summary Y
f (x)
x3
xn
(Based on slide of S. Tarantola)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 10 / 27
20. Method
Introduction Input Model Output
Global Sensitivity
Analysis
Definition
x1
Method
Variance-based
Methods
Main Effect
Higher-Order Effects
x2
Total Effect
Approach for SE
Summary Y
f (x)
x3
xn
(Based on slide of S. Tarantola)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 10 / 27
21. Variance-based Methods
Global analysis usually variance-based
Introduction
s
Global Sensitivity
“model-free”
s
Analysis
Definition
Analysis on the basis of sensitivity indices
s
Method
Variance-based
Main or first-order effect
x
Methods
Main Effect
x Higher-order effects
Higher-Order Effects
x Total effect
Total Effect
Approach for SE
Decomposition in main effects and interactions
s
Summary
k
y = f (x) = f0 + fi (xi ) + fij (xi , xj )
i=1 i j>i
+ . . . + f1,2,...,k (x1 , x2 , . . . , xk )
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 11 / 27
22. Variance-based Methods
Global analysis usually variance-based
Introduction
s
Global Sensitivity
“model-free”
s
Analysis
Definition
Analysis on the basis of sensitivity indices
s
Method
Variance-based
Main or first-order effect
x
Methods
Main Effect
x Higher-order effects
Higher-Order Effects
x Total effect
Total Effect
Approach for SE
Decomposition in main effects and interactions
s
Summary
For example k = 3:
f (x) = f0 + f1 (x1 ) + f2 (x2 ) + f3 (x3 ) + f12 (x1 , x2 )
+f13 (x1 , x3 ) + f23 (x2 , x3 ) + f123 (x1 , x2 , x3 )
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 11 / 27
23. Main Effect
Introduction
Decomposition of the variance V of f (x)
Global Sensitivity
Analysis
k
Definition
Var(y) = V = Vi + Vij Vijk + . . . + V1,2,...k
Method
Variance-based
Methods i=1 i j i j k
Main Effect
Higher-Order Effects
First-order sensitivity coefficient
Total Effect
Approach for SE
Vi
Si =
Summary
V
(Factors Priorisation)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 12 / 27
24. Higher-Order Effects
Higher-order indices analog
Introduction
Global Sensitivity
Analysis
Si1 ...io = Vi1 ...io /V
Definition
Method
Variance-based
Methods
Main Effect
Higher-Order Effects
Total Effect
Approach for SE
With k = 4:
Summary
2. order: S12 , S13 , S14 , S23 , S24 , S34
3. order: S123 , S124 , S134 , S234
4. order: S1234
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 13 / 27
25. Total Effect
Sum of the first- and higher-order effects of xi
Introduction
s
Global Sensitivity
Removal of the non-relevant parts
s
Analysis
Definition
Method
Variance-based
S1 + S2 + S3 + S4 + S12 + S13 + S14 + S23 + S24 + S34 +
Methods
Main Effect
S123 + S124 + S134 + S234 + S1234 = 1
Higher-Order Effects
Total Effect
Approach for SE
Summary
(Factors Fixing)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 14 / 27
26. Total Effect
Sum of the first- and higher-order effects of xi
Introduction
s
Global Sensitivity
Removal of the non-relevant parts
s
Analysis
Definition
Method
Variance-based
S1 + //2 + S3 + S4 + S12 + S13 + S14 + S23 + S24 + S34 +
S/
Methods
Main Effect
Higher-Order Effects
S123 + S124 + S134 + S234 + S1234 = 1
Total Effect
Approach for SE
Summary
(Factors Fixing)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 14 / 27
27. Total Effect
Sum of the first- and higher-order effects of xi
Introduction
s
Global Sensitivity
Removal of the non-relevant parts
s
Analysis
Definition
Method
Variance-based
S1 + //2 + / 3 + S4 + S12 + S13 + S14 + S23 + S24 + S34 +
S/ S //
Methods
Main Effect
Higher-Order Effects
S123 + S124 + S134 + S234 + S1234 = 1
Total Effect
Approach for SE
Summary
(Factors Fixing)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 14 / 27
28. Total Effect
Sum of the first- and higher-order effects of xi
Introduction
s
Global Sensitivity
Removal of the non-relevant parts
s
Analysis
Definition
Method
Variance-based
S1 + //2 + / 3 + /// + S12 + S13 + S14 + S23 + S24 + S34 +
S/ S // S4
Methods
Main Effect
Higher-Order Effects
S123 + S124 + S134 + S234 + S1234 = 1
Total Effect
Approach for SE
Summary
(Factors Fixing)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 14 / 27
29. Total Effect
Sum of the first- and higher-order effects of xi
Introduction
s
Global Sensitivity
Removal of the non-relevant parts
s
Analysis
Definition
Method
Variance-based
S1 + //2 + / 3 + /// + S12 + S13 + S14 + S23 + S24 + S34 +
S/ S // S4 // // //
// // //
Methods
Main Effect
Higher-Order Effects
S123 + S124 + S134 + S234 + S1234 = 1
Total Effect
Approach for SE
Summary
(Factors Fixing)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 14 / 27
30. Total Effect
Sum of the first- and higher-order effects of xi
Introduction
s
Global Sensitivity
Removal of the non-relevant parts
s
Analysis
Definition
Method
Variance-based
S1 + //2 + / 3 + /// + S12 + S13 + S14 + S23 + S24 + S34 +
S/ S // S4 // // //
// // //
Methods
Main Effect
Higher-Order Effects
S123 + S124 + S134 + / 234 + S1234 = 1
S //
// /
Total Effect
Approach for SE
Summary
(Factors Fixing)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 14 / 27
31. Total Effect
Sum of the first- and higher-order effects of xi
Introduction
s
Global Sensitivity
Removal of the non-relevant parts
s
Analysis
Definition
Method
Variance-based
S1 + //2 + / 3 + /// + S12 + S13 + S14 + S23 + S24 + S34 +
S/ S // S4 // // //
// // //
Methods
Main Effect
Higher-Order Effects
S123 + S124 + S134 + / 234 + S1234 = 1
S //
// /
Total Effect
Approach for SE
ST 1 = S1 + S12 + S13 + S14 + S123 + S124 + S134 + S1234
Summary
(Factors Fixing)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 14 / 27
32. Total Effect
Sum of the first- and higher-order effects of xi
Introduction
s
Global Sensitivity
Removal of the non-relevant parts
s
Analysis
Definition
Method
Variance-based
S1 + //2 + / 3 + /// + S12 + S13 + S14 + S23 + S24 + S34 +
S/ S // S4 // // //
// // //
Methods
Main Effect
Higher-Order Effects
S123 + S124 + S134 + / 234 + S1234 = 1
S //
// /
Total Effect
Approach for SE
ST 1 = S1 + S12 + S13 + S14 + S123 + S124 + S134 + S1234
Summary
ST 2 = S2 + S12 + S23 + S24 + S123 + S124 + S234 + S1234
(Factors Fixing)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 14 / 27
33. Introduction
Global Sensitivity
Analysis
Approach for SE
Overview of the
Approach
Determining
Distributions
Example:
Distributions
Approach for SE
Visualising Using
Scatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 15 / 27
34. Overview of the Approach
Introduction
Global
Global Sensitivity
Visualising
Determining
Analysis
sensitivity
using
distributions
Approach for SE
analyses
scatterplots
Overview of the
Approach
Determining
Distributions
Example:
Distributions
Visualising Using
Scatterplots .375
p1 and µ .045
d and µ .003
t and µ .002
inf and µ
Input Sensitivity
Applying Global SA
Scatterplots
distributions indices
Indizes
Summary
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 16 / 27
35. Determining Distributions
Distribution of the values of the input parameteters
Introduction
s
Global Sensitivity
Derived from
s
Analysis
Approach for SE
scientific literature
x
Overview of the
Approach
physical boundaries
x
Determining
expert opinion
x
Distributions
Example:
surveys
x
Distributions
Visualising Using
experiments
x
Scatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 17 / 27
36. Example: Distributions
Based on expert opinion
Introduction
s
Global Sensitivity
General parameter
s
Analysis
Approach for SE
p1 uniformly distributed between 0 and 1
x
Overview of the
Approach
d uniformly distributed between 0.9 and 1
x
Determining
Distributions
Dependent on application
s
Example:
Distributions
Visualising Using
inf e.g.. uniformly distributed between 50 and 500
x
Scatterplots
p1 and µ
For t e.g. t ∼ N (1000, 200)
x
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 18 / 27
37. Visualising Using Scatterplots
Sampling using the distributions
Introduction
s
Global Sensitivity
Pairwise relationship between factors
s
Analysis
Detection of errors in the model
s
Approach for SE
Overview of the
Indications of influences
s
Approach
Determining
Distributions
Example:
Distributions
Visualising Using
Scatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 19 / 27
38. p1 and µ
900
Introduction
Global Sensitivity
Analysis 800
Approach for SE
700
Overview of the
Approach
Determining 600
Distributions
Example:
µ 500
Distributions
Visualising Using
Scatterplots
400
p1 and µ
d and µ
300
t and µ
inf and µ
200
Applying Global SA
Indizes
100
Summary
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
p1
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 20 / 27
39. d and µ
900
Introduction
Global Sensitivity
Analysis 800
Approach for SE
700
Overview of the
Approach
Determining 600
Distributions
Example:
500
Distributions
µ
Visualising Using
Scatterplots
400
p1 and µ
d and µ
300
t and µ
inf and µ
200
Applying Global SA
Indizes
100
Summary
0
0.9 0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1
d
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 21 / 27
40. t and µ
900
Introduction
Global Sensitivity
800
Analysis
Approach for SE
700
Overview of the
Approach
600
Determining
Distributions
Example:
500
Distributions
µ
Visualising Using
400
Scatterplots
p1 and µ
300
d and µ
t and µ
200
inf and µ
Applying Global SA
100
Indizes
Summary
0
99200 99400 99600 99800 100000 100200 100400 100600 100800
t
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 22 / 27
41. inf and µ
900
Introduction
Global Sensitivity
Analysis 800
Approach for SE
700
Overview of the
Approach
Determining 600
Distributions
Example:
500
Distributions
µ
Visualising Using
Scatterplots
400
p1 and µ
d and µ
300
t and µ
inf and µ
200
Applying Global SA
Indizes
100
Summary
0
400 450 500 550 600 650 700 750 800 850 900
inf
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 23 / 27
42. Applying Global SA
Various methods for calculating indices
Introduction
s
Global Sensitivity
FAST (Fourier Amplitude Sensitivity Test) gives quantitative
s
Analysis
results
Approach for SE
Overview of the
Uses sampled data
s
Approach
Determining
Tool-support: Simlab
s
Distributions
Example:
Distributions
Visualising Using
Scatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 24 / 27
43. Indizes
Main Effect
Introduction
Global Sensitivity
Analysis
t 6.88e-5
Approach for SE
p1 0.0090
Overview of the
Approach
d 0.9026
Determining
Distributions
0.0158
inf
Example:
Distributions
Visualising Using
Scatterplots
p1 and µ
d and µ
t and µ
inf and µ
Applying Global SA
Indizes
Summary
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 25 / 27
44. Indizes
Main Effect
Introduction
Global Sensitivity
Analysis
t 6.88e-5
Approach for SE
p1 0.0090
Overview of the
Approach
d 0.9026
Determining
Distributions
0.0158
inf
Example:
Distributions
Visualising Using
Total Effect
Scatterplots
p1 and µ
d and µ
t 0.005788
t and µ
p1 0.022365
inf and µ
Applying Global SA
d 0.975155
Indizes
0.086735
inf
Summary
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 25 / 27
46. Conclusions
Sensitivity analysis useful tool for predictor models
Introduction
s
Global Sensitivity
SA can help to
s
Analysis
Approach for SE
find errors in the models
x
Summary
simplify the models
x
Conclusions
identify interactions between input parameters
x
identify parameters that should be investigated more
x
get more robust predictions
x
Experience with
s
reliability model
x
QA economics model
x
process model
x
expert system for IT tools
x
Good tool-support available (Simlab:
s
http://simlab.jrc.cec.eu.int/)
Dr. Stefan Wagner, TU M¨nchen
u PROMISE – May 20, 2007 – 27 / 27