SlideShare uma empresa Scribd logo
1 de 28
Baixar para ler offline
1 Het begint met een idee
Experiment design (basics)
Ivano Malavolta
Vrije Universiteit Amsterdam
2 Ivano Malavolta / S2 group / Empirical software engineering
Planning phases
Scope of this
lecture
Vrije Universiteit Amsterdam
Experiment design
Design principles
Basic design types
3 Ivano Malavolta / S2 group / Empirical software engineering
Roadmap
Vrije Universiteit Amsterdam
Goal: to determine the set of trials the experiment shall have to
make sure that the effect of the treatments is visible
4 Ivano Malavolta / S2 group / Empirical software engineering
Experiment design
Experiment design: how to organize and execute the trials
Vrije Universiteit Amsterdam
5
Example
● How do we combine subjects, objects
and treatments?
○ number of factors and treatments
Vrije Universiteit Amsterdam
6
What can you decide?
● Context?
● Fixed subject: software developer
● Factor: image encoding algorithm
○ other factors?
● Treatments:
○ JPEG
○ PNG
● Objects: mobile apps
○ one app vs many apps
● How to “cover” all the relevant
combinations of treatments, objects,
and subjects?
Vrije Universiteit Amsterdam
7 Ivano Malavolta / S2 group / Empirical software engineering
Design principles
● Randomization
Vrije Universiteit Amsterdam
8
Randomization
PROBLEM: when executing many trials, the chosen objects,
subjects and execution ordering may lead to biased results
SOLUTION: randomize the involved objects and subjects, and
the order in which trials are executed
Vrije Universiteit Amsterdam
9
Randomization
● Aim: remove the effects of a specific non-controlled
independent variable
● Group the subjects/objects by that variable and then
randomize
Examples:
● We randomly choose mobile apps from a dataset
● For each app we randomly assign a specific encoding algorithm for its
images
Vrije Universiteit Amsterdam
10 Ivano Malavolta / S2 group / Empirical software engineering
Design principles
● Randomization
● Blocking
Vrije Universiteit Amsterdam
11
Blocking
PROBLEM: some factors influence our results but we want to
mitigate their effects
SOLUTION: split the population in blocks with same (or similar)
level of this factor
Vrije Universiteit Amsterdam
12
Blocking
● The blocks are studied separately
● We do not study the effects between the groups
○ e.g. no conclusions on the correlation between the effects of image encoding
and the type of device
eg, Main factor: image encoding algorithm {PNG, JPEG}
Blocked factor: type of device {low-end, high-end}
Block 1: run the apps only in low-end devices
Block 2: run the apps only in high-end devices
Vrije Universiteit Amsterdam
13
Why more than 2 factors?
● {PNG, JPEG} → energy consumption
● {PNG, JPEG} and {single_render, par_render} → energy consumption
○ factor vs block?
Ivano Malavolta / S2 group / Experiment design
Vrije Universiteit Amsterdam
14
How to choose between factor or block?
Ivano Malavolta / S2 group / Experiment design
Vrije Universiteit Amsterdam
15 Ivano Malavolta / S2 group / Empirical software engineering
Design principles
● Randomization
● Blocking
● Balancing
Vrije Universiteit Amsterdam
16
Balancing
PROBLEM: many statistical analyses are more powerful and
simple when performed on balanced data
SOLUTION: consider the same (or similar) number of subjects
for each type of treatment
eg, Block 1: 20 apps
Block 2: 20 apps
Vrije Universiteit Amsterdam
17 Ivano Malavolta / S2 group / Empirical software engineering
Basic design types
Vrije Universiteit Amsterdam
18 Ivano Malavolta / S2 group / Empirical software engineering
Basic design types
We can have the following cases:
● 1 factor and 2 treatments (1F-2T)
● 1 factor and >2 treatments (1F-MT)
● 2 factors and 2 treatments (2F-2T)
● >2 factors, each one with >=2 treatments (MF-MT)
Basic
Vrije Universiteit Amsterdam
19 Ivano Malavolta / S2 group / Empirical software engineering
1 factor and 2 treatments
We assume to have 1 dependent variable P
Notation:
● i
: dependent variable mean for treatment i
○ i
= avg(P)
● yij
: j-th measure of the dependent variable for treatment i
Example:
● We are seeing whether different image encoding algorithms impact
energy consumption of mobile apps
● Factor: encoding algorithms
● Treatments:
○ PNG
○ JPEG
● Dependent variable: consumed energy during common usage scenarios
Vrije Universiteit Amsterdam
20 Ivano Malavolta / S2 group / Empirical software engineering
1F-2T: fully randomized design
● Each treatment is randomly assigned to the experimental objects
● Same number of objects per each treatment (balancing)
Object
(Application)
Treatment 1
(PNG)
Treatment 2
(JPEG)
1 X
2 X
3 X
4 X
Example of hypotheses:
H0
: 1
= 2
Ha
: 1
!= 2
or 1
> 2
or 1
< 2
Analyses:
● t-test (unpaired)
● Mann-Whitney test
Vrije Universiteit Amsterdam
21 Ivano Malavolta / S2 group / Empirical software engineering
1F-2T: paired comparison design
● Each treatment is applied on each object (crossover design)
● The order of the treatments is random
Object
(Application)
Treatment 1
(PNG)
Treatment 2
(JPEG)
1 1st 2nd
2 2nd 1st
3 2nd 1st
4 1st 2nd
Vrije Universiteit Amsterdam
22 Ivano Malavolta / S2 group / Empirical software engineering
1F-2T: paired comparison design
P
t
A1, JPEG
A1, PNG
tj
dj
d
= avg(d0...n
)
Vrije Universiteit Amsterdam
23 Ivano Malavolta / S2 group / Empirical software engineering
1F-2T: paired comparison design
Example of hypotheses:
H0
: d
= 0
Ha
: d
!= 0or d
> 0or d
< 0
Analyses:
● Paired t-test
● Sign test
● Wilcoxon
Vrije Universiteit Amsterdam
24
How to choose between the two design?
Object
(Application)
Treatment 1
(PNG)
Treatment 2
(JPEG)
1 1st 2nd
2 2nd 1st
3 2nd 1st
4 1st 2nd
Object
(Application)
Treatment 1
(PNG)
Treatment 2
(JPEG)
1 X
2 X
3 X
4 X
Vrije Universiteit Amsterdam
25 Ivano Malavolta / S2 group / Empirical software engineering
1 factor and >2 treatments
In this case the factor can have more than one value
Example:
● Factor: encoding algorithms
● Treatments:
○ PNG
○ JPEG
○ TIFF
● Dependent variable: consumed energy during common usage scenarios
Vrije Universiteit Amsterdam
26 Ivano Malavolta / S2 group / Empirical software engineering
1F-MT: fully randomized design
● Each treatment is randomly assigned to the experimental objects
● Same number of objects per each treatment (balancing)
Example of hypotheses:
H0
: 1
= 2
= 3
= … = t
Ha
: i
!= j
for at least on pair
of (i,j)
Analyses:
● ANOVA (ANalysis Of
VAriance)
● Kruskal-Wallis
Object
(Application)
Treatment 1
(PNG)
Treatment 2
(JPEG)
Treatment 2
(TIFF)
1 X
2 X
3 X
4 X
5 X
6 X
Vrije Universiteit Amsterdam
27 Ivano Malavolta / S2 group / Empirical software engineering
1F-MT: paired comparison design
● Each treatment is applied on each object (crossover design)
● The order of the treatments is random
Object
(Application)
Treatment 1
(PNG)
Treatment 2
(JPEG)
Treatment 2
(TIFF)
1 1st 3rd 2nd
2 3rd 1st 2nd
3 2nd 3rd 1st
4 2nd 1st 3rd
5 3rd 2nd 1st
6 1st 2nd 3rd
Example of hypotheses:
H0
: 1
= 2
= 3
= … = t
Ha
: i
!= j
for at least on pair
of (i,j)
Analyses:
● ANOVA
● Kruskal-Wallis
● Repeated Measures ANOVA
Vrije Universiteit Amsterdam
28 Ivano Malavolta / S2 group / Empirical software engineering
Some contents of lecture extracted from:
● Giuseppe Procaccianti’s lectures at VU
Acknowledgements

Mais conteúdo relacionado

Mais procurados

[02-A] The experimental process
[02-A] The experimental process[02-A] The experimental process
[02-A] The experimental processIvano Malavolta
 
The Green Lab - [03 A] Experiment planning
The Green Lab - [03 A] Experiment planningThe Green Lab - [03 A] Experiment planning
The Green Lab - [03 A] Experiment planningIvano Malavolta
 
[02-B] Experiment scoping
[02-B] Experiment scoping[02-B] Experiment scoping
[02-B] Experiment scopingIvano Malavolta
 
The Green Lab - [09 B] Experiment validity
The Green Lab - [09  B] Experiment validityThe Green Lab - [09  B] Experiment validity
The Green Lab - [09 B] Experiment validityIvano Malavolta
 
The Green Lab - [02 A] The experimental process
The Green Lab - [02 A] The experimental processThe Green Lab - [02 A] The experimental process
The Green Lab - [02 A] The experimental processIvano Malavolta
 
The Green Lab - [07-A] Data Analysis
The Green Lab - [07-A] Data AnalysisThe Green Lab - [07-A] Data Analysis
The Green Lab - [07-A] Data AnalysisGiuseppe Procaccianti
 
[13 - B] Experiment reporting
[13 - B] Experiment reporting[13 - B] Experiment reporting
[13 - B] Experiment reportingIvano Malavolta
 
[03-B] Measurement theory basics
[03-B] Measurement theory basics[03-B] Measurement theory basics
[03-B] Measurement theory basicsIvano Malavolta
 
[03-A] Experiment planning
[03-A] Experiment planning[03-A] Experiment planning
[03-A] Experiment planningIvano Malavolta
 
The Green Lab - [13 B] Future research challenges
The Green Lab - [13 B] Future research challengesThe Green Lab - [13 B] Future research challenges
The Green Lab - [13 B] Future research challengesIvano Malavolta
 
The Green Lab - [12-A] Data visualization in R
The Green Lab - [12-A] Data visualization in RThe Green Lab - [12-A] Data visualization in R
The Green Lab - [12-A] Data visualization in RGiuseppe Procaccianti
 
[09-A] Statistical tests and effect size
[09-A] Statistical tests and effect size[09-A] Statistical tests and effect size
[09-A] Statistical tests and effect sizeIvano Malavolta
 
[13 - A] Experiment validity
[13 - A] Experiment validity[13 - A] Experiment validity
[13 - A] Experiment validityIvano Malavolta
 
[01-B] Empirical software engineering
[01-B] Empirical software engineering[01-B] Empirical software engineering
[01-B] Empirical software engineeringIvano Malavolta
 
[07-A] Descriptive Statistics and data exploration
[07-A] Descriptive Statistics and data exploration[07-A] Descriptive Statistics and data exploration
[07-A] Descriptive Statistics and data explorationIvano Malavolta
 
[07-B] Statistical hypothesis testing
[07-B] Statistical hypothesis testing[07-B] Statistical hypothesis testing
[07-B] Statistical hypothesis testingIvano Malavolta
 
The Green Lab - [04-A] Lab environment and tools
The Green Lab - [04-A] Lab environment and toolsThe Green Lab - [04-A] Lab environment and tools
The Green Lab - [04-A] Lab environment and toolsGiuseppe Procaccianti
 
Collaborative Model-Driven Software Engineering: a Classification Framework a...
Collaborative Model-Driven Software Engineering: a Classification Framework a...Collaborative Model-Driven Software Engineering: a Classification Framework a...
Collaborative Model-Driven Software Engineering: a Classification Framework a...Ivano Malavolta
 

Mais procurados (20)

[02-A] The experimental process
[02-A] The experimental process[02-A] The experimental process
[02-A] The experimental process
 
The Green Lab - [03 A] Experiment planning
The Green Lab - [03 A] Experiment planningThe Green Lab - [03 A] Experiment planning
The Green Lab - [03 A] Experiment planning
 
[02-B] Experiment scoping
[02-B] Experiment scoping[02-B] Experiment scoping
[02-B] Experiment scoping
 
The Green Lab - [09 B] Experiment validity
The Green Lab - [09  B] Experiment validityThe Green Lab - [09  B] Experiment validity
The Green Lab - [09 B] Experiment validity
 
The Green Lab - [02 A] The experimental process
The Green Lab - [02 A] The experimental processThe Green Lab - [02 A] The experimental process
The Green Lab - [02 A] The experimental process
 
The Green Lab - [07-A] Data Analysis
The Green Lab - [07-A] Data AnalysisThe Green Lab - [07-A] Data Analysis
The Green Lab - [07-A] Data Analysis
 
[13 - B] Experiment reporting
[13 - B] Experiment reporting[13 - B] Experiment reporting
[13 - B] Experiment reporting
 
[03-B] Measurement theory basics
[03-B] Measurement theory basics[03-B] Measurement theory basics
[03-B] Measurement theory basics
 
[03-A] Experiment planning
[03-A] Experiment planning[03-A] Experiment planning
[03-A] Experiment planning
 
The Green Lab - [13 B] Future research challenges
The Green Lab - [13 B] Future research challengesThe Green Lab - [13 B] Future research challenges
The Green Lab - [13 B] Future research challenges
 
The Green Lab - [12-A] Data visualization in R
The Green Lab - [12-A] Data visualization in RThe Green Lab - [12-A] Data visualization in R
The Green Lab - [12-A] Data visualization in R
 
[09-A] Statistical tests and effect size
[09-A] Statistical tests and effect size[09-A] Statistical tests and effect size
[09-A] Statistical tests and effect size
 
[13 - A] Experiment validity
[13 - A] Experiment validity[13 - A] Experiment validity
[13 - A] Experiment validity
 
[01-B] Empirical software engineering
[01-B] Empirical software engineering[01-B] Empirical software engineering
[01-B] Empirical software engineering
 
[07-A] Descriptive Statistics and data exploration
[07-A] Descriptive Statistics and data exploration[07-A] Descriptive Statistics and data exploration
[07-A] Descriptive Statistics and data exploration
 
[07-B] Statistical hypothesis testing
[07-B] Statistical hypothesis testing[07-B] Statistical hypothesis testing
[07-B] Statistical hypothesis testing
 
The Green Lab - [04-A] Lab environment and tools
The Green Lab - [04-A] Lab environment and toolsThe Green Lab - [04-A] Lab environment and tools
The Green Lab - [04-A] Lab environment and tools
 
Resume_xuezhi
Resume_xuezhiResume_xuezhi
Resume_xuezhi
 
Collaborative Model-Driven Software Engineering: a Classification Framework a...
Collaborative Model-Driven Software Engineering: a Classification Framework a...Collaborative Model-Driven Software Engineering: a Classification Framework a...
Collaborative Model-Driven Software Engineering: a Classification Framework a...
 
Uses of accelerometer sensor and its application in m-Learning environments: ...
Uses of accelerometer sensor and its application in m-Learning environments: ...Uses of accelerometer sensor and its application in m-Learning environments: ...
Uses of accelerometer sensor and its application in m-Learning environments: ...
 

Destaque

The Green Lab - [04 B] [PWA] Experiment setup
The Green Lab - [04 B] [PWA] Experiment setupThe Green Lab - [04 B] [PWA] Experiment setup
The Green Lab - [04 B] [PWA] Experiment setupIvano Malavolta
 
The Green Lab - [02 C] [case study] Progressive web apps
The Green Lab - [02 C] [case study] Progressive web appsThe Green Lab - [02 C] [case study] Progressive web apps
The Green Lab - [02 C] [case study] Progressive web appsIvano Malavolta
 
The Green Lab - [07-B] Hypothesis Testing
The Green Lab - [07-B] Hypothesis TestingThe Green Lab - [07-B] Hypothesis Testing
The Green Lab - [07-B] Hypothesis TestingGiuseppe Procaccianti
 
The Green Lab - [01-B] Case study presentation
The Green Lab - [01-B] Case study presentationThe Green Lab - [01-B] Case study presentation
The Green Lab - [01-B] Case study presentationGiuseppe Procaccianti
 
Beyond Native Apps: Web Technologies to the Rescue! [SPLASH 2016 - Mobile! k...
Beyond Native Apps:  Web Technologies to the Rescue! [SPLASH 2016 - Mobile! k...Beyond Native Apps:  Web Technologies to the Rescue! [SPLASH 2016 - Mobile! k...
Beyond Native Apps: Web Technologies to the Rescue! [SPLASH 2016 - Mobile! k...Ivano Malavolta
 
Experiment Design - strategy and markets determine what you should test
Experiment Design - strategy and markets determine what you should testExperiment Design - strategy and markets determine what you should test
Experiment Design - strategy and markets determine what you should testFirmhouse
 
CHIP Project: Personalized Museum Tour with Real-Time Adaptation on a Mobile ...
CHIP Project: Personalized Museum Tour with Real-Time Adaptation on a Mobile ...CHIP Project: Personalized Museum Tour with Real-Time Adaptation on a Mobile ...
CHIP Project: Personalized Museum Tour with Real-Time Adaptation on a Mobile ...Lora Aroyo
 
WebSci2013 Harnessing Disagreement in Crowdsourcing
WebSci2013 Harnessing Disagreement in CrowdsourcingWebSci2013 Harnessing Disagreement in Crowdsourcing
WebSci2013 Harnessing Disagreement in CrowdsourcingLora Aroyo
 
PiLOD talk: Dutch Ships and Sailors
PiLOD talk: Dutch Ships and Sailors PiLOD talk: Dutch Ships and Sailors
PiLOD talk: Dutch Ships and Sailors Victor de Boer
 
Talk of Europe – Linking European Parliament Proceedings
Talk of Europe – Linking European Parliament ProceedingsTalk of Europe – Linking European Parliament Proceedings
Talk of Europe – Linking European Parliament ProceedingsAstrid van Aggelen
 
Agora User Committee Meeting 2013
Agora User Committee Meeting 2013Agora User Committee Meeting 2013
Agora User Committee Meeting 2013Lora Aroyo
 
Dive exploring history presentation
Dive exploring history presentationDive exploring history presentation
Dive exploring history presentationVictor de Boer
 
SealincMedia Accurator Demos
SealincMedia Accurator DemosSealincMedia Accurator Demos
SealincMedia Accurator DemosLora Aroyo
 
Future TV is Now: Personalized & Social
Future TV is Now: Personalized & SocialFuture TV is Now: Personalized & Social
Future TV is Now: Personalized & SocialLora Aroyo
 

Destaque (14)

The Green Lab - [04 B] [PWA] Experiment setup
The Green Lab - [04 B] [PWA] Experiment setupThe Green Lab - [04 B] [PWA] Experiment setup
The Green Lab - [04 B] [PWA] Experiment setup
 
The Green Lab - [02 C] [case study] Progressive web apps
The Green Lab - [02 C] [case study] Progressive web appsThe Green Lab - [02 C] [case study] Progressive web apps
The Green Lab - [02 C] [case study] Progressive web apps
 
The Green Lab - [07-B] Hypothesis Testing
The Green Lab - [07-B] Hypothesis TestingThe Green Lab - [07-B] Hypothesis Testing
The Green Lab - [07-B] Hypothesis Testing
 
The Green Lab - [01-B] Case study presentation
The Green Lab - [01-B] Case study presentationThe Green Lab - [01-B] Case study presentation
The Green Lab - [01-B] Case study presentation
 
Beyond Native Apps: Web Technologies to the Rescue! [SPLASH 2016 - Mobile! k...
Beyond Native Apps:  Web Technologies to the Rescue! [SPLASH 2016 - Mobile! k...Beyond Native Apps:  Web Technologies to the Rescue! [SPLASH 2016 - Mobile! k...
Beyond Native Apps: Web Technologies to the Rescue! [SPLASH 2016 - Mobile! k...
 
Experiment Design - strategy and markets determine what you should test
Experiment Design - strategy and markets determine what you should testExperiment Design - strategy and markets determine what you should test
Experiment Design - strategy and markets determine what you should test
 
CHIP Project: Personalized Museum Tour with Real-Time Adaptation on a Mobile ...
CHIP Project: Personalized Museum Tour with Real-Time Adaptation on a Mobile ...CHIP Project: Personalized Museum Tour with Real-Time Adaptation on a Mobile ...
CHIP Project: Personalized Museum Tour with Real-Time Adaptation on a Mobile ...
 
WebSci2013 Harnessing Disagreement in Crowdsourcing
WebSci2013 Harnessing Disagreement in CrowdsourcingWebSci2013 Harnessing Disagreement in Crowdsourcing
WebSci2013 Harnessing Disagreement in Crowdsourcing
 
PiLOD talk: Dutch Ships and Sailors
PiLOD talk: Dutch Ships and Sailors PiLOD talk: Dutch Ships and Sailors
PiLOD talk: Dutch Ships and Sailors
 
Talk of Europe – Linking European Parliament Proceedings
Talk of Europe – Linking European Parliament ProceedingsTalk of Europe – Linking European Parliament Proceedings
Talk of Europe – Linking European Parliament Proceedings
 
Agora User Committee Meeting 2013
Agora User Committee Meeting 2013Agora User Committee Meeting 2013
Agora User Committee Meeting 2013
 
Dive exploring history presentation
Dive exploring history presentationDive exploring history presentation
Dive exploring history presentation
 
SealincMedia Accurator Demos
SealincMedia Accurator DemosSealincMedia Accurator Demos
SealincMedia Accurator Demos
 
Future TV is Now: Personalized & Social
Future TV is Now: Personalized & SocialFuture TV is Now: Personalized & Social
Future TV is Now: Personalized & Social
 

Semelhante a The Green Lab - [05 A] Experiment design (basics)

Adapting neural networks for the estimation of treatment effects
Adapting neural networks for the estimation of treatment effectsAdapting neural networks for the estimation of treatment effects
Adapting neural networks for the estimation of treatment effectsViswanath Gangavaram
 
Planning for power systems
Planning for power systemsPlanning for power systems
Planning for power systemsOlivier Teytaud
 
Dynamic Optimization without Markov Assumptions: application to power systems
Dynamic Optimization without Markov Assumptions: application to power systemsDynamic Optimization without Markov Assumptions: application to power systems
Dynamic Optimization without Markov Assumptions: application to power systemsOlivier Teytaud
 
An Application of Assignment Problem in Laptop Selection Problem Using MATLAB
An Application of Assignment Problem in Laptop Selection Problem Using MATLAB An Application of Assignment Problem in Laptop Selection Problem Using MATLAB
An Application of Assignment Problem in Laptop Selection Problem Using MATLAB mathsjournal
 
An Application of Assignment Problem in Laptop Selection Problem Using MATLAB
An Application of Assignment Problem in Laptop Selection Problem Using MATLABAn Application of Assignment Problem in Laptop Selection Problem Using MATLAB
An Application of Assignment Problem in Laptop Selection Problem Using MATLABmathsjournal
 
Ilab Metis: we optimize power systems and we are not afraid of direct policy ...
Ilab Metis: we optimize power systems and we are not afraid of direct policy ...Ilab Metis: we optimize power systems and we are not afraid of direct policy ...
Ilab Metis: we optimize power systems and we are not afraid of direct policy ...Olivier Teytaud
 
DC02. Interpretation of predictions
DC02. Interpretation of predictionsDC02. Interpretation of predictions
DC02. Interpretation of predictionsAnton Kulesh
 
An Application Of Assignment Problem In Laptop Selection Problem Using Matlab
An Application Of Assignment Problem In Laptop Selection Problem Using MatlabAn Application Of Assignment Problem In Laptop Selection Problem Using Matlab
An Application Of Assignment Problem In Laptop Selection Problem Using MatlabBrandi Gonzales
 
Self-sampling Strategies for Multimemetic Algorithms in Unstable Computationa...
Self-sampling Strategies for Multimemetic Algorithms in Unstable Computationa...Self-sampling Strategies for Multimemetic Algorithms in Unstable Computationa...
Self-sampling Strategies for Multimemetic Algorithms in Unstable Computationa...Rafael Nogueras
 
Optimization Problems Solved by Different Platforms Say Optimum Tool Box (Mat...
Optimization Problems Solved by Different Platforms Say Optimum Tool Box (Mat...Optimization Problems Solved by Different Platforms Say Optimum Tool Box (Mat...
Optimization Problems Solved by Different Platforms Say Optimum Tool Box (Mat...IRJET Journal
 
Jay Yagnik at AI Frontiers : A History Lesson on AI
Jay Yagnik at AI Frontiers : A History Lesson on AIJay Yagnik at AI Frontiers : A History Lesson on AI
Jay Yagnik at AI Frontiers : A History Lesson on AIAI Frontiers
 
Uncertainty Awareness in Integrating Machine Learning and Game Theory
Uncertainty Awareness in Integrating Machine Learning and Game TheoryUncertainty Awareness in Integrating Machine Learning and Game Theory
Uncertainty Awareness in Integrating Machine Learning and Game TheoryRikiya Takahashi
 
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15MLconf
 
10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConfXavier Amatriain
 
10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systemsXavier Amatriain
 
A review of automatic differentiationand its efficient implementation
A review of automatic differentiationand its efficient implementationA review of automatic differentiationand its efficient implementation
A review of automatic differentiationand its efficient implementationssuserfa7e73
 
Universal job embedding in recommendation (public ver.)
Universal job embedding in recommendation (public ver.)Universal job embedding in recommendation (public ver.)
Universal job embedding in recommendation (public ver.)Marsan Ma
 

Semelhante a The Green Lab - [05 A] Experiment design (basics) (20)

Adapting neural networks for the estimation of treatment effects
Adapting neural networks for the estimation of treatment effectsAdapting neural networks for the estimation of treatment effects
Adapting neural networks for the estimation of treatment effects
 
Planning for power systems
Planning for power systemsPlanning for power systems
Planning for power systems
 
Dynamic Optimization without Markov Assumptions: application to power systems
Dynamic Optimization without Markov Assumptions: application to power systemsDynamic Optimization without Markov Assumptions: application to power systems
Dynamic Optimization without Markov Assumptions: application to power systems
 
An Application of Assignment Problem in Laptop Selection Problem Using MATLAB
An Application of Assignment Problem in Laptop Selection Problem Using MATLAB An Application of Assignment Problem in Laptop Selection Problem Using MATLAB
An Application of Assignment Problem in Laptop Selection Problem Using MATLAB
 
An Application of Assignment Problem in Laptop Selection Problem Using MATLAB
An Application of Assignment Problem in Laptop Selection Problem Using MATLABAn Application of Assignment Problem in Laptop Selection Problem Using MATLAB
An Application of Assignment Problem in Laptop Selection Problem Using MATLAB
 
Ilab Metis: we optimize power systems and we are not afraid of direct policy ...
Ilab Metis: we optimize power systems and we are not afraid of direct policy ...Ilab Metis: we optimize power systems and we are not afraid of direct policy ...
Ilab Metis: we optimize power systems and we are not afraid of direct policy ...
 
DC02. Interpretation of predictions
DC02. Interpretation of predictionsDC02. Interpretation of predictions
DC02. Interpretation of predictions
 
An Application Of Assignment Problem In Laptop Selection Problem Using Matlab
An Application Of Assignment Problem In Laptop Selection Problem Using MatlabAn Application Of Assignment Problem In Laptop Selection Problem Using Matlab
An Application Of Assignment Problem In Laptop Selection Problem Using Matlab
 
presentationIDC - 14MAY2015
presentationIDC - 14MAY2015presentationIDC - 14MAY2015
presentationIDC - 14MAY2015
 
Chapter 2 ds
Chapter 2 dsChapter 2 ds
Chapter 2 ds
 
Self-sampling Strategies for Multimemetic Algorithms in Unstable Computationa...
Self-sampling Strategies for Multimemetic Algorithms in Unstable Computationa...Self-sampling Strategies for Multimemetic Algorithms in Unstable Computationa...
Self-sampling Strategies for Multimemetic Algorithms in Unstable Computationa...
 
Optimization Problems Solved by Different Platforms Say Optimum Tool Box (Mat...
Optimization Problems Solved by Different Platforms Say Optimum Tool Box (Mat...Optimization Problems Solved by Different Platforms Say Optimum Tool Box (Mat...
Optimization Problems Solved by Different Platforms Say Optimum Tool Box (Mat...
 
Lecture 1.pptx
Lecture 1.pptxLecture 1.pptx
Lecture 1.pptx
 
Jay Yagnik at AI Frontiers : A History Lesson on AI
Jay Yagnik at AI Frontiers : A History Lesson on AIJay Yagnik at AI Frontiers : A History Lesson on AI
Jay Yagnik at AI Frontiers : A History Lesson on AI
 
Uncertainty Awareness in Integrating Machine Learning and Game Theory
Uncertainty Awareness in Integrating Machine Learning and Game TheoryUncertainty Awareness in Integrating Machine Learning and Game Theory
Uncertainty Awareness in Integrating Machine Learning and Game Theory
 
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
Xavier Amatriain, VP of Engineering, Quora at MLconf SF - 11/13/15
 
10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf
 
10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems
 
A review of automatic differentiationand its efficient implementation
A review of automatic differentiationand its efficient implementationA review of automatic differentiationand its efficient implementation
A review of automatic differentiationand its efficient implementation
 
Universal job embedding in recommendation (public ver.)
Universal job embedding in recommendation (public ver.)Universal job embedding in recommendation (public ver.)
Universal job embedding in recommendation (public ver.)
 

Mais de Ivano Malavolta

Conducting Experiments on the Software Architecture of Robotic Systems (QRARS...
Conducting Experiments on the Software Architecture of Robotic Systems (QRARS...Conducting Experiments on the Software Architecture of Robotic Systems (QRARS...
Conducting Experiments on the Software Architecture of Robotic Systems (QRARS...Ivano Malavolta
 
The Green Lab - Research cocktail @Vrije Universiteit Amsterdam (October 2020)
The Green Lab - Research cocktail  @Vrije Universiteit Amsterdam (October 2020)The Green Lab - Research cocktail  @Vrije Universiteit Amsterdam (October 2020)
The Green Lab - Research cocktail @Vrije Universiteit Amsterdam (October 2020)Ivano Malavolta
 
Software sustainability and Green IT
Software sustainability and Green ITSoftware sustainability and Green IT
Software sustainability and Green ITIvano Malavolta
 
Navigation-aware and Personalized Prefetching of Network Requests in Android ...
Navigation-aware and Personalized Prefetching of Network Requests in Android ...Navigation-aware and Personalized Prefetching of Network Requests in Android ...
Navigation-aware and Personalized Prefetching of Network Requests in Android ...Ivano Malavolta
 
How Maintainability Issues of Android Apps Evolve [ICSME 2018]
How Maintainability Issues of Android Apps Evolve [ICSME 2018]How Maintainability Issues of Android Apps Evolve [ICSME 2018]
How Maintainability Issues of Android Apps Evolve [ICSME 2018]Ivano Malavolta
 
Experimenting on Mobile Apps Quality - a tale about Energy, Performance, and ...
Experimenting on Mobile Apps Quality - a tale about Energy, Performance, and ...Experimenting on Mobile Apps Quality - a tale about Energy, Performance, and ...
Experimenting on Mobile Apps Quality - a tale about Energy, Performance, and ...Ivano Malavolta
 
Modeling objects interaction via UML sequence diagrams [Software Design] [Com...
Modeling objects interaction via UML sequence diagrams [Software Design] [Com...Modeling objects interaction via UML sequence diagrams [Software Design] [Com...
Modeling objects interaction via UML sequence diagrams [Software Design] [Com...Ivano Malavolta
 
Modeling behaviour via UML state machines [Software Design] [Computer Science...
Modeling behaviour via UML state machines [Software Design] [Computer Science...Modeling behaviour via UML state machines [Software Design] [Computer Science...
Modeling behaviour via UML state machines [Software Design] [Computer Science...Ivano Malavolta
 
Object-oriented design patterns in UML [Software Design] [Computer Science] [...
Object-oriented design patterns in UML [Software Design] [Computer Science] [...Object-oriented design patterns in UML [Software Design] [Computer Science] [...
Object-oriented design patterns in UML [Software Design] [Computer Science] [...Ivano Malavolta
 
Structure modeling with UML [Software Design] [Computer Science] [Vrije Unive...
Structure modeling with UML [Software Design] [Computer Science] [Vrije Unive...Structure modeling with UML [Software Design] [Computer Science] [Vrije Unive...
Structure modeling with UML [Software Design] [Computer Science] [Vrije Unive...Ivano Malavolta
 
Requirements engineering with UML [Software Design] [Computer Science] [Vrije...
Requirements engineering with UML [Software Design] [Computer Science] [Vrije...Requirements engineering with UML [Software Design] [Computer Science] [Vrije...
Requirements engineering with UML [Software Design] [Computer Science] [Vrije...Ivano Malavolta
 
Modeling and abstraction, software development process [Software Design] [Com...
Modeling and abstraction, software development process [Software Design] [Com...Modeling and abstraction, software development process [Software Design] [Com...
Modeling and abstraction, software development process [Software Design] [Com...Ivano Malavolta
 
[2017/2018] Agile development
[2017/2018] Agile development[2017/2018] Agile development
[2017/2018] Agile developmentIvano Malavolta
 
Reconstructing microservice-based architectures
Reconstructing microservice-based architecturesReconstructing microservice-based architectures
Reconstructing microservice-based architecturesIvano Malavolta
 
[2017/2018] AADL - Architecture Analysis and Design Language
[2017/2018] AADL - Architecture Analysis and Design Language[2017/2018] AADL - Architecture Analysis and Design Language
[2017/2018] AADL - Architecture Analysis and Design LanguageIvano Malavolta
 
[2017/2018] Architectural languages
[2017/2018] Architectural languages[2017/2018] Architectural languages
[2017/2018] Architectural languagesIvano Malavolta
 
[2017/2018] Introduction to Software Architecture
[2017/2018] Introduction to Software Architecture[2017/2018] Introduction to Software Architecture
[2017/2018] Introduction to Software ArchitectureIvano Malavolta
 
[2017/2018] RESEARCH in software engineering
[2017/2018] RESEARCH in software engineering[2017/2018] RESEARCH in software engineering
[2017/2018] RESEARCH in software engineeringIvano Malavolta
 
Mobile Apps quality - a tale about energy, performance, and users’ perception
Mobile Apps quality - a tale about energy, performance, and users’ perceptionMobile Apps quality - a tale about energy, performance, and users’ perception
Mobile Apps quality - a tale about energy, performance, and users’ perceptionIvano Malavolta
 

Mais de Ivano Malavolta (20)

Conducting Experiments on the Software Architecture of Robotic Systems (QRARS...
Conducting Experiments on the Software Architecture of Robotic Systems (QRARS...Conducting Experiments on the Software Architecture of Robotic Systems (QRARS...
Conducting Experiments on the Software Architecture of Robotic Systems (QRARS...
 
The H2020 experience
The H2020 experienceThe H2020 experience
The H2020 experience
 
The Green Lab - Research cocktail @Vrije Universiteit Amsterdam (October 2020)
The Green Lab - Research cocktail  @Vrije Universiteit Amsterdam (October 2020)The Green Lab - Research cocktail  @Vrije Universiteit Amsterdam (October 2020)
The Green Lab - Research cocktail @Vrije Universiteit Amsterdam (October 2020)
 
Software sustainability and Green IT
Software sustainability and Green ITSoftware sustainability and Green IT
Software sustainability and Green IT
 
Navigation-aware and Personalized Prefetching of Network Requests in Android ...
Navigation-aware and Personalized Prefetching of Network Requests in Android ...Navigation-aware and Personalized Prefetching of Network Requests in Android ...
Navigation-aware and Personalized Prefetching of Network Requests in Android ...
 
How Maintainability Issues of Android Apps Evolve [ICSME 2018]
How Maintainability Issues of Android Apps Evolve [ICSME 2018]How Maintainability Issues of Android Apps Evolve [ICSME 2018]
How Maintainability Issues of Android Apps Evolve [ICSME 2018]
 
Experimenting on Mobile Apps Quality - a tale about Energy, Performance, and ...
Experimenting on Mobile Apps Quality - a tale about Energy, Performance, and ...Experimenting on Mobile Apps Quality - a tale about Energy, Performance, and ...
Experimenting on Mobile Apps Quality - a tale about Energy, Performance, and ...
 
Modeling objects interaction via UML sequence diagrams [Software Design] [Com...
Modeling objects interaction via UML sequence diagrams [Software Design] [Com...Modeling objects interaction via UML sequence diagrams [Software Design] [Com...
Modeling objects interaction via UML sequence diagrams [Software Design] [Com...
 
Modeling behaviour via UML state machines [Software Design] [Computer Science...
Modeling behaviour via UML state machines [Software Design] [Computer Science...Modeling behaviour via UML state machines [Software Design] [Computer Science...
Modeling behaviour via UML state machines [Software Design] [Computer Science...
 
Object-oriented design patterns in UML [Software Design] [Computer Science] [...
Object-oriented design patterns in UML [Software Design] [Computer Science] [...Object-oriented design patterns in UML [Software Design] [Computer Science] [...
Object-oriented design patterns in UML [Software Design] [Computer Science] [...
 
Structure modeling with UML [Software Design] [Computer Science] [Vrije Unive...
Structure modeling with UML [Software Design] [Computer Science] [Vrije Unive...Structure modeling with UML [Software Design] [Computer Science] [Vrije Unive...
Structure modeling with UML [Software Design] [Computer Science] [Vrije Unive...
 
Requirements engineering with UML [Software Design] [Computer Science] [Vrije...
Requirements engineering with UML [Software Design] [Computer Science] [Vrije...Requirements engineering with UML [Software Design] [Computer Science] [Vrije...
Requirements engineering with UML [Software Design] [Computer Science] [Vrije...
 
Modeling and abstraction, software development process [Software Design] [Com...
Modeling and abstraction, software development process [Software Design] [Com...Modeling and abstraction, software development process [Software Design] [Com...
Modeling and abstraction, software development process [Software Design] [Com...
 
[2017/2018] Agile development
[2017/2018] Agile development[2017/2018] Agile development
[2017/2018] Agile development
 
Reconstructing microservice-based architectures
Reconstructing microservice-based architecturesReconstructing microservice-based architectures
Reconstructing microservice-based architectures
 
[2017/2018] AADL - Architecture Analysis and Design Language
[2017/2018] AADL - Architecture Analysis and Design Language[2017/2018] AADL - Architecture Analysis and Design Language
[2017/2018] AADL - Architecture Analysis and Design Language
 
[2017/2018] Architectural languages
[2017/2018] Architectural languages[2017/2018] Architectural languages
[2017/2018] Architectural languages
 
[2017/2018] Introduction to Software Architecture
[2017/2018] Introduction to Software Architecture[2017/2018] Introduction to Software Architecture
[2017/2018] Introduction to Software Architecture
 
[2017/2018] RESEARCH in software engineering
[2017/2018] RESEARCH in software engineering[2017/2018] RESEARCH in software engineering
[2017/2018] RESEARCH in software engineering
 
Mobile Apps quality - a tale about energy, performance, and users’ perception
Mobile Apps quality - a tale about energy, performance, and users’ perceptionMobile Apps quality - a tale about energy, performance, and users’ perception
Mobile Apps quality - a tale about energy, performance, and users’ perception
 

Último

Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 

Último (20)

Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 

The Green Lab - [05 A] Experiment design (basics)

  • 1. 1 Het begint met een idee Experiment design (basics) Ivano Malavolta
  • 2. Vrije Universiteit Amsterdam 2 Ivano Malavolta / S2 group / Empirical software engineering Planning phases Scope of this lecture
  • 3. Vrije Universiteit Amsterdam Experiment design Design principles Basic design types 3 Ivano Malavolta / S2 group / Empirical software engineering Roadmap
  • 4. Vrije Universiteit Amsterdam Goal: to determine the set of trials the experiment shall have to make sure that the effect of the treatments is visible 4 Ivano Malavolta / S2 group / Empirical software engineering Experiment design Experiment design: how to organize and execute the trials
  • 5. Vrije Universiteit Amsterdam 5 Example ● How do we combine subjects, objects and treatments? ○ number of factors and treatments
  • 6. Vrije Universiteit Amsterdam 6 What can you decide? ● Context? ● Fixed subject: software developer ● Factor: image encoding algorithm ○ other factors? ● Treatments: ○ JPEG ○ PNG ● Objects: mobile apps ○ one app vs many apps ● How to “cover” all the relevant combinations of treatments, objects, and subjects?
  • 7. Vrije Universiteit Amsterdam 7 Ivano Malavolta / S2 group / Empirical software engineering Design principles ● Randomization
  • 8. Vrije Universiteit Amsterdam 8 Randomization PROBLEM: when executing many trials, the chosen objects, subjects and execution ordering may lead to biased results SOLUTION: randomize the involved objects and subjects, and the order in which trials are executed
  • 9. Vrije Universiteit Amsterdam 9 Randomization ● Aim: remove the effects of a specific non-controlled independent variable ● Group the subjects/objects by that variable and then randomize Examples: ● We randomly choose mobile apps from a dataset ● For each app we randomly assign a specific encoding algorithm for its images
  • 10. Vrije Universiteit Amsterdam 10 Ivano Malavolta / S2 group / Empirical software engineering Design principles ● Randomization ● Blocking
  • 11. Vrije Universiteit Amsterdam 11 Blocking PROBLEM: some factors influence our results but we want to mitigate their effects SOLUTION: split the population in blocks with same (or similar) level of this factor
  • 12. Vrije Universiteit Amsterdam 12 Blocking ● The blocks are studied separately ● We do not study the effects between the groups ○ e.g. no conclusions on the correlation between the effects of image encoding and the type of device eg, Main factor: image encoding algorithm {PNG, JPEG} Blocked factor: type of device {low-end, high-end} Block 1: run the apps only in low-end devices Block 2: run the apps only in high-end devices
  • 13. Vrije Universiteit Amsterdam 13 Why more than 2 factors? ● {PNG, JPEG} → energy consumption ● {PNG, JPEG} and {single_render, par_render} → energy consumption ○ factor vs block? Ivano Malavolta / S2 group / Experiment design
  • 14. Vrije Universiteit Amsterdam 14 How to choose between factor or block? Ivano Malavolta / S2 group / Experiment design
  • 15. Vrije Universiteit Amsterdam 15 Ivano Malavolta / S2 group / Empirical software engineering Design principles ● Randomization ● Blocking ● Balancing
  • 16. Vrije Universiteit Amsterdam 16 Balancing PROBLEM: many statistical analyses are more powerful and simple when performed on balanced data SOLUTION: consider the same (or similar) number of subjects for each type of treatment eg, Block 1: 20 apps Block 2: 20 apps
  • 17. Vrije Universiteit Amsterdam 17 Ivano Malavolta / S2 group / Empirical software engineering Basic design types
  • 18. Vrije Universiteit Amsterdam 18 Ivano Malavolta / S2 group / Empirical software engineering Basic design types We can have the following cases: ● 1 factor and 2 treatments (1F-2T) ● 1 factor and >2 treatments (1F-MT) ● 2 factors and 2 treatments (2F-2T) ● >2 factors, each one with >=2 treatments (MF-MT) Basic
  • 19. Vrije Universiteit Amsterdam 19 Ivano Malavolta / S2 group / Empirical software engineering 1 factor and 2 treatments We assume to have 1 dependent variable P Notation: ● i : dependent variable mean for treatment i ○ i = avg(P) ● yij : j-th measure of the dependent variable for treatment i Example: ● We are seeing whether different image encoding algorithms impact energy consumption of mobile apps ● Factor: encoding algorithms ● Treatments: ○ PNG ○ JPEG ● Dependent variable: consumed energy during common usage scenarios
  • 20. Vrije Universiteit Amsterdam 20 Ivano Malavolta / S2 group / Empirical software engineering 1F-2T: fully randomized design ● Each treatment is randomly assigned to the experimental objects ● Same number of objects per each treatment (balancing) Object (Application) Treatment 1 (PNG) Treatment 2 (JPEG) 1 X 2 X 3 X 4 X Example of hypotheses: H0 : 1 = 2 Ha : 1 != 2 or 1 > 2 or 1 < 2 Analyses: ● t-test (unpaired) ● Mann-Whitney test
  • 21. Vrije Universiteit Amsterdam 21 Ivano Malavolta / S2 group / Empirical software engineering 1F-2T: paired comparison design ● Each treatment is applied on each object (crossover design) ● The order of the treatments is random Object (Application) Treatment 1 (PNG) Treatment 2 (JPEG) 1 1st 2nd 2 2nd 1st 3 2nd 1st 4 1st 2nd
  • 22. Vrije Universiteit Amsterdam 22 Ivano Malavolta / S2 group / Empirical software engineering 1F-2T: paired comparison design P t A1, JPEG A1, PNG tj dj d = avg(d0...n )
  • 23. Vrije Universiteit Amsterdam 23 Ivano Malavolta / S2 group / Empirical software engineering 1F-2T: paired comparison design Example of hypotheses: H0 : d = 0 Ha : d != 0or d > 0or d < 0 Analyses: ● Paired t-test ● Sign test ● Wilcoxon
  • 24. Vrije Universiteit Amsterdam 24 How to choose between the two design? Object (Application) Treatment 1 (PNG) Treatment 2 (JPEG) 1 1st 2nd 2 2nd 1st 3 2nd 1st 4 1st 2nd Object (Application) Treatment 1 (PNG) Treatment 2 (JPEG) 1 X 2 X 3 X 4 X
  • 25. Vrije Universiteit Amsterdam 25 Ivano Malavolta / S2 group / Empirical software engineering 1 factor and >2 treatments In this case the factor can have more than one value Example: ● Factor: encoding algorithms ● Treatments: ○ PNG ○ JPEG ○ TIFF ● Dependent variable: consumed energy during common usage scenarios
  • 26. Vrije Universiteit Amsterdam 26 Ivano Malavolta / S2 group / Empirical software engineering 1F-MT: fully randomized design ● Each treatment is randomly assigned to the experimental objects ● Same number of objects per each treatment (balancing) Example of hypotheses: H0 : 1 = 2 = 3 = … = t Ha : i != j for at least on pair of (i,j) Analyses: ● ANOVA (ANalysis Of VAriance) ● Kruskal-Wallis Object (Application) Treatment 1 (PNG) Treatment 2 (JPEG) Treatment 2 (TIFF) 1 X 2 X 3 X 4 X 5 X 6 X
  • 27. Vrije Universiteit Amsterdam 27 Ivano Malavolta / S2 group / Empirical software engineering 1F-MT: paired comparison design ● Each treatment is applied on each object (crossover design) ● The order of the treatments is random Object (Application) Treatment 1 (PNG) Treatment 2 (JPEG) Treatment 2 (TIFF) 1 1st 3rd 2nd 2 3rd 1st 2nd 3 2nd 3rd 1st 4 2nd 1st 3rd 5 3rd 2nd 1st 6 1st 2nd 3rd Example of hypotheses: H0 : 1 = 2 = 3 = … = t Ha : i != j for at least on pair of (i,j) Analyses: ● ANOVA ● Kruskal-Wallis ● Repeated Measures ANOVA
  • 28. Vrije Universiteit Amsterdam 28 Ivano Malavolta / S2 group / Empirical software engineering Some contents of lecture extracted from: ● Giuseppe Procaccianti’s lectures at VU Acknowledgements