3. A test engineer walks into
a bar and
• orders a beer
• orders 0 beers
• orders 9999999 beers
• orders a lizard
• orders -1 beers
• orders a "sfdeljknesv"
— Bill Sempf (@sempf)
4. A test engineer walks into
a bar and
• orders a beer
• orders 0 beers
• orders 9999999 beers
• orders a lizard
• orders -1 beers
• orders a "sfdeljknesv"
— Bill Sempf (@sempf)
5. A test engineer walks into
a bar and
• orders a beer
• orders 0 beers
• orders 9999999 beers
• orders a lizard
• orders -1 beers
• orders a "sfdeljknesv"
— Bill Sempf (@sempf)
6. A test engineer walks into
a bar and
• orders a beer
• orders 0 beers
• orders 9999999 beers
• orders a lizard
• orders -1 beers
• orders a "sfdeljknesv"
— Bill Sempf (@sempf)
7. A test engineer walks into
a bar and
• orders a beer
• orders 0 beers
• orders 9999999 beers
• orders a lizard
• orders -1 beers
• orders a "sfdeljknesv"
— Bill Sempf (@sempf)
8. A test engineer walks into
a bar and
• orders a beer
• orders 0 beers
• orders 9999999 beers
• orders a lizard
• orders -1 beers
• orders a "sfdeljknesv"
— Bill Sempf (@sempf)
9. A test engineer walks into
a bar and
• orders a beer
• orders 0 beers
• orders 9999999 beers
• orders a lizard
• orders -1 beers
• orders a "sfdeljknesv"
— Bill Sempf (@sempf)
9,999,999
10. A test engineer walks into
a bar and
• orders a beer
• orders 0 beers
• orders 9999999 beers
• orders a lizard
• orders -1 beers
• orders a "sfdeljknesv"
— Bill Sempf (@sempf)
11. A test engineer walks into
a bar and
• orders a beer
• orders 0 beers
• orders 9999999 beers
• orders a lizard
• orders -1 beers
• orders a "sfdeljknesv"
— Bill Sempf (@sempf) ?!?
12. A test engineer walks into
a bar and
• orders a beer
• orders 0 beers
• orders 9999999 beers
• orders a lizard
• orders -1 beers
• orders a "sfdeljknesv"
— Bill Sempf (@sempf)
13. A test engineer walks into
a bar and
• orders a beer
• orders 0 beers
• orders 9999999 beers
• orders a lizard
• orders -1 beers
• orders a "sfdeljknesv"
— Bill Sempf (@sempf)
-1
?!?
14. A test engineer walks into
a bar and
• orders a beer
• orders 0 beers
• orders 9999999 beers
• orders a lizard
• orders -1 beers
• orders a "sfdeljknesv"
— Bill Sempf (@sempf)
15. A test engineer walks into
a bar and
• orders a beer
• orders 0 beers
• orders 9999999 beers
• orders a lizard
• orders -1 beers
• orders a "sfdeljknesv"
— Bill Sempf (@sempf)
"sfdeljknesv"
?!?
26. Software testing consists of the dynamic verification
that a program provides expected behaviours
on a finite set of test cases, suitably selected from the
usually infinite execution domain.
[SWEBOK v3]
Software-intensive Systems Testing
28. [A Software Product Line (SPL) is]
a set of software-intensive systems that share
a common, managed set of features satisfying the specific
needs of a particular market segment or mission and that are
developed from a common set of
core assets in a prescribed way
P. C. Clements and L. Northrop — ‘‘Software Product Lines:
Practices and Patterns’’.
SEI Series in Software Engineering. Addison-Wesley, 2001.
Variability-Intensive Systems: SPL
39. 21
gilles.perrouin@unamur.be 21
We therefore regroup all these systems under the
“variability-intensive systems” umbrella for feature-
based testing….
…thanks to a very liberal notion of feature!
Variability-Intensive Systems
47. Feature Model (FM)
CPTerminal t
Identification id
DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig
requires
PaymentSchema ps Connectivity c
[1..*]
opt.
48. Feature Model (FM)
CPTerminal t
Identification id
DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig
requires
PaymentSchema ps Connectivity c
[1..*]
opt.
49. Feature Model (FM)
CPTerminal t
Identification id
DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig
requires
PaymentSchema ps Connectivity c
[1..*]
opt.
50. Feature Model (FM)
CPTerminal t
Identification id
DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig
requires
PaymentSchema ps Connectivity c
[1..*]
opt.
51. Feature Model (FM)
CPTerminal t
Identification id
DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig
requires
PaymentSchema ps Connectivity c
[1..*]
opt.
52. Feature Model (FM)
CPTerminal t
Identification id
DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig
requires
PaymentSchema ps Connectivity c
[1..*]
opt.
53. Feature Model (FM)
CPTerminal t
Identification id
DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig
requires
PaymentSchema ps Connectivity c
[1..*]
opt.
54. Software testing consists of the dynamic verification
that a program provides expected behaviours
on a finite set of test cases, suitably selected from the
usually infinite execution domain.
Software-intensive Systems Testing
[SWEBOK v3]
55. Variability-intensive systems testing consists of the
dynamic verification
that a a set of software-intensive systems programs
provides expected behaviours
on a finite set of test cases, suitably selected from the
usually infinite execution domain.
Variability-Intensive Systems Testing
64. 33
gilles.perrouin@unamur.be 33
We would like to provide you an overview of the
model-based techniques to test variability-
intensive systems, considering both structural and
behavioural concerns.
Tutorial Objective
65. 34
gilles.perrouin@unamur.be 34
Part I (Until 16h) : sample configurations to test
Feature modelling
Sampling techniques
Discussion on the JHipster case
Part II (16h30-18h) : select behaviours to test
A (short) introduction to behaviour modelling
Behaviour-driven test case selection
Test concretisation
Open Discussion (18h-18h30)
Tutorial Agenda
67. 36
gilles.perrouin@unamur.be 36
Feature Modelling Ecosystem
Academic
Marcilio Mendonca, Moises Branco, Donald
Cowan: S.P.L.O.T. - Software Product Lines
Online Tools. OOPSLA Companion, October
2009, Orlando, Florida, USA.
Christian Kästner, Thomas Thüm, Gunter Saake, Janet Feigenspan,
Thomas Leich, Fabian Wielgorz, and Sven Apel. FeatureIDE: Tool
Framework for Feature-Oriented Software Development. ICSE 2009.
Formal Demonstration paper.
Mathieu Acher, Philippe
Collet, Philippe Lahire, Robert
France. FAMILIAR: A Domain-
Specific Language for Large
Scale Management of
Feature Models. Science of
Computer Programming,
Elsevier, 2013, 78 (6)
Antkiewicz, M., K. Bąk, A. Murashkin, R. Olaechea, J. Liang,
and K. Czarnecki, "Clafer Tools for Product Line
Engineering", Software Product Line Conference, Tokyo,
Japan, 2013.
root Website {
group [0..2] {
Foo,
Bar,
Baz,
}
Baz requires Bar;
}Andreas Classen, Quentin Boucher, Patrick Heymans, A text-based
approach to feature modelling: Syntax and semantics of TVL, In
Science of Computer Programming, Volume 76, Issue 12, 2011
TVL
Haugen, Ø., Wasowski, A., & Czarnecki, K. (2012, September). CVL:
common variability language. In SPLC (2) (pp. 266-267).
68. 36
gilles.perrouin@unamur.be 36
Feature Modelling Ecosystem
Academic
Marcilio Mendonca, Moises Branco, Donald
Cowan: S.P.L.O.T. - Software Product Lines
Online Tools. OOPSLA Companion, October
2009, Orlando, Florida, USA.
Christian Kästner, Thomas Thüm, Gunter Saake, Janet Feigenspan,
Thomas Leich, Fabian Wielgorz, and Sven Apel. FeatureIDE: Tool
Framework for Feature-Oriented Software Development. ICSE 2009.
Formal Demonstration paper.
Mathieu Acher, Philippe
Collet, Philippe Lahire, Robert
France. FAMILIAR: A Domain-
Specific Language for Large
Scale Management of
Feature Models. Science of
Computer Programming,
Elsevier, 2013, 78 (6)
Antkiewicz, M., K. Bąk, A. Murashkin, R. Olaechea, J. Liang,
and K. Czarnecki, "Clafer Tools for Product Line
Engineering", Software Product Line Conference, Tokyo,
Japan, 2013.
root Website {
group [0..2] {
Foo,
Bar,
Baz,
}
Baz requires Bar;
}Andreas Classen, Quentin Boucher, Patrick Heymans, A text-based
approach to feature modelling: Syntax and semantics of TVL, In
Science of Computer Programming, Volume 76, Issue 12, 2011
TVL
Haugen, Ø., Wasowski, A., & Czarnecki, K. (2012, September). CVL:
common variability language. In SPLC (2) (pp. 266-267).
Most Influential Paper Award SPLC 2017 !
69. 37
gilles.perrouin@unamur.be 37
Feature Modelling Ecosystem
Commercial
Tutorial: Using Feature Models to Manage
Variability and Requirements Reuse -
Danilo Beuche
SPLC 2017
Tutorial: Feature-Based
Systems and Software
Product Line Engineering:
PLE for the Enterprise -
Charles Krueger and Paul
Clements
SPLC 2017
70. AIM HIGHER
WITH SKALUP
Discover what we can do
for you
VISIT US @ SKALUP.COM
MAXIME.CORDY@SKALUP.COM
PATRICK.HEYMANS@SKALUP.COM
79. 40
gilles.perrouin@unamur.be 40
Source code
Existing
configurators
Product Descriptions
Architecture
…
Reverse Engineering of Feature Models
J. Sincero et al. Efficient extraction and analysis of preprocessor-based
variability. GPCE '10.
S. She et al., Reverse engineering feature models, ICSE 2011
80. 40
gilles.perrouin@unamur.be 40
Source code
Existing
configurators
Product Descriptions
Architecture
…
Reverse Engineering of Feature Models
J. Sincero et al. Efficient extraction and analysis of preprocessor-based
variability. GPCE '10.
S. She et al., Reverse engineering feature models, ICSE 2011
JM. Davril et al. Feature model extraction from large collections of
informal product descriptions. ESEC/FSE 2013.
81. 40
gilles.perrouin@unamur.be 40
Source code
Existing
configurators
Product Descriptions
Architecture
…
Reverse Engineering of Feature Models
J. Sincero et al. Efficient extraction and analysis of preprocessor-based
variability. GPCE '10.
S. She et al., Reverse engineering feature models, ICSE 2011
JM. Davril et al. Feature model extraction from large collections of
informal product descriptions. ESEC/FSE 2013.
M. Acher, et al . Reverse engineering architectural feature models. ECSA'11
82. 40
gilles.perrouin@unamur.be 40
Source code
Existing
configurators
Product Descriptions
Architecture
…
Reverse Engineering of Feature Models
REVE 2017
5th International Workshop on Reverse
Variability Engineering
J. Sincero et al. Efficient extraction and analysis of preprocessor-based
variability. GPCE '10.
S. She et al., Reverse engineering feature models, ICSE 2011
JM. Davril et al. Feature model extraction from large collections of
informal product descriptions. ESEC/FSE 2013.
M. Acher, et al . Reverse engineering architectural feature models. ECSA'11
98. 44
gilles.perrouin@unamur.be 44
Assumption
Most of the faults are due to the undesired
interactions of a limited number of options
Core idea
Extract configurations so that all combinations of
options are covered (at least) once
Combinatorial Interaction Testing
D. M. Cohen et al., The AETG system: an approach to testing based on
combinatorial design. TSE, 1997
D. Kuhn et al, Software fault
interactions and implications for software testing. TSE 2004
99. 45
CPTerminal t
Identification id
DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig
requires
PaymentSchema ps Connectivity c
[1..*]
opt.
Combinatorial Interaction Testing
100. 45
CPTerminal t
Identification id
DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig
requires
PaymentSchema ps Connectivity c
[1..*]
opt.{t, ps},
{t, c},
{dd, cr},
{dd,on},
{dd, off},
…
Combinatorial Interaction Testing
101. 45
CPTerminal t
Identification id
DirectDebit dd CreditCard cr Online on Offline off PIN pin Signature sig
requires
PaymentSchema ps Connectivity c
[1..*]
opt.{t, ps},
{t, c},
{dd, cr},
{dd,on},
{dd, off},
…
Combinatorial Interaction Testing
not only for 2-tuples
(pairwise), interaction
strength=[1(?),6]
103. 46
gilles.perrouin@unamur.be 46
Huge reduction of the number of configurations to
consider (2-wise):
Printers: 172 features, 1.14E27 configs => 180
configs
Linux: 6,888 features, ? configs => 480 configs
CIT for Variability Intensive Systems
104. 46
gilles.perrouin@unamur.be 46
Huge reduction of the number of configurations to
consider (2-wise):
Printers: 172 features, 1.14E27 configs => 180
configs
Linux: 6,888 features, ? configs => 480 configs
CIT for Variability Intensive Systems
Martin Fagereng Johansen et al. An algorithm for
generating t-wise covering arrays from large feature
models. SPLC ‘12
105. 46
gilles.perrouin@unamur.be 46
Huge reduction of the number of configurations to
consider (2-wise):
Printers: 172 features, 1.14E27 configs => 180
configs
Linux: 6,888 features, ? configs => 480 configs
The main issue is that CIT extraction is an NP-
complete problem when there are constraints…
CIT for Variability Intensive Systems
Martin Fagereng Johansen et al. An algorithm for
generating t-wise covering arrays from large feature
models. SPLC ‘12
107. 47
gilles.perrouin@unamur.be 47
Alloy
Software Product Line
Feature Diagram
1. Transformation
FeatureDiagram2Alloy
2. Generation of
initial T-wise tuples l
3. Detection of
Valid tuples
4. Creating and Solving
Conjunctions of Tuples
BinarySplit Incremental
Growth
5. Analysis
Value of T
Alloy
Feature Diagram
AF
Set of Valid Tuples
V
[Min,Max] Scope
[Min,Max] Duration
Selection Strategy
Set of Configurations
covering all Valid Tuples
P
108. 47
gilles.perrouin@unamur.be 47
Alloy
Software Product Line
Feature Diagram
1. Transformation
FeatureDiagram2Alloy
2. Generation of
initial T-wise tuples l
3. Detection of
Valid tuples
4. Creating and Solving
Conjunctions of Tuples
BinarySplit Incremental
Growth
5. Analysis
Value of T
Alloy
Feature Diagram
AF
Set of Valid Tuples
V
[Min,Max] Scope
[Min,Max] Duration
Selection Strategy
Set of Configurations
covering all Valid Tuples
P
G. Perrouin et al., Automated and
Scalable T-wise Test Case
Generation Strategies for Software
Product Lines, ICST 2010
109. 48
gilles.perrouin@unamur.be 48
MoSo-Polite
4. MOSO-POLITE
MoSo-PoLiTe combines the aforementioned techniques Reu-
se Techniques and Subset-Heuristics providing a framework
to generate a subset of products including an automatic
model-based test case generation for each product of the
subset [22]. Within this contribution we only focus on the
subset generation and ignore the model-based test case ge-
neration.
Feature Model
Flat Feature Model
Flat Feature Model CSP
Flattening
Parameter
Extraction
Subset
Extractor
Set of Configurations
Feature Model
Parsing
Variant Models
Products
Feature Model
Combinatorial
Testing
Configuration
Derivation
Product
Derivation
select
Figure 2: MoSo-PoLiTe Framework
S. Oster et al., "Automated incremental pairwise testing of
software product lines.”SPLC 2010.
M. Steffens et al., Industrial evaluation of pairwise spl
testing with moso-polite. VAMOS 2012
110. 49
gilles.perrouin@unamur.be 49
Another constraint based approach based on prolog
Generates 30% less configurations than MoSo-
Polite
PACOGEN
Hervieu et al., PACOGEN: Automatic Generation of Pairwise Test
Configurations from Feature Models. ISSRE’ 11
111. 50
gilles.perrouin@unamur.be 50
Use search-based techniques (Simulated Annealing)
to minimise the size of covering array
Usually gives very small covering array compared to
others
CASA
M. B. Cohen et al., “Augmenting simulated annealing to build interaction
test suites,” ISSRE 2003
112. 51
gilles.perrouin@unamur.be 51
Use an incremental way of accommodating tuples
Deterministic algorithm always produce the same
answer for a given FM
Focus on scalability
Industrial version: pairwiser (see pairwiser.org)
SPLCAT
M. F. Johansen, Ø. Haugen, and F. Fleurey, “An algorithm for generating t-
wise covering arrays from large feature models,” SPLC 2012.
116. 52
gilles.perrouin@unamur.be 52
Despite the improvements in the CIT techniques that
can now scale to thousands of features, there are
two problems:
Higher order interaction remains expensive/
intractable
Open Issues in CIT
117. 52
gilles.perrouin@unamur.be 52
Despite the improvements in the CIT techniques that
can now scale to thousands of features, there are
two problems:
Higher order interaction remains expensive/
intractable
100% coverage implies no flexibility on your
testing budget (time + number of configurations)
Open Issues in CIT
119. 53
gilles.perrouin@unamur.be 53
Diversity has been proposed as a successful
technique to find bugs
Mimicking CIT with Dissimilarity
H. Hemmati et al., Achieving scalable model-based testing through test case
diversity, TOSEM 2012.
120. 53
gilles.perrouin@unamur.be 53
Diversity has been proposed as a successful
technique to find bugs
Intuition
Dissimilar configurations cover more t-tuples than
similar ones
Mimicking CIT with Dissimilarity
H. Hemmati et al., Achieving scalable model-based testing through test case
diversity, TOSEM 2012.
121. 53
gilles.perrouin@unamur.be 53
Diversity has been proposed as a successful
technique to find bugs
Intuition
Dissimilar configurations cover more t-tuples than
similar ones
Mimicking CIT with Dissimilarity
H. Hemmati et al., Achieving scalable model-based testing through test case
diversity, TOSEM 2012.
C. Henard et al., "Bypassing the Combinatorial Explosion: Using Similarity to Generate
and Prioritize T-Wise Test Configurations for Software Product Lines," TSE 2014
123. 54
gilles.perrouin@unamur.be 54
SBSE well-suited for large configurations spaces
Generated configurations are valid (via SAT4J)
Designed for Scalability
Fitness function based on distance: correlated
with t-wise coverage but easier to compute
Flexibility
Testers decide on generation time and #
configurations
Configurations are prioritized w.r.t fitness function:
use a subset if lack of resources
Mimicking CIT with Dissimilarity
131. 59
gilles.perrouin@unamur.be 59
Multi-objective
Combining similarity and cost
Optimisation w.r.t. to defects, costs, # unused/
selected features, correctness
Other Sampling Techniques
C. Henard, et al. Multi-objective test generation for
software product lines. SPLC 2013
C. Henard et al. Combining multi-objective search and
constraint solving for configuring large software product
lines. ICSE 2015.
132. 59
gilles.perrouin@unamur.be 59
Multi-objective
Combining similarity and cost
Optimisation w.r.t. to defects, costs, # unused/
selected features, correctness
Random
Other Sampling Techniques
C. Henard, et al. Multi-objective test generation for
software product lines. SPLC 2013
C. Henard et al. Combining multi-objective search and
constraint solving for configuring large software product
lines. ICSE 2015.
Andrea Arcuri et al. Formal Analysis of the Probability of
Interaction Fault Detection Using Random Testing. TSE
2012
133. 59
gilles.perrouin@unamur.be 59
Multi-objective
Combining similarity and cost
Optimisation w.r.t. to defects, costs, # unused/
selected features, correctness
Random
Min-Max selection
Other Sampling Techniques
C. Henard, et al. Multi-objective test generation for
software product lines. SPLC 2013
C. Henard et al. Combining multi-objective search and
constraint solving for configuring large software product
lines. ICSE 2015.
Andrea Arcuri et al. Formal Analysis of the Probability of
Interaction Fault Detection Using Random Testing. TSE
2012
I. Abal, et al. 42 variability bugs in the Linux kernel: A qualitative analysis. ASE 2014
F. Medeiros et al., A Comparison of 10 Sampling Algorithms for Configurable Systems,
ICSE 16
134. 60
gilles.perrouin@unamur.be 60
Direct comparison
Using mutation analysis
Comparing/Evaluating Sampling
Techniques
F. Medeiros et al., A Comparison of 10 Sampling
Algorithms for Configurable Systems, ICSE 16
G. Perrouin et al., Pairwise testing for software
product lines: comparison of two approaches,
Softw. Qual. Journal 2012.
C. Henard et al., Assessing software product line
testing via model-based mutation: An application
to similarity testing, ICST Workshops 2013
136. 61
gilles.perrouin@unamur.be 61
Many sampling ways
CIT effective at finding bugs but expensive in some
cases
SBSE useful for fitting test budget while ensuring
decent coverage
Conclusion on Sampling Techniques
137. 61
gilles.perrouin@unamur.be 61
Many sampling ways
CIT effective at finding bugs but expensive in some
cases
SBSE useful for fitting test budget while ensuring
decent coverage
Conclusion on Sampling Techniques
138. 61
gilles.perrouin@unamur.be 61
Many sampling ways
CIT effective at finding bugs but expensive in some
cases
SBSE useful for fitting test budget while ensuring
decent coverage
More research is required to compare
techniques…
Conclusion on Sampling Techniques
139. 61
gilles.perrouin@unamur.be 61
Many sampling ways
CIT effective at finding bugs but expensive in some
cases
SBSE useful for fitting test budget while ensuring
decent coverage
More research is required to compare
techniques…
Conclusion on Sampling Techniques
140. 61
gilles.perrouin@unamur.be 61
Many sampling ways
CIT effective at finding bugs but expensive in some
cases
SBSE useful for fitting test budget while ensuring
decent coverage
More research is required to compare
techniques…
…on various case studies
Conclusion on Sampling Techniques
143. 64
gilles.perrouin@unamur.be 64
Web-apps generator
Spring-Boot
Bootstrap / AngularJS
100 % Open Source
Yeoman
Bower, npm
yo
Used all over the world
Large companies (HBO, Google, Adobe)1
Independent developers
GitHub
5000+ stars
100+ releases (JHipster 3.6.1, 18 Aug
2016)
JHipster Introduction
1 https://jhipster.github.io/companies-using-jhipster
151. 70
gilles.perrouin@unamur.be 70
They like the idea of being able to run all JHipster
configurations (26,000) and finding all the 6 faults
reported via the analysis of the failing
configurations….
…but they only have the infrastructure to test only 12
configurations (now expanded to 20) !
JHipster Team Feedback
(Work In Progress)
152. 70
gilles.perrouin@unamur.be 70
They like the idea of being able to run all JHipster
configurations (26,000) and finding all the 6 faults
reported via the analysis of the failing
configurations….
…but they only have the infrastructure to test only 12
configurations (now expanded to 20) !
JHipster Team Feedback
(Work In Progress)
W
hich sampling
strategy would you
use?
