TEST CASE PRIORITIZATION FOR OPTIMIZING A REGRESSION TEST
Ecec09presentation
1. inspection
optimization
for MSPS
Sofie Van
Volsem
Joint optimization of all inspection
Introduction
parameters for multi-stage processes:
MSPS
Inspection
algorithm, simulation and test set
Cost
Process model
Method
Finding solutions
Part 1: TIC
Sofie Van Volsem
Part 2: EA
Conclusion
Department of Industrial Management
Ghent University
Bruges, April 15, 2009
2. Overview
inspection
optimization
for MSPS
Introduction
1
Sofie Van
Volsem
Multistage production systems
Inspection strategy
Introduction
MSPS
Cost-efficient inspection
Inspection
Cost
Process model
Process model
Method
Finding solutions
Method
2
Part 1: TIC
Part 2: EA
Finding solutions
Conclusion
First problem: calculating inspection costs
Second problem: an intelligent solution space search
Conclusion
3
3. Sequential linear multistage production system
(MSPS)
inspection
optimization
for MSPS
Sofie Van
Volsem
Introduction
MSPS
Inspection
Cost
Process model
Method
Finding solutions
Part 1: TIC
Part 2: EA
Conclusion
4. Sequential linear multistage production system
(MSPS)
inspection
optimization
for MSPS
Sofie Van
Volsem
Introduction
MSPS
Inspection
Cost
example: Production of chocolate cookies
Process model
production stage 1: preparation of dough
Method
Finding solutions
production stage 2: baking of cookies
Part 1: TIC
Part 2: EA
production stage 3: finishing with chocolate
Conclusion
5. Inspection strategies for MSPS
inspection
optimization
for MSPS
Sofie Van
Volsem
An inspection strategy for MSPS is
Introduction
MSPS
a set of decisions
Inspection
Cost
WHERE to inspect:
1
Process model
after which of the production stages?
Method
HOW STRINGENT to inspect:
Finding solutions 2
Part 1: TIC
what are the acceptance limits?
Part 2: EA
HOW MUCH to inspect:
3
Conclusion
all products or only a sample?
6. Inspection strategies for MSPS
inspection
optimization
for MSPS
Sofie Van
Volsem
An inspection strategy for MSPS is
Introduction
MSPS
a set of decisions
Inspection
Cost
WHERE to inspect:
1
Process model
after which of the production stages?
Method
HOW STRINGENT to inspect:
Finding solutions 2
Part 1: TIC
what are the acceptance limits?
Part 2: EA
HOW MUCH to inspect:
3
Conclusion
all products or only a sample?
7. Inspection strategies for MSPS
inspection
optimization
for MSPS
Sofie Van
Volsem
An inspection strategy for MSPS is
Introduction
MSPS
a set of decisions
Inspection
Cost
WHERE to inspect:
1
Process model
after which of the production stages?
Method
HOW STRINGENT to inspect:
Finding solutions 2
Part 1: TIC
what are the acceptance limits?
Part 2: EA
HOW MUCH to inspect:
3
Conclusion
all products or only a sample?
8. Inspection strategies for MSPS
inspection
optimization
for MSPS
Sofie Van
Volsem
An inspection strategy for MSPS is
Introduction
MSPS
a set of decisions
Inspection
Cost
WHERE to inspect:
1
Process model
after which of the production stages?
Method
HOW STRINGENT to inspect:
Finding solutions 2
Part 1: TIC
what are the acceptance limits?
Part 2: EA
HOW MUCH to inspect:
3
Conclusion
all products or only a sample?
9. Inspection costs
inspection
optimization
for MSPS
Sofie Van
Costs associated with a selected inspection strategy:
Volsem
execute inspection
1
Introduction
(test cost, TC)
MSPS
Inspection
repair or replace faulty products internally
2
Cost
(rework cost, RC)
Process model
Method
repair or replace faulty products externally
3
Finding solutions
(penalty cost, PC)
Part 1: TIC
Part 2: EA
Total costs also includes (loss of) production time,
Conclusion
capacity, product image, ...
Simplified: more and tighter inspection will lead to
higher quality, but will also induce higher costs.
10. Inspection costs
inspection
optimization
for MSPS
Sofie Van
Costs associated with a selected inspection strategy:
Volsem
execute inspection
1
Introduction
(test cost, TC)
MSPS
Inspection
repair or replace faulty products internally
2
Cost
(rework cost, RC)
Process model
Method
repair or replace faulty products externally
3
Finding solutions
(penalty cost, PC)
Part 1: TIC
Part 2: EA
Total costs also includes (loss of) production time,
Conclusion
capacity, product image, ...
Simplified: more and tighter inspection will lead to
higher quality, but will also induce higher costs.
11. Inspection costs
inspection
optimization
for MSPS
Sofie Van
Costs associated with a selected inspection strategy:
Volsem
execute inspection
1
Introduction
(test cost, TC)
MSPS
Inspection
repair or replace faulty products internally
2
Cost
(rework cost, RC)
Process model
Method
repair or replace faulty products externally
3
Finding solutions
(penalty cost, PC)
Part 1: TIC
Part 2: EA
Total costs also includes (loss of) production time,
Conclusion
capacity, product image, ...
Simplified: more and tighter inspection will lead to
higher quality, but will also induce higher costs.
12. Inspection costs
inspection
optimization
for MSPS
Sofie Van
Costs associated with a selected inspection strategy:
Volsem
execute inspection
1
Introduction
(test cost, TC)
MSPS
Inspection
repair or replace faulty products internally
2
Cost
(rework cost, RC)
Process model
Method
repair or replace faulty products externally
3
Finding solutions
(penalty cost, PC)
Part 1: TIC
Part 2: EA
Total costs also includes (loss of) production time,
Conclusion
capacity, product image, ...
Simplified: more and tighter inspection will lead to
higher quality, but will also induce higher costs.
13. Inspection optimization for MSPS: process
model
inspection
optimization
for MSPS
Sofie Van
Volsem
Introduction
MSPS
Inspection
Cost
Process model
For each production stage:
Method
Finding solutions
Cost parameters
Part 1: TIC
Part 2: EA
(test cost TC, rework cost RC,
Conclusion
penalty cost, PC (only after final production stage))
Process parameters
(process characteristics: mean and variance)
Inspection parameters
(where, how much and how stringent to inspect?)
14. Optimization: what are the decision variables?
inspection
optimization
for MSPS
Sofie Van
Volsem
Cost and process parameters are given.
Introduction
Only the inspection parameters are decision variables.
MSPS
Inspection
In multistage systems three types of inspection
Cost
parameters can be distinguished, namely
Process model
Method
inspection type
1
Finding solutions
Part 1: TIC
100% inspection (F)
Part 2: EA
sampling inspection (S)
Conclusion
no inspection (N)
inspection (acceptance) limits
2
sampling parameters
3
15. Optimization: what are the decision variables?
inspection
optimization
for MSPS
Sofie Van
Volsem
Cost and process parameters are given.
Introduction
Only the inspection parameters are decision variables.
MSPS
Inspection
In multistage systems three types of inspection
Cost
parameters can be distinguished, namely
Process model
Method
inspection type
1
Finding solutions
Part 1: TIC
100% inspection (F)
Part 2: EA
sampling inspection (S)
Conclusion
no inspection (N)
inspection (acceptance) limits
2
sampling parameters
3
16. Decision variables: illustration
inspection
optimization
for MSPS
Sofie Van
Volsem
Introduction
MSPS
Inspection
Cost
Process model
Method
Finding solutions
Part 1: TIC
Part 2: EA
Conclusion
17. Finding solutions
inspection
optimization
for MSPS
Sofie Van
Volsem
Solution = cost-efficient inspection strategy for MSPS
Introduction
Best solution => lowest total inspection cost (TIC)
MSPS
Inspection
Cost
For every possible solution we need to be able to
1
Process model
Method
calculate TIC
Finding solutions
Part 1: TIC
Number of possible solutions is infinite
2
Part 2: EA
=> naive heuristic = calculate every possibility to find
Conclusion
the best = impossible
=> development of an intelligent search method =
metaheuristic
18. Finding solutions
inspection
optimization
for MSPS
Sofie Van
Volsem
Solution = cost-efficient inspection strategy for MSPS
Introduction
Best solution => lowest total inspection cost (TIC)
MSPS
Inspection
Cost
For every possible solution we need to be able to
1
Process model
Method
calculate TIC
Finding solutions
Part 1: TIC
Number of possible solutions is infinite
2
Part 2: EA
=> naive heuristic = calculate every possibility to find
Conclusion
the best = impossible
=> development of an intelligent search method =
metaheuristic
19. Finding solutions
inspection
optimization
for MSPS
Sofie Van
Volsem
Solution = cost-efficient inspection strategy for MSPS
Introduction
Best solution => lowest total inspection cost (TIC)
MSPS
Inspection
Cost
For every possible solution we need to be able to
1
Process model
Method
calculate TIC
Finding solutions
Part 1: TIC
Number of possible solutions is infinite
2
Part 2: EA
=> naive heuristic = calculate every possibility to find
Conclusion
the best = impossible
=> development of an intelligent search method =
metaheuristic
20. Calculating TIC: formula
inspection
optimization
for MSPS
Sofie Van
TIC TTC + TRC + TPC (1)
=
Volsem
with
Introduction
n
MSPS
Inspection
TTC TCi (2)
=
Cost
Process model
i=1
Method
n
Finding solutions
Part 1: TIC
TRC RCi (3)
=
Part 2: EA
i=1
Conclusion
TPC cP .dn (4)
=
and with
TCi cT ,i .(αF ,i .K + αS,i .si ) (5)
=
RCi cR,i .pi .αF ,i .K (6)
=
21. Calculating TIC: illustration
inspection
optimization
for MSPS
Sofie Van
Volsem
Introduction
MSPS
Inspection
Cost
Process model
Method
Finding solutions
Part 1: TIC
Part 2: EA
Conclusion
22. Calculating TIC: method
inspection
optimization
for MSPS
Sofie Van
Volsem
With known defect rates pi , analytical calculation of TIC
is straightforward.
Introduction
MSPS
Alas, no closed analytical formula for pi available for
Inspection
Cost
non-trivial cases.
Process model
Method
Definition:
Finding solutions
Part 1: TIC
Part 2: EA
pi = P [Xi ∈ [LILi , UILi ]] = 1 − P[LILi ≤ Xi ≤ UILi ]
/
Conclusion
=> TIC is therefore calculated (approximated) through
Monte Carlo simulation.
23. Search strategy: evolutionary algorithm
Applied metaheuristic search method: Evolutionary
inspection
optimization
Algorithm (EA)
for MSPS
Sofie Van
based on Darwin’s theory on biological evolution:
Volsem
desirable characteristics => better chance of survival
Introduction
MSPS
=> better chance of transferral to next generation.
Inspection
Cost
characteristics quot;storedquot; in genes; genes are transferred
Process model
through reproduction/breeding.
Method
Finding solutions
principles evolutionary algorithm:
Part 1: TIC
Part 2: EA
encoding of candidate solutions;
1
Conclusion
creation of an intital population
evaluating and ordering candidate solutions
2
creating a new generation of candidate solutions from
3
promising (parts of) candidate solutions of the previous
generation
iterating steps 2 and 3 until stopping criterium;
4
decoding of quot;bestquot; solution
24. Search strategy: evolutionary algorithm
Applied metaheuristic search method: Evolutionary
inspection
optimization
Algorithm (EA)
for MSPS
Sofie Van
based on Darwin’s theory on biological evolution:
Volsem
desirable characteristics => better chance of survival
Introduction
MSPS
=> better chance of transferral to next generation.
Inspection
Cost
characteristics quot;storedquot; in genes; genes are transferred
Process model
through reproduction/breeding.
Method
Finding solutions
principles evolutionary algorithm:
Part 1: TIC
Part 2: EA
encoding of candidate solutions;
1
Conclusion
creation of an intital population
evaluating and ordering candidate solutions
2
creating a new generation of candidate solutions from
3
promising (parts of) candidate solutions of the previous
generation
iterating steps 2 and 3 until stopping criterium;
4
decoding of quot;bestquot; solution
25. Search strategy: evolutionary algorithm
Applied metaheuristic search method: Evolutionary
inspection
optimization
Algorithm (EA)
for MSPS
Sofie Van
based on Darwin’s theory on biological evolution:
Volsem
desirable characteristics => better chance of survival
Introduction
MSPS
=> better chance of transferral to next generation.
Inspection
Cost
characteristics quot;storedquot; in genes; genes are transferred
Process model
through reproduction/breeding.
Method
Finding solutions
principles evolutionary algorithm:
Part 1: TIC
Part 2: EA
encoding of candidate solutions;
1
Conclusion
creation of an intital population
evaluating and ordering candidate solutions
2
creating a new generation of candidate solutions from
3
promising (parts of) candidate solutions of the previous
generation
iterating steps 2 and 3 until stopping criterium;
4
decoding of quot;bestquot; solution
26. Search strategy: evolutionary algorithm
Applied metaheuristic search method: Evolutionary
inspection
optimization
Algorithm (EA)
for MSPS
Sofie Van
based on Darwin’s theory on biological evolution:
Volsem
desirable characteristics => better chance of survival
Introduction
MSPS
=> better chance of transferral to next generation.
Inspection
Cost
characteristics quot;storedquot; in genes; genes are transferred
Process model
through reproduction/breeding.
Method
Finding solutions
principles evolutionary algorithm:
Part 1: TIC
Part 2: EA
encoding of candidate solutions;
1
Conclusion
creation of an intital population
evaluating and ordering candidate solutions
2
creating a new generation of candidate solutions from
3
promising (parts of) candidate solutions of the previous
generation
iterating steps 2 and 3 until stopping criterium;
4
decoding of quot;bestquot; solution
27. Search strategy: evolutionary algorithm
Applied metaheuristic search method: Evolutionary
inspection
optimization
Algorithm (EA)
for MSPS
Sofie Van
based on Darwin’s theory on biological evolution:
Volsem
desirable characteristics => better chance of survival
Introduction
MSPS
=> better chance of transferral to next generation.
Inspection
Cost
characteristics quot;storedquot; in genes; genes are transferred
Process model
through reproduction/breeding.
Method
Finding solutions
principles evolutionary algorithm:
Part 1: TIC
Part 2: EA
encoding of candidate solutions;
1
Conclusion
creation of an intital population
evaluating and ordering candidate solutions
2
creating a new generation of candidate solutions from
3
promising (parts of) candidate solutions of the previous
generation
iterating steps 2 and 3 until stopping criterium;
4
decoding of quot;bestquot; solution
28. Search strategy: evolutionary algorithm
Applied metaheuristic search method: Evolutionary
inspection
optimization
Algorithm (EA)
for MSPS
Sofie Van
based on Darwin’s theory on biological evolution:
Volsem
desirable characteristics => better chance of survival
Introduction
MSPS
=> better chance of transferral to next generation.
Inspection
Cost
characteristics quot;storedquot; in genes; genes are transferred
Process model
through reproduction/breeding.
Method
Finding solutions
principles evolutionary algorithm:
Part 1: TIC
Part 2: EA
encoding of candidate solutions;
1
Conclusion
creation of an intital population
evaluating and ordering candidate solutions
2
creating a new generation of candidate solutions from
3
promising (parts of) candidate solutions of the previous
generation
iterating steps 2 and 3 until stopping criterium;
4
decoding of quot;bestquot; solution
29. Search strategy: evolutionary algorithm
Applied metaheuristic search method: Evolutionary
inspection
optimization
Algorithm (EA)
for MSPS
Sofie Van
based on Darwin’s theory on biological evolution:
Volsem
desirable characteristics => better chance of survival
Introduction
MSPS
=> better chance of transferral to next generation.
Inspection
Cost
characteristics quot;storedquot; in genes; genes are transferred
Process model
through reproduction/breeding.
Method
Finding solutions
principles evolutionary algorithm:
Part 1: TIC
Part 2: EA
encoding of candidate solutions;
1
Conclusion
creation of an intital population
evaluating and ordering candidate solutions
2
creating a new generation of candidate solutions from
3
promising (parts of) candidate solutions of the previous
generation
iterating steps 2 and 3 until stopping criterium;
4
decoding of quot;bestquot; solution
30. Evolutionary algorithm: example
inspection
optimization
for MSPS
Sofie Van
Volsem
Introduction
MSPS
Inspection
Cost
Process model
Method
Finding solutions
Part 1: TIC
Part 2: EA
Conclusion
31. Evolutionary algorithm: example
inspection
optimization
for MSPS
Sofie Van
Volsem
Introduction
MSPS
Inspection
Cost
Process model
Method
Finding solutions
Part 1: TIC
Part 2: EA
Conclusion
32. Evolutionary algorithm: example
inspection
optimization
for MSPS
Sofie Van
Volsem
Introduction
MSPS
Inspection
Cost
Process model
Method
Finding solutions
Part 1: TIC
Part 2: EA
Conclusion
33. Does the method work?
inspection
optimization
1◦ EA’s convergence is established
for MSPS
Sofie Van
Volsem
Introduction
MSPS
Inspection
Cost
Process model
Method
Finding solutions
Part 1: TIC
Part 2: EA
Conclusion
34. Does the method work?
inspection
optimization
1◦ EA’s convergence is established
for MSPS
Sofie Van
Volsem
Introduction
MSPS
Inspection
Cost
Process model
Method
Finding solutions
Part 1: TIC
Part 2: EA
Conclusion
35. Does the method work?
inspection
optimization
for MSPS
Sofie Van
Volsem
2◦ EA’s capability to find meaningful solutions is established
Introduction
MSPS
Inspection
Cost
Process model
Method
Finding solutions
Part 1: TIC
Part 2: EA
Conclusion
36. Does the method work?
inspection
optimization
for MSPS
Sofie Van
Volsem
2◦ EA’s capability to find meaningful solutions is established
Introduction
10 processes (A through J) were analyzed and compared
MSPS
Inspection
Cost
Process model
Method
Finding solutions
Part 1: TIC
Part 2: EA
Conclusion
37. Does the method work?
inspection
optimization
for MSPS
Sofie Van
Volsem
2◦ EA’s capability to find meaningful solutions is established
Introduction
10 processes (A through J) were analyzed and compared
MSPS
Inspection
Cost
cases A through J process mean exp. value
Process model
Method
step 1 normal µ = 10 10
Finding solutions
Part 1: TIC
step 2 + normal µ = 10 20
Part 2: EA
step 3 + normal µ = 10 30
Conclusion
step 4 + normal µ = 10 40
38. Does the method work?
inspection
optimization
for MSPS
Sofie Van
Volsem
Introduction
case A B C D E
MSPS
Inspection
all steps σ = 0.1 σ = 0.1 σ = 0.1 σ = 0.2 σ = 0.2
Cost
Process model
penalty 1 000 10 000 100 000 1 000 10 000
Method
Finding solutions
case F G H I J
Part 1: TIC
Part 2: EA
steps 1&3 σ = 0.2 σ = 0.2 σ = 0.2 σ = 0.1 σ = 0.01
Conclusion
steps 2&4 σ = 0.1 σ = 0.1 σ = 0.01 σ = 0.2 σ = 0.2
penalty 1 000 10 000 1 000 1 000 1 000
39. Solutions from the case study
inspection
optimization
case winner solution vector TIC
for MSPS
Sofie Van
A N N N N 45 900
Volsem
10.060 25 40.405
B S9.940 0 N N F39.595 67 255
Introduction
MSPS
10.012 40.405
C F9.988 N N F39.595 102 590
Inspection
Cost
10.210 100 40.402 50
D S9.790 1 N N S39.592 1 133 450
Process model
Method
10.071 31.434 40.403
E F9.929 N F28.566 F39.5957 178 940
Finding solutions
Part 1: TIC
40.417 25
Part 2: EA
10.166
F F9.834 N N S39.583 0 102 935
Conclusion
10.034 25 40.406
G S9.966 0 N N F39.594 138 015
10.165 100 30.425
H S9.835 1 N F29.575 N 72 550
40.418
I N N N F39.582 73 520
40.411
J N N N F39.589 58 840
40. Further research
inspection
optimization
for MSPS
Sofie Van
Volsem
Suggestions:
Introduction
MSPS
Extensions to the current EA
Inspection
Cost
non-sequential MSPS
Process model
imperfect inspection
Method
Finding solutions
variable number of simulation runs
Part 1: TIC
Part 2: EA
further development of standard test sets
Conclusion
validation through real life case studies
41. inspection
optimization
for MSPS
Sofie Van
Volsem
Joint optimization of all inspection
Introduction
parameters for multi-stage processes:
MSPS
Inspection
algorithm, simulation and test set
Cost
Process model
Method
Finding solutions
Part 1: TIC
Sofie Van Volsem
Part 2: EA
Conclusion
Department of Industrial Management
Ghent University
Bruges, April 15, 2009