This is a presentation made at the VanQ (www.vanq.org) meeting on Feb.28, 2013. CMMI looks a very complex and expensive model which is hard in implementing and even harder in maintenance. The presentation intends to show how elements of CMMI high maturity practices could be utilized by a testing team to make its work more predictable and more stable. CMMI high maturity levels expect critical processes be statistically managed. Quantitative objectives are established for quality and performance at the organizational and project levels. Process baselines are created and maintained for key organization and project quality and performance indicators. Testing gives a wide range of opportunities to collect and analyze metrics and statistical parameters. Several examples from my experience in CMMI Level 5 company are shared process performance baselines (PPB) and process performance models (PPM) are discussed: how to create and maintain PPB; how PPB can help in everyday project activities, particularly in setting and monitoring project measurable goals; what is PPM; workable PPM samples for software testing indicators.

1. Mikhail Pavlov
Process Performance
Baselines and Models
CMMI High Maturity Practices in
Software Testing

2. Hello!
My name is Mikhail Pavlov
60 minutes silent mode questions
2

3. Disclaimer
All numbers, charts and trends in this
presentation are used solely for illustrative
purposes.
Samples of measurements are not based
either on industry or a certain company
project results.
All coincidences with real measurements
results and companies are accidental.
3

4. Software Testing
Background
in software testing since 1988
defended Ph.D. thesis on compiler testing 1994
test lead/test analyst 2000-2004
quality manager 2009-2012
translated two software bestsellers into Russian
ISTQB FL certified 2011
4

5. Experience in CMMI
participated in 4 official CMMI Appraisals in roles of:
test lead
head of training organization
appraisal team member
consultant
implemented CMMI high maturity requirements in 3
organizations
Certified Internal Appraisal Team Member
5

6. What is CMMI?
Designed for software
engineering organizations
Current version 1.3
Three constellations
CMMI for Development
CMMI for Acquisition
CMMI for Services
6

7. CMMI Maturity Levels
7

8. CMMI Can Help:
High Maturity Process Areas
Level 4
Organization Process Performance
Quantitative Project Management
Level 5
Causal Analysis and Resolution
Organization Performance
Management
8

9. What we will be talking
about?
SIMPLE RULES SIMPLE SAMPLES
9

10. What And How Do You
Measure?
Elementary
measurements
Compound
measurements
Indicators/KPIs
10

11. Why Do You Collect
Measurements?
Do you know how your team works?
Can you predict a level of your product
quality?

12. Three Simple Rules
Set a goal
Build/find a model
Separate common
variations from special
12

13. 0
100
200
300
400
500
600
700
0
1
No. of defects
100
50
0
50
100
150
200
250
0
2
Le
1
3
ga
c
Bu y
1
4
ild
_
92 96 100 108
Bu 0_
3
ild IBM
_
5
Bu 7_
ild 0_
_ 0
7
6
Bu 7_ 01
ild 0_
_ 0
7
Bu 7_ 02
ild 0_
132 143 149
_ 0
Bu 7_ 03
8
ild 0_
_ 0
Bu 7_ 04
ild 0_
9
_ 0
173 194
Bu 7_ 05
ild 0_
_ 0
10
Bu 7_ 06
ild 0_
_ 0
11
Bu 7_ 07
ild 0_
_ 0
8 16 16 17 24 32
Bu 7_ 07
ild 0_ _Le
_ 0
Bu 7_ 08 gac
ild 0_ y
_ 0 205 215 216 224
Bu 7_ 09
ild 0_
_ 0
Bu 7_ 10
ild 0_
_ 0
63 90 90
Bu 7_ 11
247 248
ild 0_
_ 0
103
Bu 7_ 13
ild 0_ _B
_ 0
Bu 7_ 13
ild 0_ _Le
0
260 269
_
Bu 7_ 14 gac
y
154 164
ild 0_
_ 0
314
Bu 7_ 15
ild 0_
_ 0
Bu 7_ 17
ild 0_
_ 0
Bu 7_ 18
349 357
ild 0_
_ 0
Bu 7_ 19
ild 0_
Not closed defects
_ 0
Bu 7_ 20
ild 0_
_ 0
Bu 7_ 21
ild 0_
Testing?
_ 0
379 382 402
Bu 7_ 22
ild 0_
_ 0
Bu 7_ 23
ild 0_
_ 0
188 197 199 200 205 211 211
Bu 7_ 23
ild 0_ _Le
0
Ratio of Closed defects to Acknowledged defects
_
220
Bu 7_ 24 gac
y
414 418 438
ild 0_
_ 0
Bu 7_ 25
Verified
ild 0_
_ 0
Bu 7_ 26
278 304
ild 0_
_ 0
Bu 7_ 27
ild 0_
_ 0
When Can I Stop
364
New
456 460 480
Bu 7_ 28
ild 0_
_ 0
400
Bu 7_ 29
510
ild 0_
_ 0
Bu 7_ 30
Verified
0_
567
ild
_ 0
Bu 7_ 31
464 490
ild 0_
_ 0
Not Closed
Bu 7_ 33
ild 0_
_7 03
_0 4
_0
35
Acknowl'd total
506 515 524 525
12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
599 610 624 625 625

14. Sample for Quick
Analysis
30
22.5
Defect density (DD) =
15 number of defects /
added product size
7.5
0
1 2 3 4 5 6 7 8
14

15. Separate Common from
Special
variations
common causes -
system behavior
special causes -
subject for root-
cause analysis
(ultra performance,
problems, influence
of external factors,
or etc.)
15

16. Control Chart 1/3 XmR Chart
30.0
25.51
22.5
15.0
11.88
7.5
0 0
1 2 3 4 5 6 7 8
DD LCL Mean ULC
Mean = (x1+...xn)/n
R(i)=|X(i+1)-X(i)| - moving range Sigma - 68.27%
Rav=(R(1)+...+R(n-1))/(n-1) 2Sigma - 95.45%
UCL(X)=Mean + 2.66*Rav 3Sigma - 99.73%
LCL(X)=Mean - 2.66*Rav
16

17. Control Chart 2/3
XmR Chart
30.0
22.5
15.0
7.5
0
1 2 3 4 5 6 7 8
-2sigma DD LCL ULC
-sigma Mean +sigma +2sigma
Assuming that a stochastic variable has a normal
distribution we have sigma equal to standard deviation
17

18. Control Chart 3/3
Special reason or assignable cause sample
Assignable causes require additional analysis
DRE=PreUAT/
(PreUAT+UAT)
18

19. Assignable (Special)
Causes of Variation
There are 8 main types of assignable causes
Point outside control limits
Nine points in a row on one side of the central line
Six points in a row steadily in(de)creasing
Fourteen points in a row alternating up and down
...
Every assignable cause is a subject for analysis
19

20. Process Performance
Baseline
A snapshot of a set of process performance
indicators which characterize process or
subprocess behavior
Based on statistical elaboration and segregation
of project data
statistical and probability models
categorization of objects under measurements
20

21. Single Malt vs Blended
Scotch
21

22. Process Performance
Baseline: PDDD (sample)
Project
Project Type Platform Indicator Unit Mean LCL UCL
Size
Defect/
Blend Middle Java 1.32 0 4.71
KSLOC
Defect/
Blend Small Other 1.39 0 3.58
KSLOC
Defect/
Dev Middle .Net 1.21 0 4.21
KSLOC
Defect/
Maintenance Small Java 0.35 0 0.89
KSLOC
22

23. Key PPB Rules
Defined for statistically stable process
Recalculated on regular basis until historical
data are not “enough”
Needs redefinition from scratch if the process
it determines is significantly changed
23

24. Process Improvements
Narrowing distribution
Shift of the Mean
range
24

25. PPB: Shift Sample
PDDD
PPB FOR PDDD
5.00
3.00
3.75 2.25
2.50 1.50
1.25 0.75
0
0 Release 11 13 15 17
Release 2 4 6 8 10 12 14 16 18
DD LCL Mean ULC
Mann-Whitney Test:
verifies a hypothesis that two sets of data come from
different populations
25

26. Summary: Process
Performance Baseline
Based on historical data
Mathematical statistics is used to create
Defines natural process range of an indicator
Assists in project estimation and planning,
monitoring and control and process improvement
26

27. Process Performance
Models
Process-performance models are used to
PREDICT the value of a process-performance
measure from the values of other process,
product, and service measurements and
CONTROL the process outcome
Process-performance models are based on
probabilistic nature of software development
process
27

28. How It Works
Ask question
about your
observations
Analyze data
and conclude Create working
(is hypothesis hypothesis
true)?
Collect data
and use
controllable
factors
28

29. PPM Sample: Prediction
of Delivered Quality
Goal - ensure that a level of product quality
would be as expected by customer
Model - find measures which can be obtained
before delivery and their relation with the
aimed measure
29

30. Mapping Goals and
Forming Working
Hypothesis
Customer-defined KPI - post-delivery defect
density should be not greater than 5
Internal measurable objective is PDDD < 4.31
Analyze relationship between PDDD and DD
(defect density before delivery)
30

31. Create Model: Input
DD XmR Chart
20
Release DD PDDD AC Excl
1 5.32 0.07
15
2 6.56 0.27
3 8.61 1.14
10
4.0 8.25 1.06
4.1 8.27 0.80
5
5.0 6.33 1.12
5.1 13.00 1.67
0
1 2 3 4 4.1 5 5.1 5.1.1 6 6.1 7 5.1.1 8.67 0.67
DD LCL Mean UCL 6 19.06 1.55 Yes Yes
6.1 10.28 0.42
7 13.49 1.22
31

32. Create Statistics-Based
Model
Build plot
Select linear as criterion function
Calculate linear function’s
coefficients
Hypothesis is verified (regression
analysis) - 71% is fine
Calculate predicted values using
the obtained function
If DD does not exceed 15.2 a
predicted PDDD value would not
exceed 4.31 with 95% probability
32

33. Monitor, Control and
Analysis
Acknowledged Size Added
Build Id defects (cumulative) (cumulative) DD (current) Flag raised
Build_8_001 99 5832 16.98 Yes
Build_8_011 233 17116 13.61
Build_8_021 396 37872 10.46
Build_8_031 637 48268 13.2
Build_8_036 655 54002 12.13
Post delivery defect density for
Release 8.0 = 1.98
33

34. Summary
Testing measurements are an empirical and scientific assistance in resolution of
problems in software development process
When performing measurements
do not forget to set a goal;
build a model to control its parameters;
separate common causes from special ones
Use process performance baseline
to know how you work
to estimate and monitor project execution
Create a process performance model
if you want to predict and control parameters on the phase when they
cannot be measured

35. See you again!
In April at yvrTesting session
I will present
“Forgotten? Ignored? Obsolete?”
Static Testing Techniques

36. Thank you!
Mikhail Pavlov
email mikhail.pavlov@gmail.com
LinkedIn http://www.linkedin.com/
in/mikhailpavlov
Twitter @m__pavlov
phone 604 4403601
36