3. Problem Statement Introduction
Quality improvement needed in many organizations
Business case
• Identification of problem areas
• Selected improvement
• Decision
Quantified
• Costs & benefits
• Lead time to result
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 3
3
4. Quantification problems Introduction
Much time needed to gather data
Difficult to measure things
Hard to keep management commitment
Expensive
Required: Business case, with limited but sufficient
measurement effort, to gain management
commitment and funding
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 4
4
5. Affiliate Collaboration Introduction
SEI Pittsburgh, PA:
Software Engineering Measurement & Analysis Group
Ericsson Netherlands:
Market Unit Northern Europe & Main R&D Center
The Software Engineering Institute Affiliate Program provides
sponsoring organizations with an opportunity to contribute their best
ideas and people to a uniquely collaborative peer group who combine
their technical knowledge and experience to help define superior
software engineering practices.
Affiliates: http://www.sei.cmu.edu/collaborating/affiliates/affiliates.html
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 5
5
6. Two models Introduction
Defect Estimation Model Resident Defects in
Design Base
• Data, tuned with expert opinion Design Process
Competence, skills
Defects Inserted
(documentation,
Defect Density
Tools, environment code)
Estimate Fault Slip Through
Detection Rate
• Test Process
Competence, skills
Test Capacity
Defects Detected
(Inspection, test)
Fault Slip Through
Tools, environment Defect Classification
• Project/Product Quality Resident Defects in
Delivered Product
(Un)happy customers Process
Inputs and outputs
Influencing factors
Defect Level Measurement
Quality Factor Model
• Expert opinion, extend with data
• Quick Quality Scan
• Prediction Fault Slip Through
• Improvement Areas
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 6
6
7. Measuring quality Business
Cases
Insertion: Where are defects made? How to prevent?
Detection: Where are defects found? Early/economic removal?
Quality: How many defect are left in the product at release?
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 7
7
8. Process View Business
Cases
Resident Defects in
Design Base
Design Process Defects Inserted Defect Density
Competence, skills (documentation,
Tools, environment code)
Detection Rate
Test Process
Competence, skills Defects Detected Fault Slip Through
Test Capacity (Inspection, test)
Tools, environment Defect Classification
Resident Defects in (Un)happy customers Process
Delivered Product Inputs and outputs
Influencing factors
Defect Level Measurement
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 8
8
9. Fault Slip Through Business
Cases
Lead
??? Time
Cost
??? FST
Quality
???
Fault Slip Through = Number of defects detected in integration
& customer test that should have been detected earlier
“Should” implies that the defect is more cost effective to find earlier.
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 9
9
15. Quality performance assessment Agile Req.
Survey based upon Quality Factors
• 34 respondents from management & technical roles
• 4 management areas & 7 technical areas
2 sub questions for each quality factor:
• How relevant is the factor when we want to improve quality?
“little if any,” “moderate,” “substantial,” or “extensive,”
• How well are we doing currently?
“poor,” “fair,” “good,” and “excellent.”
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 15
15
17. Pilot “Business Case for Quality” Agile Req.
Context:
• Process management
• Quality steering
• Starting with Agile
Pilot: Agile for Requirements
• Calculate value of process change
• Run the pilot
• Evaluate the result
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 17
17
18. Improve: Requirements Stability Agile Req.
Requirements Stability – Inverse of the amount of requirement
changes over time. (The less changes, the higher stability.)
Agile deployment
• Backlog with Prioritized User Stories
• Product manager as Product Owner
• (Pre-) Planning game
• Architecture team
• Stand up meetings
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 18
18
19. Improve: Scope Stability Agile Req.
Scope Stability – Impact of major changes in projects that are
related to changes in the product roadmap, including stability of
the products to be developed, development teams involved in
projects, and major changes in project funding or delivery
dates.
Agile deployment
• Backlog
• Responsibility of Agile teams and Product Owner
• (Pre-) Planning game
• Retrospectives
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 19
19
20. Improve: Requirement Definition Agile Req.
Capability
Requirements Definition Capability – The skill and experience level
of the people doing requirements definition (e.g., product
managers).
Agile deployment
• (Pre-) Planning game
• Stand up meetings
• Collaborative Culture
• Retrospectives
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 20
20
21. Steering Agile Quality Agile Req.
• Estimate latent defects after demo (planning game)
• Collect defects during test (after demo).
• Classify defects:
• “introduction phase“
• “should have been detected phase”
• Root cause analysis: Prevention
• Decide improvement actions and communicate
• Re-estimate and predict release quality.
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 21
21
22. Results Agile for Requirements Agile Req.
• Very low number of requirement defects
• Previous projects also had a low number
• Based upon the data no conclusion could be drawn
Root Cause Analysis:
• understanding requirements increased:
planning game & stand-up meetings.
• Improvements from retrospectives increased cooperation
between development team and product owner.
Requirements quality performance increased!
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 22
22
23. Conclusions Conclusions
Quicker Business Case:
• Quality Factors/Performance
• Fault Slip Through
• Combining data and expert opinion
Improved Requirements Performance
• Agile increased requirements quality
• Less defects after release
• Increased flexibility and collaboration
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 23
23
24. More information Conclusions
Publications:
• Building Process Improvement Business Cases
SEI Technical Note: http://www.sei.cmu.edu/library/abstracts/reports/09tn017.cfm
• Controlling Project Performance by Using the Project Defect Model
in proceedings PSQT West Conference 2005
• The Business Benefit of Root Cause Analysis
in proceedings SM/ASM conference 2003
• SPI, the agile way!
To be presented at the SPIder conference, october 2009
www.spiderconferentie.nl
Contact:
• Email: benlinders@gmail.com
• http://www.linkedin.com/in/benlinders
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 24
24
26. Solution Introduction
Technologies
• Bayesian Belief Networks (BBN)
• Monte Carlo Simulation
• Root Cause Analysis
• Cost of Quality, Defect Slippage
Six Sigma DMAIC Approach
• Modeling Business Cases
• Research Quality Factors & quantify Quality Improvement
• Validate “Business Case for Quality”
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 26
26
27. Building a business case Business
Cases
Quality Quality Fault
Quality
Factor Slip
BBN Quality
Factor
Phase
Through
Performance
Factor
Quality
Factor
Historical
Industry
Project
Data
Data
Monte Current Improved
Quality Phase Quality Phase
Carlo Performance Performance
Subjective
Expert
Opinion
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 27
27
28. Bayes Belief Network (BBN) Business
Cases
• Probabilistic graphical model, to model uncertainty
• Diagnose and explain why an outcome happened
• Predict outcomes based on insight to one or more factors
Used:
• Modeling Quality Factors
• Predicting Quality Phase Performance
• What if Scenario
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 28
28
29. Monte Carlo Simulation Business
Cases
• Compute a result based on random sampling
• Modeling distributions of data
• Can make uncertainty visible
Used:
• Calculate value of process changes
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 29
29
30. Quality Prediction Business
Cases
Current Model: Estimation
• Extrapolate past performance
• Based on inserted/detected defects
• Plan & track
Wanted: Prediction
• Causes of defects
• What if Scenarios All models are wrong
• Decision taking
Some models are useful
Deming
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 30
30
31. Step 2: Defect Prediction Business
Cases
Fault Slip Through
Defect found in a (later) test phase that should have been found earlier
“Should”: More Cost effective (economical)
Predict Defect Reduction
• Determine process impact
• Simulate quality change
• Predict savings
Pilots
• Agile
• Model Driven Development
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 31
31
32. Quantify Quality Improvement Quality
Factors
Connect defect data with Quality performance
• Maximum quality factor => Industry best in class
Published industry data from various sources
• Distribution: Linear (keep it simple)
Extend BBN to calculate remaining defects after each phase
Result: Model for “what if scenario’s”
• Calculate defects in release products, when quality performance improves
• Cost of Quality data to calculate savings
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 32
32
33. Monte Carlo: Quality performance Quality
Factors
Monte Carlo simulation
• Input from 5 experts
• Estimated chance of occurrence and impact on FST (1-5 scale)
• Simulation done to calculate impact on quality factors
• Result used in BBN model to calculate effect on defect slippage
Expected result:
• Reduced number of requirement defects introduced
• Increased effectiveness of late testing phases
• Less defects in products shipped to customers
• Cost saving:
— Limited saving in the project
— Major saving during maintenance
Agile Requirements Agile Consortium Benelux, sep 30, 2009 (C) Ben Linders 33
33
Notas do Editor
After an agile pilot, ETM decided to fully roll out the agile way of working. All projects have setup cross functional teams, some including I&V. Stakeholders like product managers, system managers and projects managers enable that teams can produce the right software. Support functions, like CM, QA, CPI, and managers, and process responsibles are also using agile techniques, to prioritize work and deliver value to the projects. So how does this impact process management? This presentation shows that processes are still important at ETM, but in a different way: agile integrated into enhanced streamline development. The main goal here is to get an "every employee improving our value stream and efficiency continuously" environment/culture.
Color coded agenda, so that the audience can see where we are. Intro: Problem statement, approach as SE affiliate, overview of technologies used Business Cases: Defect slippage as target, overview of the BBN/Monte Carlo model Quality Factors: Details of the model, what influences quality Pilot: Survey to determine improvement area (BBN), agile improvement (Monte Carlo) Conclusions: Is this approach useful, what did we learn?
Link between improvement and a business case: Investment/benefit
Why the current approach doesn’t work. What do we need to solve it?
Show quickly to let the audience know that this was a collaboration (maybe too much text?)
Position Fault Slip Through as main indicator of Quality, since: Accepted within Ericsson Indicates significant savings Can also be used to get cost/time savings
First sheet of the Quality Factors part: Zoom into factors that determine quality. This is the BBN model. Overview of the Quality phases: Management factors Defect Insertion Defect detection Background. This model has been made to investigate quality improvement at Ericsson R&D The Netherlands. It includes quality factors that most probably will have an impact on quality at that development site. It is by no means intended to be a complete model, and different factors may be applicable for other companies.
Zoom into the 4 areas of management factors Explain shortly why the influence of strategic and operational line management is indirect (via project/process)
Shortly show the main phases in software development where potentially defects are included.
Show the main phases where defects are detected. Explain that defects left is input for the defects slippage (main indicator of quality).
Show some detail on how the quality phase performance is build up, example on code inspection.
How the assessment was done: Survey based SEI survey tool Questions reviewed with the SEI Representation of compete R&D flow 2 axes: Relevant & performance
Show some detail on how the quality phase performance is build up, example on code inspection.
First sheet of the pilot part of the presentation. Explain the 2 steps: Assessment to determine potential areas of improvement (BBN) Actual improvement (agile requirements), calculate & validate results
Very low number of requirement defects, from which Some on not yet release functionality (implemented later) Some due to changes in platform software, could not be prevented Only 1 defect which could have been prevented in planning game
Mention the technologies used, and that the project was run in a Six Sigma way.
Animated sheet! BBN technology is used to model the quality phase performance, based up Quality factors. This is used to models the target that we focus upon: defect slippage. Historical and industry data have been used to quantify the relationship between quality phase performance and fault slippage. The BBN models current performance. Based upon expert opinion, using Monte Carlo, a calculation is done of the quality phase performance after the improvement. This is fed into the BBN to calculate the impact on defect slippage, and the potential savings.
Explain how the Quality Phase performance is linked to defect slippage, though the use of industry and historical data. This will probably trigger questions, propose to discuss them after the presentation (BoF or …)?