1. Agile Methods
Cost of Quality
Benefits of Testing Early & Often
Dr. David F. Rico, PMP, ACP, CSM
Twitter: @dr_david_f_rico
Website: http://www.davidfrico.com
LinkedIn: http://www.linkedin.com/in/davidfrico
Facebook: http://www.facebook.com/profile.php?id=1540017424
Dave’s Agile Articles: http://davidfrico.com/agile-message.doc

2. Author Background
 DoD contractor with 28+ years of IT experience
 B.S. Comp. Sci., M.S. Soft. Eng., & D.M. Info. Sys.
 Large gov’t projects in U.S., Far/Mid-East, & Europe
2
 Published six books & numerous journal articles
 Adjunct at George Wash, UMBC, UMUC, Argosy
 Agile Program Management & Lean Development
 Specializes in metrics, models, & cost engineering
 Six Sigma, CMMI, ISO 9001, DoDAF, & DoD 5000
 Cloud Computing, SOA, Web Services, FOSS, etc.

3. Today’s Whirlwind Environment
3
Overruns
Attrition
Escalation
Runaways
Cancellation
Global
Competition
Demanding
Customers
Organization
Downsizing
System
Complexity
Technology
Change
Vague
Requirements
Work Life
Imbalance
Inefficiency
High O&M
Lower DoQ
Vulnerable
N-M Breach
Reduced
IT Budgets
81 Month
Cycle Times
Redundant
Data Centers
Lack of
Interoperability
Poor
IT Security
Overburdening
Legacy Systems
Obsolete
Technology & Skills
Pine, B. J. (1993). Mass customization: The new frontier in business competition. Boston, MA: Harvard Business School Press.
Pontius, R. W. (2012). Acquisition of IT: Improving efficiency and effectiveness in IT acquisition in the DoD. Second Annual
AFEI/NDIA Conference on Agile in DoD, Springfield, VA, USA.

4. Traditional Projects
4
 Big projects result in poor quality and scope changes
 Productivity declines with long queues/wait times
 Large projects are unsuccessful or canceled
Jones, C. (1991). Applied software measurement: Assuring productivity and quality. New York, NY: McGraw-Hill.
Size vs. Quality
DefectDensity
0.00
3.20
6.40
9.60
12.80
16.00
0 2 6 25 100 400
Lines of Code (Thousands)
Size vs. Productivity
CodeProductionRate
0.00
1.00
2.00
3.00
4.00
5.00
0 2 6 25 100 400
Lines of Code (Thousands)
Size vs. Requirements Growth
Percentage
0%
8%
16%
24%
32%
40%
0 2 6 25 100 400
Lines of Code (Thousands)
Size vs. Success
Percentage
0%
12%
24%
36%
48%
60%
0 2 6 25 100 400
Lines of Code (Thousands)

5. Global Project Failures
5
Standish Group. (2010). Chaos summary 2010. Boston, MA: Author.
Sessions, R. (2009). The IT complexity crisis: Danger and opportunity. Houston, TX: Object Watch.
 Challenged and failed projects hover at 67%
 Big projects fail more often, which is 5% to 10%
 Of $1.7T spent on IT projects, over $858B were lost
16% 53% 31%
27% 33% 40%
26% 46% 28%
28% 49% 23%
34% 51% 15%
29% 53% 18%
35% 46% 19%
32% 44% 24%
33% 41% 26%
0% 20% 40% 60% 80% 100%
1994
1996
1998
2000
2002
2004
2006
2008
2010
Year
Successful Challenged Failed
$0.0
$0.4
$0.7
$1.1
$1.4
$1.8
2002 2003 2004 2005 2006 2007 2008 2009 2010
Trillions(USDollars)
Expenditures Failed Investments

6. Requirements Defects & Waste
6
Sheldon, F. T. et al. (1992). Reliability measurement: From theory to practice. IEEE Software, 9(4), 13-20
Johnson, J. (2002). ROI: It's your job. Extreme Programming 2002 Conference, Alghero, Sardinia, Italy.
 Requirements defects are #1 reason projects fail
 Traditional projects specify too many requirements
 More than 65% of requirements are never used at all
Other 7%
Requirements
47%
Design
28%
Implementation
18%
Defects
Always 7%
Often 13%
Sometimes
16%
Rarely
19%
Never
45%
Waste

7. Network
Computer
Operating System
Middleware
Applications
APIs
GUI
How Agile Works
 Agile requirements implemented in slices vs. layers
 User needs with higher business value are done first
 Reduces cost & risk while increasing business success
7Shore, J. (2011). Evolutionary design illustrated. Norwegian Developers Conference, Oslo, Norway.
Agile Traditional
1 2 3 Faster
 Early ROI
 Lower Costs
 Fewer Defects
 Manageable Risk
 Better Performance
 Smaller Attack Surface
Late 
No Value 
Cost Overruns 
Very Poor Quality 
Uncontrollable Risk 
Slowest Performance 
More Security Incidents Seven Wastes
1.Rework
2.Motion
3.Waiting
4.Inventory
5.Transportation
6.Overprocessing
7.Overproduction
MINIMIZES MAXIMIZES
 JIT, Just-enough architecture
 Early, in-process system V&V
 Fast continuous improvement
 Scalable to systems of systems
 Maximizes successful outcomes
 Myth of perfect architecture
 Late big-bang integration tests
 Year long improvement cycles
 Breaks down on large projects
 Undermines business success

8. What is Agile Testing?
 Traditional testing is a late, manual process
 Agile testing is an early and automated process
 The goal of agile testing is to deliver early and often
8
Rico, D. F. (2012). Agile testing resources. Retrieved Sep. 9, 2012, from http://davidfrico.com/agile-testing-resources.txt
Crispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.
Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.
Traditional Testing
Combining source files
Combining software and environment
Combining software and data
Combining software and tests
Combining developers
Agile Testing
Code is frequently checked in
Code is automatically retrieved
Compilation is done automatically
Tests are done automatically
Code reports are generated
Developers get instant feedback
Code is automatically deployed or
packaged for delivery

9. Thousands of Tests
Continuously Executed
No More Late Big
Bang Integration
Agile Testing Model
 User needs designed & developed one-at-a-time
 Changes automatically detected, built, and tested
 System fully tested and deployed as changes occur
9Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.
Build
Integration
Server
Version
Control
Server
Build
Scripts
UsesWatches
Build
Status
ProvidesDeveloper A
Developer B
Developer C
Commits
Changes
Commits
Changes
Commits
Changes
Builds
Database
Analysis
Testing
Reporting
Documentation
Deployment
Early, Automated, Fast,
Efficient, & Repeatable
Constant Readiness
State & CM Control
Lean, Waste Free, Low WIP,
No Deadlocked Test Queues
Rapidly & Successfully
Dev. Complex Systems

10. Agile Testing Done Early & Often
 Eliminates big-bang integration in the 11th hour
 Creates a repeatable and reliable testing process
 Evaluates system-wide changes throughout project
10Maeda, M. K. (2009). Agile testing: Early, often, and Smart. Arlington, MA: Cutter Consortium.

11. Agile Testing Practices
 Agile testing consists of seven broad practices
 Includes automated builds, testing, inspections, etc.
 Also includes reporting, documentation, deployment, etc.
11
Practice
Building
Database
Inspections
Testing
Feedback
Documentation
Deployment
Description
Frequently assembling products and services to ensure delivery readiness
Frequently generating/analyzing database schemas, queries, and forms
Frequently performing automated static analysis of product/service quality
Frequently performing automated dynamic product and service evaluation
Frequently generating automated status reports/messages for all stakeholders
Frequently performing automated technical/customer document generation
Frequently performing automated delivery of products/services to end users
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.

12. Agile Testing Workflow
12
Traditional vs. Agile Cumulative Flow
Work(Story,Point,Task)orEffort(Week,Day,Hour)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Work(Story,Point,Task)orEffort(Week,Day,Hour)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Traditional Cumulative Flow Agile Cumulative Flow
 Late big bang integration increases WIP backlog
 Agile testing early and often reduces WIP backlog
 Improves workflow and reduces WIP and lead times
Anderson, D. J. (2004). Agile management for software engineering. Upper Saddle River, NJ: Pearson Education.
Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.

13. Agile Testing Costs & Benefits
Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.
Fredrick, J. (2008). Accelerate software delivery with continuous integration and testing. Japanese Symposium on Software Testing, Tokyo, Japan.
 Most agile testing tools are “free” open source
 A build server is no more than a commodity PC
 10x more efficient/effective than traditional testing
13

14. Agile Testing Economics
 Traditional testing finds a defect in about 10 hours
 Manual code inspections find a defect in 1 hour
 Agile testing finds a defect every 6 minutes
14
Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.

15. Agile Cost of Quality (CoQ)
 Agile testing is 10x better than code inspections
 Agile testing is 100x better than traditional testing
 Agile testing is done earlier “and” 1,000x more often
15
Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.

16. Agile Testing Statistics
 Fewer builds leave in higher bug counts
 A high number of builds eliminates the defects
 Goal is to have as many, early builds as possible
16
Lacoste, F. J. (2009). Killing the gatekeeper: Introducing a continuous integration system. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 387-392.
 

17. Scaling Agile Testing
 Agile testing slows down with very large systems
 Slow testing slows integration and increases bugs
 Agile testing can speed back up with proper attention
17
Kokko, H. (2009). Increase productivity with large scale continuous integration. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.
Wide Impact Tuning
Fast builds – less changes – more green
Parallelization of test runs
ClearCase to subversion
Pre-installing as much as possible
Removal of randomness
Compilation in memory
Installation starting parallel with system
build
Focused Impact Tuning
More memory and CPUs
Parallelize builds
Replace 3rd party test libraries
Reduce/remove timeouts in tests
Select different tests
Refactor code & components
Tune the network & software
Tune the database

18. Agile Cost & Benefit Analysis
 Costs based on avg. productivity and quality
 Productivity ranged from 4.7 to 5.9 LOC an hour
 Costs were $588,202 and benefits were $3,930,631
18
Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods: Maximizing ROI with just-in-time processes and documentation.
Ft. Lauderdale, FL: J. Ross Publishing.
d1 = [ln(Benefits  Costs) + (Rate + 0.5  Risk2)  Years]  Risk   Years, d2 = d1  Risk   Years
 
5
1i

19. Benefits of Agile Methods
 Analysis of 23 agile vs. 7,500 traditional projects
 Agile projects are 54% better than traditional ones
 Agile has lower costs (61%) and fewer defects (93%)
Mah, M. (2008). Measuring agile in the enterprise: Proceedings of the Agile 2008 Conference, Toronto, Canada.
Project Cost in Millions $
0.75
1.50
2.25
3.00
2.8
1.1
Before Agile
After Agile
61%
Lower
Cost
Total Staffing
18
11
Before Agile
After Agile
39%
Less
Staff
5
10
15
20
Delivery Time in Months
5
10
15
20
18
13.5
Before Agile
After Agile
24%
Faster
Cumulative Defects
625
1250
1875
2500
2270
381
Before Agile
After Agile
93%
Less
Defects
19





20. Agile vs. Traditional Success
 Traditional projects succeed at 50% industry avg.
 Traditional projects are challenged 20% more often
 Agile projects succeed 3x more and fail 3x less often
Standish Group. (2012). Chaos manifesto. Boston, MA: Author.
20
Agile Traditional
Success
42%
Failed
9%
Challenged
49%
Success
14%
Failed
29%
Challenged
57%

21. Hoque, F., et al. (2007). Business technology convergence. The role of business technology convergence in innovation
and adaptability and its effect on financial performance. Stamford, CT: BTM Institute.
21
 Study of 15 agile vs. non-agile Fortune 500 firms
 Based on models to measure organizational agility
 Agile firms out perform non agile firms by up to 36%
Benefits of Organizational Agility

22. Agile Recap
 Agile methods DON’T mean deliver it now & fix it later
 Lightweight, yet disciplined approach to development
 Reduced cost, risk, & waste while improving quality
22
Rico, D. F. (2012). What’s really happening in agile methods: Its principles revisited? Retrieved June 6, 2012, from http://davidfrico.com/agile-principles.pdf
Rico, D. F. (2012). The promises and pitfalls of agile methods. Retrieved February 6, 2013 from, http://davidfrico.com/agile-pros-cons.pdf
Rico, D. F. (2012). How do lean & agile intersect? Retrieved February 6, 2013, from http://davidfrico.com/agile-concept-model-3.pdf
What How Result
Flexibility Use lightweight, yet disciplined processes and artifacts Low work-in-process
Customer Involve customers early and often throughout development Early feedback
Prioritize Identify highest-priority, value-adding business needs Focus resources
Descope Descope complex programs by an order of magnitude Simplify problem
Decompose Divide the remaining scope into smaller batches Manageable pieces
Iterate Implement pieces one at a time over long periods of time Diffuse risk
Leanness Architect and design the system one iteration at a time JIT waste-free design
Swarm Implement each component in small cross-functional teams Knowledge transfer
Collaborate Use frequent informal communications as often as possible Efficient data transfer
Test Early Incrementally test each component as it is developed Early verification
Test Often Perform system-level regression testing every few minutes Early validation
Adapt Frequently identify optimal process and product solutions Improve performance

















23. Conclusion
23
 Agility is the evolution of management thought
 Confluence of traditional and non-traditional ideas
 Improve performance by over an order of magnitude
“The world of traditional methods belongs to yesterday”
“Don’t waste your time using traditional methods on 21st century projects”
Agile methods are …







Systems development approaches
New product development approaches
Expertly designed to be fast and efficient
Intentionally lean and free of waste (muda)
Systematic highly-disciplined approaches
Capable of producing high quality systems
Right-sized, just-enough, and just-in-time tools
 Scalable to large, complex mission-critical systems
 Designed to maximize business value for customers
Wysocki, R.F. (2010). Adaptive project framework: Managing complexity in the face of uncertainty. Boston, MA: Pearson Education.
 

24. Books on ROI of SW Methods
 Guides to software methods for business leaders
 Communicates business value of software methods
 Rosetta stones to unlocking ROI of software methods
 http://davidfrico.com/agile-book.htm (Description)
 http://davidfrico.com/roi-book.htm (Description)
24