1. Agile Development:
The Importance of Testing
Presented by
Bruce D. Rosenblum
CEO
Inera Incorporated
Society for Scholarly Publishers
30 May 2008

2. Real World Software Development
Credit: http://www.eos-india.net/abhisek/blog/softwaredevcycle-2006-06-14-20-38.html
© 2008, Inera Inc. All Rights Reserved

3. Software Development Goal
 Develop Applications
• Deliver value to the user
• Easy to use
• Provide reliable process
© 2008, Inera Inc. All Rights Reserved

4. The Old Days: Waterfall Development
Scope Change
Feature Creep
Credit: http://www.csse.monash.edu.au/~jonmc/CSE2305/Topics/07.13.SWEng1/html/text.html
© 2008, Inera Inc. All Rights Reserved

5. Why Waterfall No Longer Works
 Computers are more powerful
 Therefore projects are bigger
• And take longer to develop
 No longer possible to anticipate all requirements
• And cannot anticipate all changes during project
 By delivery time
• Customer interests changed
• Requirements changed
 “What I asked for, but not what I want”
© 2008, Inera Inc. All Rights Reserved

6. Web Flexible World
 Old distribution model
• Disk shipments
• Waterfall development
• Slow, late deliveries
 New distribution model
• Internet delivery
• Automatic updates of client applications
• Continual update of ASP applications
© 2008, Inera Inc. All Rights Reserved

7. New Development Methods
 Customer-centric
 Incremental changes
 Rapid Deliveries
 Continuous Testing
© 2008, Inera Inc. All Rights Reserved

8. Agile Model
© 2008, Inera Inc. All Rights Reserved

9. Agile Goals
 “Agile” means responsive to customer needs
 The goal is the customer, not the “project”
 Simplicity and simple steps
© 2008, Inera Inc. All Rights Reserved

10. Getting Started
 Design simple task that does “something useful”
• Even if that task won’t be delivered to “customer”
 Start testing immediately
 Test early and often
© 2008, Inera Inc. All Rights Reserved

11. Team Test Types
 Unit tests
• Test assumptions; vital for maintenance
 Integration tests
• Does it all work together correctly?
 Code Reviews
• Developers can learn from each other
© 2008, Inera Inc. All Rights Reserved

12. Developers Can’t Test
 Developers are lousy testers
 Developers lie
• Is it done yet?
• When will it be done?
 Great developers test 2/3 of their time
• Write a little, test a lot
© 2008, Inera Inc. All Rights Reserved

13. Customer Test Types
 Formal Quality Assurance
• Scripted test plans
 Ad hoc testing
• Sometimes called “monkey testing”
 Usability testing
• One-way mirrors
 User testing
• The importance of watching users
 “Run it yourself” testing
© 2008, Inera Inc. All Rights Reserved

14. Test Administration
 Bug database
• TestTrack, Bugzilla, etc.
• FileMaker, Excel
 Short Cycles
• Daily bug review
• Fix bugs immediately
− New features must wait
© 2008, Inera Inc. All Rights Reserved

15. Automated Testing
 Use of software to control testing
 Automatically run application features
 Results compared to previous “good”
result
© 2008, Inera Inc. All Rights Reserved

16. Use of Automated Testing
 Inera relies heavily on automated testing
 Three months of development over 6 years
 Today: 30+ hours of automated testing per night
 Manual validation performed on all releases
© 2008, Inera Inc. All Rights Reserved

17. Example: CrossRef Testing
 Inera
implemented CrossRef query module
 Automated testing of CrossRef module
• Queried ~1000 references per night
• Compared DOIs with previous “good” results
 Sometimes links go away
• Usually incorrect deposit (e.g. metadata conflict)
 Sometimes new links appear
• Replaces previous “good” results
© 2008, Inera Inc. All Rights Reserved

18. Impact of Automated Testing
 New problems caught early
• Problems found quickly when code is changed
 Stability of code maintained
 Fixes are never “lost”
 Short release cycles
© 2008, Inera Inc. All Rights Reserved

19. Cost of Automated Testing
 Sustainable and scalable model
• Up-front investment yields large dividends
 Lower costs
• QA cost ~ $60,000/person/year
• Versus 30+ hours of PC time per night
− 4 PCs + electricity + 3 months engineering time
 No fulltime QA personnel
• Manual testing performed by multiple team members
© 2008, Inera Inc. All Rights Reserved

20. Time for Testing
 Testing does NOT take extra time
• Even manual testing
 Testing reallocates time
• Less time spent tracking latent bugs
• Less time reworking code to fix bugs
 Great testing allows faster releases
© 2008, Inera Inc. All Rights Reserved

21. “True” Cost of Automated Testing
 Faster Release cycles
• Can release latest code daily if stable
 Fewer support calls
 Happier customers
© 2008, Inera Inc. All Rights Reserved

22. Who Tests?
 Developers
 Quality Assurance Specialists
 Customer
 Actual Users
 Machines (automated testing)
 So who tests? The Agile Team
© 2008, Inera Inc. All Rights Reserved

23. Conclusions
 System stability requires constant testing
 Great testing lowers costs
 Great testing delivers software faster
© 2008, Inera Inc. All Rights Reserved

24. Questions?
Bruce Rosenblum
Inera Incorporated
+1 (617) 969 - 3053
brosenblum@inera.com
www.inera.com
© 2008, Inera Inc. All Rights Reserved