This presentation, given at QAI QUEST on 24 May 2018, describes the concept of testing debt during the development process, how to identify it, how to measure it, and how to remediate it.

1. Identifying and Measuring
Testing Debt
Peter Varhol

2. About me
• International speaker and writer
• Graduate degrees in Math, CS, Psychology
• Technology communicator
• Former university professor, tech journalist
• Cat owner and distance runner
• peter@petervarhol.com

3. Agenda
• What is testing debt?
• How do we accumulate testing debt?
• Identifying and measuring testing debt
• Ways to remediate testing debt
• Conclusions

4. Let’s Start With Technical Debt
• We write code
• It has to change later
• Don’t fully understand the problem/feature/requirement
• Have a need to deliver quickly
• Building a prototype
• It is kludgy/performs slowly/doesn’t handle edge cases
• It has to be fixed at some point to be useful
• That is technical debt

5. What is Testing Debt?
• In an agile project
• Testing often works at a different cadence than development
• When testing is deferred
• Dev used the entire sprint
• Not enough time or no resources
• No tests available
• Tests blocked by other issues
• That is testing debt

6. What is Testing Debt?
• The effort needed to find and fix the issues that remain in the code
when an application is released
• This is not the same as a groomed bug list
• The difference (in bugs or other issues) between what is expected and
what is delivered
• If we don’t identify and measure testing debt:
• We can’t assess our quality
• We can’t address the debt

7. Many Things Can Cause Testing Debt
• Poor project definition
• Incomplete preparation
• Lack of domain expertise
• Poor test case definition
• Lack of time
• Inability to automate

8. Development Can Also Cause Testing Debt
• End of sprint incomplete work
• Code changes and refactorings
• Refactorings are people too
• Incomplete unit tests
• Poor communication on development work

9. Testing Debt is Composed Of the Need For
• New and redesigned tests
• Retesting
• Regression testing
• Integration testing
• System-level test
• Bug fixes and verification

10. The Problem With Testing Debt
• Testing debt means something has to be paid back
• Additional testing effort
• Product quality
• With interest
• The longer you wait, the longer it takes
• And the more the interest accrues
• Are these correct assumptions?

11. Why Do We Have to Pay Back Debt?
• Loss of quality
• Uncertain of fitness for purpose
• Lacking confidence in critical features
• Hasn’t kept pace with business requirements

12. How Do We Know We Have Testing Debt?
• Unfinished testing tasks
• Uncertainty on the results of testing
• Lack of a clear picture of the state of quality

13. Identifying Testing Debt
• Indicators
• Tests are blocked
• This is indicative of technical debt in general
• The sprint ends before testing does
• Test results are ambiguous
• Or give incorrect results
• Tracing these back to debt
• In an ideal world, how long will it take?

14. Measuring Testing Debt
• It’s all about the metrics
• But what should we be measuring?
• Time
• Skills required
• Opportunity lost

15. Measuring Testing Debt
• Time
• How much time to complete known testing?
• Based on averages from previous tests
• Plus the need to fix defects identified
• Is known testing all there is?
• Do we need to figure out what we don’t know?

16. Measuring Testing Debt
• Skills required
• Some skills cost more than others
• Some skills are more difficult to acquire
• Skills may also have to be scheduled in
• Cost and time are involved, but are covered above
• Instead, exactly what skills are needed
• Automation
• Database
• Scripting
• Machine learning
• Reporting

17. Measuring Testing Debt
• Opportunity lost
• Paying back debt means other activities can’t be done
• More features
• Time to market
• Testing debt also requires coordination
• Development
• Documentation
• Product owner/users

18. What is the Interest on Testing Debt?
• What will you have to pay?
• Studies say each subsequent adds 10 percent interest
• Per sprint
• Testing debt interest can add up quickly
• When will you have to pay it?

19. Solutions to Testing Debt
• Avoidance
• Control scope
• Reduce testing times
• Removing legacy tests
• Time box testing

20. Avoidance
• Make sure testing tasks are complete when scheduled
• Make sure the solution is fully understood
• Problems with that
• Not practical
• Not enough time or resources
• May require rework

21. Control Scope
• Don’t let project goals/features change after initial requirements
• Problems with that
• Agility means adapting
• Projects change

22. Reduce Testing Times
• Spend less time testing
• Problems with that
• How do we reduce testing times?
• Without affecting quality?
• Automation can help
• But it still requires an investment in time and skills

23. Removing Legacy Tests
• Getting rid of tests that are no longer necessary
• Problems
• Are they now regression tests?
• Are they really not necessary?
• How do we determine that?

24. Time-Box Testing
• Strictly limit testing times
• Testing work is a subset of needed work
• Only addresses high-priority debt

25. Managing and Triaging Testing Debt
• Use planning poker on debt
• Provide a team-based assessment on the cost of addressing it
• Also use planning poker on importance
• How important is it to recover this debt?
• Scheduling a sprint to pay off debt
• These are often called “hardening sprints”

26. Summary
• Teams have difficult decisions to make about testing debt
• They can ignore it at their peril
• Or
• They can acknowledge it
• Track and measure it
• Decide whether or not to do something about it

27. Thank you
• Peter Varhol
• peter@petervarhol.com
• @pvarhol

28. References
• http://it-cisq.org/standards/technical-debt/