Continuous delivery makes it possible to develop and deploy software within hours instead of months or weeks or days. But what about testing? Testing is not fast. Which means you will have to cut corners, but add additional measures to compensate. Many companies market their continuous delivery practices, so others can see how cool they are, but also learn from their experiences. In this talk I will go through some highlights and things that work well and not so well in these example continuous delivery implementations, and then will concentrate on testing patterns, approaches and little hints that I have learnt both from these examples and from my own experience.

1. Testing in Continuous Delivery
Aistė Stikliūtė, Agile turas 2015

2. Agenda
What is Continuous Delivery
and why it’s (slowly) going
mainstream
Examples
Challenge - testing
• Problems
• Solutions

3. What is Continuous Delivery
C.D.
Build
Deploy
Test
Release

4. Do you do Continuous Delivery?
How often do you / can you deploy to Prod?
What if you’ve got to do a big change?
Who decides when to deploy / release?

5. Some examples

7. … examples

8. The testing challenge
Months → Weeks → Days → Hours → Minutes

9. Have the cake and eat it too?
● do thorough regression testing
and do it often?
● add changes fast
and keep up with automated tests?
● ensure good quality
without (much) manual testing?

10. Cycle time: idea  production
Design Develop Test
Merge to
trunk
Deploy to
test
environment
Test
Deploy to
stage
Test
Deploy to
production

11. Cycle time to production: testing
Manual tests
before merge
Automated
tests after
merge
Staging tests
Analyse results
Update tests
Bugfix
Clarify open questions
Bugfix
Analyse results
Bugfix

12. Solve the testing challenge
Patterns and ideas

13. [obvious] agile testing
Testers involved in team:
grooming, planning, development
“Show me”
testing before it’s finished, input for automated tests
When it’s finished, tester must be ready to test

14. [obvious] developer testing
Develop Test

15. developer responsibility
You are responsible for your code
• Testing
• Deploying
• Supporting
• Defined code ownership
• Better automated tests
• Monitoring. Faster corrections
• Faster & more successful deployments

16. branching patterns
Centralized Finds and fixes problems ASAP
Distributed Develop every change in its own branch and test
Temporary
branches
Effective, combines centralized and distributed
Matrix of
services
Use centralized CI for a massively scalable system

17. test automation
Fast tests Reliable tests

18. test automation
Test
optimization
Well-thought
infrastructure
Run in
parallel
Test
categories

19. test automation
Test
coverage
Test
environment
Keep tests
green
“Pass” ratio

20. testing in production
A/B
testing
feature
toggling
monitoring

21. feature toggling
● Needs to be implemented
● Reduces need for lab-testing, but it’s still needed
● Needs to be tested itself
● Reduces risk
● Enables A/B testing
on off staff user_list opt-in 0-100%

22. A/B testing

23. monitoring
Errors Performance
User
behavior
User
community /
support
automatic notifications

24. The future