These are the slides of the tutorial I gave at QA&Testing in Bilbao on 17 October 2018 Continuous integration and deployment (CI/CD) empowers organizations to bring their solution in production fast and frequent. This interactive session will share the benefits of this concept and introduce eight conditions that need to be met in order to make CI/CD a success. After this brief introduction, we will make small groups and explore these conditions, exchange experiences and you will get an understanding what needs to be improved in your organization. Talk to your peers and learn where they stand. Of course each of the groups will share their learnings, so we all go home with an understanding of how you can benefit from CI/CD and what needs to be done to make it work. Finally we will see what test strategy we would advise if our company would decide to move towards CI/CD and this cover we consider much more than just automate our tests…

1. Tester’s considerations when moving
towards successful CI/CD
1
Derk-Jan de Grood
QA& Testing 2018
Bilbao

2. Aim of this session
2

3. 3
Tester’s considerations when moving towards
successful CI/CD
Continuous integration and deployment (CI/CD)
empowers organizations to bring their solution in
production fast and frequent. This interactive session
will share the benefits of this concept and introduce
eight conditions that need to be met in order to make
CI/CD a success. After this brief introduction, we will
make small groups and explore these conditions,
exchange experiences and you will get an
understanding what needs to be improved in your
organization. Talk to your peers and learn where they
stand. Of course each of the groups will share their
learnings, so we all go home with an understanding of
how you can benefit from CI/CD and what needs to be
done to make it work.

4. Embracing Change
4

5. 5
“The only constant I am sure of
Is this accelerating rate of
change”
Source:[PeterGrabriel–downsideup]

6. 6

7. Organizational Life Cycle
7

8. Organizational Life Cycle
8
Legacy
Inertia
1
2
3
4
5

9. 9
Over the last 50 years the
average life expectation
of companies is reduced
to 1/3 which equals 15
years
(based upon the standard & poor index)

10. 10

11. 11
Dutch defense system

12. Other examples of disruptive
innovations
12

13. 13
The impact of disruptive innovations
is hard to predict!

14. IT Development & Delivery
14

15. Our End Goal
15

16. Our End Goal
16
Fail Forward

17. Building Blocks
17
CI/CD

18. Continuous Integration & Deployment
A brief introduction
18

19. Workshops and Book
19

20. Take a look around you !
20
Releasing the
bugfix on the
test
environment
takes ages
We have been
working hard,
but nothing is
ready for the
Demo
We have
delivered as IT
but the system
is not used yet
We have
figured out how
to configure the
system, but
operations
lacks this
knowledge
We have fixed
the critical but
testing takes
multiple days
Every time we
release we
reserve support
time to sort out
prod. errors

21. Antipattern: Deploying Manually
If your deployments are not fully automated you:
• Need to extensively document the deployment process
• Rely on a manual to confirm the application is running OK
• Get frequent calls to dev team about release problems
• Need to correct the process during deployment
• End up with different configurations on various environments
• Have unpredictable outcome and have to reserve fixing time
to make it work
• Rely on SPOC (the release expert)
21[Continuous Delivery – Jezz Humble & David Farley]

22. Antipattern: Deploy to PRD-like
once dev is complete
When you deploy late in the cycle to a production like
environment you
• Do most of your test on development machines
• Create an hand-over between dev and ops: config files,
database migrations, deployment scripts etc.
• Ask operations do the deployment for the first time on the
release date
• Do not stimulate dev and ops to collaborate
• Create a late-feedback cycle when dealing with production
errors
• Make a lot of assumptions about the PRD environment
22[Continuous Delivery – Jezz Humble & David Farley]

23. Antipattern: Manual Configuration
When you have a manual configuration management, you:
• Risk that deployment fail to configuration differences
• Need a lot of preparation time to release
• Cannot easily step back to an earlier configuration
• Risk that different machines have different versions of
OS, libraries, patch-level
• Probably adapt configuration items live on the
production environment
23[Continuous Delivery – Jezz Humble & David Farley]

24. 24

25. Levels
25
Code &
commit
Integrate
& build
Test Deliver

26. Working application decomposed
26
Executable
Code Configuration
Host
Environment
Data

27. Continuous Delivery Example
27
Local
• Code
• Unit Test
• Commit
Development
• Save Repo
• Get config
• Deploy
• Test
Test
• Save Repo
• Get config
• Deploy
• Test
Acceptance
• Save Repe
• Get config
• Deploy
• Test
1
Daily
• Nightly regression
test
• Monitoring
Daily
• Nightly
performance test
• Monitoring
Daily
• E2E test
• Monitoring

28. Definitions
Continuous integration (CI) is the practice of merging all
developer working copies to a shared mainline several
times a day.
With continuous delivery (CD) teams produce software in
short cycles, ensuring that the software can be reliably
released at any time. It aims at building, testing, and
releasing software faster and more frequently.
Continuous deployment (CD) is the next step of
continuous delivery: Every change that passes the
automated tests is deployed to production automatically.
[Sources: Wikipedia and puppet.com]
28

29. SAFe 4.6
29

30. 30

31. Some advantages
• Efficiency
• Reduces amount of rework
• Predictability
• Reduced time to market
• Fast feedback
• Reducing risk
• Better quality solutions
32

32. How often to production ?
33
[https://devops.com/often-release-continuous-delivery/]
How important are your updates?
Is there a marketing advantage in delaying the release?
Do other apps depend on your app?
Is your development pace consistent?
Do your updates create a learning curve?
Can multiple versions of your app coexist?
ChristopherTozzi

33. How often to production
How important are your updates? If changes to your app mostly involve mundane
things such as interface tweaks, continuous deployment probably will confuse users more
than please them. They will have to adapt constantly to new features. But if you can roll
out big, user-friendly changes on a continual basis, then deploying continuously can help
keep your users happy.
Is there a marketing advantage to be gained by waiting longer to
release? Sometimes, advertising new features but waiting a while to release them can
help build excitement. Case in point: Apple Inc.
Do other apps depend on your app? If you’re working within an ecosystem where your
app integrates with other apps or platforms, continuous deployment makes the lives of
partner developers more difficult because it requires them to adjust constantly to changes
in your app. To help them ensure their work will always be compatible with yours, provide
space between your releases.
Is your development pace consistent? If not, continuous deployment can be
problematic because users will see rapid updates at some times, but stagnation at other
times. Continuous deployment only works well for users if they can count on a relatively
steady stream of updates.
Do your updates create a learning curve? If users will have to learn new tricks to use
an updated app, continuous deployment may create more confusion than happiness.
Can multiple versions of your app coexist?
34
[https://devops.com/often-release-continuous-delivery/]

34. Will it work ?
Requisites for CI/CD
35

35. 36
What are the
requisites to make
it work?

36. CI/CD requires that (in random order)
3
Teams collaborate
with each other
Integration is
continuous
Deployment is a
hands-off process
Feedback loop to
improve quality
Features are
launched frequently
Acceptance criteria
are clear
Teams have all
required skills and
knowledge
Tests are
automated
Valori-version1.2-2017

37. How can you
use it?
Sales
Maturity
scan
Triggering
discussions
Defining
roadmap

38. Integration is continuous
Should be in place
• Teams integrate as fast as
possible
• Teams work with the
same DOD
• Integration tests can be
run by everyone or
started upon check-in
• Team make agreements
on how to deal with
integration tests
Challenges
• Number of teams
• Integration at system level
• Hardware components
• Organizational boundaries
• Dependencies not known
• Infrastructure
• Different heartbeats

40. CI/CD requires that (in random order)
4
Teams collaborate
with each other
Deployment is a
hands-off process
Feedback loop to
improve quality
Features are
launched frequently
Acceptance criteria
are clear
Teams have all
required skills and
knowledge
Tests are
automated
Valori-version1.2-2017
Architecture supports
partial development
and release

41. What do the requisites mean to you?
42

42. Assignment
1. Divide in groups
2. Define what the requisite means to you (5 min)
3. Rate individually how your team/ organization
scores (1 min)
4. List as a group some (20 min)
• things you have in place to fulfill this requisite
• challenges you recognize
• Time is limited
• Mission is information gathering
• Tip: Identify items, don’t get lost in the
discussion

43. Canvas
44
Description
The teams…..bladibla
Rating
😟 😀
In Place Challenges

44. How to pick your team
• You got this covered and are
willing to share some tips
• You are currently struggling with
this and want to discuss it with
your peers
• You find this important and
want to compare your status
• You want to learn about this
topic since its quite new for you

45. Exchange Experiences
46

46. Workshop Outcome
47

47. Workshop results (QA&Test)

48. Workshop results (QA&Test)

49. Workshop results (QA&Test)

50. [SC]2 morning workshop, 4 Oct.
2017
[SC]2 afternoon workshop, 4 Oct.
2017
TestNet workshop, 11 Oct. 2017
Teams collaborate
with each other
Architecture supports
partial development
and release
Tests are
automated
Deployment is a
hands-off process
Acceptance criteria are
clear
Features are launched
frequently
Feedback loop to
improve quality
Teams have all
required skills and
knowledge

51. Getting into CI/CD

52. Context based
Green field

53. Green field
Do it right from day 1
Ensure that everybody
supports the approach
Write automated test
for each story
54
Testers are involved during
refinement to ensure that each
acceptance criterium can be
tested

54. Legacy Environment
1. Set up version control
2. Create an automatic build process
3. Sit with users to identify high value scenario’s
4. Create an Automated Functional Test as smoke test
5. Expand tests while developing
• E.g. Test hardening to add alternative and exception paths
• E.g. By defining tests for new stories
55
Challenge 1: Business does not
understand why you have to
spend time with defining tests
for a system that is already
accepted and in production
Challenge 2: System is not
suited for Automation e.g.
structure or object recognition

55. Testing Considerations

56. When to test?
When anything changes in:
57
Executable
Code
Configuration
Host
Environment
Data

57. What to test….
58
Process of creating executable code
- Syntax of source code is correct
User Acceptance Test
- System supports customer journey
Exploratory tests
- Validation check Integration test
- System works within system of other
systems
Functional and NF tests
- System delivers value as intended
- System complies to capacity,
security, availability, etc.
requirements
Smoke test and deployment test
- Systems is correctly and installed and
configured on target environment
Configuration test
- 1: ensure references to external
service in configuration setting are
OK
- 2: Run smoke test
Code Quality Criteria
- e.g. test coverage and code metrics
Unit testing
- Code behaves as expected

58. 59

59. Assignment: CI/CD Test Strategy
Your organization is moving towards CI/CD. They ask you to
draw up a plan to (re)organize testing to ensure that
Quality is built-in
Time to market is reduced
We can do Validated learning
Explain
What tests (or quality measures) you think are necessary?
Which of them should you start with
How would you organize them
What benefits would this yield.
60

60. Wrap-up
61

61. Aim of this session
62

62. Derk-Jan
Valori
Coltbaan 4a
3439 NG NIEUWEGEIN
The Netherlands
• derkjandegrood@valori.nl
• +31(0)651807878
• www.valori.nl
• @DerkJanDeGrood
• http://djdegrood.wordpress.com
63