The deployment pipeline is a key feature of continuous delivery. Avoiding fragmentation of deployment pipelines is crucial if the maximum benefit is to be derived from pipelines. This presentation shows why pipeline fragmentation is bad, examines some of the reasons why pipeline fragmentation may occur, and explores some of the challenges involved in implementing end-to-end deployment pipelines.

1. End-to-end Deployment Pipelines:
Avoiding Pipeline Fragmentation
Matthew Skelton :: 13 June 2012

2. End-to-end Deployment Pipelines
 Introduction
AGENDA
 Context: Continuous Delivery
 Value stream mapping
 What are deployment pipelines?
 Pipeline fragmentation
 Deriving value from end-to-end pipelines
 Q&A
End-to-end Deployment Pipelines :: Matthew Skelton
2

3. End-to-end Deployment Pipelines
 Introduction
AGENDA
 Context: Continuous Delivery
 Value stream mapping
 What are deployment pipelines?
 Pipeline fragmentation
 Deriving value from end-to-end pipelines
 Q&A
End-to-end Deployment Pipelines :: Matthew Skelton
3

4. Continuous Delivery
 Empowering product owners & the business to
choose when to deploy
 Software always ready to go live
 „One-click‟ deployments
 Visibility of work-in-progress
End-to-end Deployment Pipelines :: Matthew Skelton
4

5. Continuous Delivery
 Context
 http://continuousdelivery.com/
 Jez Humble & David Farley
 (ex-)ThoughtWorks
 Chapter 5, Anatomy of the
Deployment Pipeline, is free to
download:
 http://www.informit.com/articles/article.as
px?p=1621865
 Read the book!
 + screen casts on InfoQ
End-to-end Deployment Pipelines :: Matthew Skelton
5

6. End-to-end Deployment Pipelines
 Introduction
AGENDA
 Context: Continuous Delivery
 Value stream mapping
 What are deployment pipelines?
 Pipeline fragmentation
 Deriving value from end-to-end pipelines
 Q&A
End-to-end Deployment Pipelines :: Matthew Skelton
6

7. Value Stream Maps
 Identify stages in the release cycle
 „Value-added‟ time
 Elapsed time
 Remove bottlenecks, eliminate waste
 Further reading:
 Lean Software Development, Mary & Tom Poppendieck
 “How long would it take your organization to deploy a change
that involves just one single line of code?”
End-to-end Deployment Pipelines :: Matthew Skelton
7

8. Value Stream Maps
 Example (from Continuous Delivery)
 Where is this visible within the organisation?
 Is the information up-to-date?
End-to-end Deployment Pipelines :: Matthew Skelton
8

9. End-to-end Deployment Pipelines
 Introduction
AGENDA
 Context: Continuous Delivery
 Value stream mapping
 What are deployment pipelines?
 Pipeline fragmentation
 Deriving value from end-to-end pipelines
 Q&A
End-to-end Deployment Pipelines :: Matthew Skelton
9

10. What are deployment pipelines?
 Automated manifestations of value stream maps
End-to-end Deployment Pipelines :: Matthew Skelton
10

11. The Deployment Pipeline tells us...
 At what stage is version X of the software?
 Which version is in environment Y?
 Which versions made it to Production?
 What version should I test next?
 ...and so on.
End-to-end Deployment Pipelines :: Matthew Skelton
11

12. The Deployment Pipeline tells us...
 At what stage is version X of the software?
 Which version is in environment Y?
 Which versions made it to Production?
 What version should I test next?
End-to-end Deployment Pipelines :: Matthew Skelton
12

13. The Deployment Pipeline
 GO from ThoughtWorks Studios:
http://www.thoughtworks-studios.com/go-agile-release-management/features-benefits
End-to-end Deployment Pipelines :: Matthew Skelton
13

14. The Deployment Pipeline
 At a glance:
http://www.thoughtworks-studios.com/go-agile-release-management/features-benefits
End-to-end Deployment Pipelines :: Matthew Skelton
14

15. End-to-end Deployment Pipelines
 Introduction
AGENDA
 Context: Continuous Delivery
 Value stream mapping
 What are deployment pipelines?
 Pipeline fragmentation
 Deriving value from end-to-end pipelines
 Q&A
End-to-end Deployment Pipelines :: Matthew Skelton
15

16. Pipeline Fragmentation
 Deployment pipeline is broken into small chunks
 End-to-end view is missing
?
http://blog.mikeobrien.net/2010/09/setting-up-thoughtworks-go-build-on.html
End-to-end Deployment Pipelines :: Matthew Skelton
16

17. Pipeline Fragmentation
End-to-end Pipeline Fragmented Pipeline
From Dev to Production Multiple separate isolated steps
End-to-end Deployment Pipelines :: Matthew Skelton
17

18. Pipeline Fragmentation
 Questions (again):
 At what stage is version X of the software?
 Which version is in environment Y?


Which versions made it to Production?
What version should I test next?
?
End-to-end Deployment Pipelines :: Matthew Skelton
18

19. Problems with Pipeline Fragmentation
 Difficult to get the overall picture
 Multiple places to look
 Limited traceability
 Difficult to answer the common questions:
 At what stage is version X of the software?
 Which version is in environment Y?
 Which versions made it to Production?
 What version should I test next?
 What should be automated becomes manual
End-to-end Deployment Pipelines :: Matthew Skelton
19

20. Causes of Pipeline Fragmentation
 Teams and Roles
 Each team responsible for a single GO server
 GO servers organised around responsibilities
 Location
 Teams in different locations
 Tricky to achieve inter-site view
 Performance
 Slow-downs when > 200 pipelines
 „Misuse‟ of pipelines
End-to-end Deployment Pipelines :: Matthew Skelton
20

21. End-to-end Deployment Pipelines
 Introduction
AGENDA
 Context: Continuous Delivery
 Value stream mapping
 What are deployment pipelines?
 Pipeline fragmentation
 Deriving value from end-to-end pipelines
 Q&A
End-to-end Deployment Pipelines :: Matthew Skelton
21

22. Deriving value from end-to-end pipelines
 Challenges:
 Versioning and dependency management
 Separating subsystems can be tricky
 A product- or service-centric view makes this easier
 Goal is one pipeline per product or subsystem
 Design of deployment pipelines
 Triggers, build dependencies, re-using components as test harnesses
 Multiple test environments complicate pipelines
 One environment per product better than one per team?
 New „diamond dependencies‟ in GO 12.2 might help here
 Optimising the automated test suites is essential
End-to-end Deployment Pipelines :: Matthew Skelton
22

23. Deriving value from end-to-end pipelines
 Choose small, simple products to begin with
 Limited or no dependency on other components
 Involve the business sponsor
 Aim for a “walking skeleton”, end-to-end
 Forces some tricky problems to be solved
 For example:
 Change the text on a button
 Commit, test, deploy to Production within a day
 More complex scenarios can follow later
End-to-end Deployment Pipelines :: Matthew Skelton
23

24. Deriving value from end-to-end pipelines
 Answer questions from:
 Product owners
 The Business
 QA team
 Developers
 Visible progress
 Take more informed decisions, more rapidly
 Reduce cycle time
End-to-end Deployment Pipelines :: Matthew Skelton
24

25. End-to-end Deployment Pipelines
 Introduction
AGENDA
 Context: Continuous Delivery
 Value stream mapping
 What are deployment pipelines?
 Pipeline fragmentation
 Deriving value from end-to-end pipelines
 Q&A
End-to-end Deployment Pipelines :: Matthew Skelton
25

26. Any questions?
http://matthewskelton.net/