DevOps is about culture and practices, not tools, right? Well, what if the tools we chose are keeping us away from good practices? Adopting a “DevOps one-stop solution” often means you are trading knowledge of your toolchain (and the adequate tool boundaries for your delivery) for ease of setup. Yes, you’ll go faster at the start. But you will inevitably slow down over time, as technology changes and new practices emerge. A flexible toolchain standardizes on practices, not tools, so it’s important to understand the trade-offs we’re making when selecting tools, especially if we’re driven only by familiarity and ease of setup, an anti-pattern we’ve seen across several of our clients.

1. La La Land of DevOps Integration
Manuel Pais
@manupaisable | manuelpais.net
Continuous Lifecycle London 2018

2. Continuous Delivery
(Tooling) (Anti-)Patterns
Manuel Pais
@manupaisable | manuelpais.net
Continuous Lifecycle London 2018

3. Today
1. DevOps integration: two camps
2. Pattern #1: Best tool for the job
3. Pattern #2: Pipeline = value stream
4. Pattern #3: Recover quickly
@manupaisable | manuelpais.net 3

4. About me
Manuel Pais
MS Software Eng
@manupaisable
manuelpais.net
me@manuelpais.net
DevOps and Delivery Consultant
Focused on teams and flow
@manupaisable | manuelpais.net 4

5. DevOpsTopologies.com
@manupaisable | manuelpais.net 5

6. Today
1. DevOps integration: two camps
2. Pattern #1: Best tool for the job
3. Pattern #2: Pipeline = value stream
4. Pattern #3: Recover quickly
@manupaisable | manuelpais.net 6

7. “integrating various
tools in software
delivery is very hard”
“a toolchain is a
design pattern
that everybody
can agree on”
https://www.infoq.com/articles/tool-integration-hard-truth
https://www.infoq.com/interviews/interview-damon-edwards-qcon-2014

8. Integrated
DevOps Tool
In the left corner…
@manupaisable | manuelpais.net 8
Integrated
DevOps
Toolchain
In the right corner…

9. Integrated
DevOps
Toolchain
Integrated
DevOps Tool
@manupaisable | manuelpais.net 9

10. @manupaisable | manuelpais.net 10

11. Today
1. DevOps integration: two camps
2. Pattern #1: Best tool for the job
3. Pattern #2: Pipeline = value stream
4. Pattern #3: Recover quickly
@manupaisable | manuelpais.net 11

12. practices first,
tools second
@manupaisable | manuelpais.net 12

13. but some practices
can be quite hard
with integrated tools…
@manupaisable | manuelpais.net 13

14. @manupaisable | manuelpais.net 14

15. “ability to get changes of all types,
into production, or into the hands of
users, safely and quickly in a
sustainable way”
–Dave Farley & Jez Humble
continuousdelivery.com
@manupaisable | manuelpais.net 15

16. CD in the real world
•Focus on individual tools
(Jenkins, VSTS, GoCD, etc)
•Focus on familiarity, cost, ease of
use, security (sometimes…)
•Focus on “scaling”
@manupaisable | manuelpais.net 16

17. CD in the real world
•Focus on individual tools
(Jenkins, VSTS, GoCD, etc)
•Focus on familiarity, cost, ease of
use, security (sometimes…)
•Focus on “scaling”
@manupaisable | manuelpais.net 17

18. CD in the real world
•Focus on individual tools
(Jenkins, VSTS, GoCD, etc)
•Focus on familiarity, cost, ease of
use, security (sometimes…)
•Focus on “scaling”
@manupaisable | manuelpais.net 18

19. CD in the real world
Not enough focus on
expected practices & results
from pipeline / delivery
@manupaisable | manuelpais.net 19

20. Resilient delivery system
that enables
fast(er) feedback loops
20@manupaisable | manuelpais.net

21. Over time issues arise…
•Scalability
•Integration
•Evolvability
•Performance
•Resilience (aka coffee time)
•Bad coding/scripting practices
@manupaisable | manuelpais.net 21

22. Delivery becomes bottleneck
@manupaisable | manuelpais.net 22
ProductionDelivery

23. Today
1. DevOps integration: two camps
2. Pattern #1: Best tool for the job
3. Pattern #2: Pipeline = value stream
4. Pattern #3: Recover quickly
@manupaisable | manuelpais.net 23

24. Integrated DevOps Toolchain
@manupaisable | manuelpais.net 24

25. API-driven tools
are table stakes
for pipeline integration
@manupaisable | manuelpais.net 25

26. @manupaisable | manuelpais.net 26

27. allows for
repeatable mechanisms
and minimizing rework
@manupaisable | manuelpais.net 27

28. @manupaisable | manuelpais.net 28

29. @manupaisable | manuelpais.net 29

30. Integrated DevOps Toolchain
@manupaisable | manuelpais.net 30

31. Integrated DevOps Toolchain
@manupaisable | manuelpais.net 31

32. Integrated DevOps Toolchain
@manupaisable | manuelpais.net 32

33. a flexible toolchain
standardizes practices,
not tools
@manupaisable | manuelpais.net 33

34. tooling should evolve
when YOU need it
@manupaisable | manuelpais.net 34

35. Limit blast radius
@manupaisable | manuelpais.net 35

36. tightly coupled tooling
@manupaisable | manuelpais.net 36

37. tightly coupled tooling
@manupaisable | manuelpais.net 37

38. tightly coupled tooling
@manupaisable | manuelpais.net 38

39. tightly coupled tooling
@manupaisable | manuelpais.net 39

40. @manupaisable | manuelpais.net 40

41. @manupaisable | manuelpais.net 41

42. limit blast radius
@manupaisable | manuelpais.net 42

43. @manupaisable | manuelpais.net 43

44. Today
1. DevOps integration: two camps
2. Pattern #1: Best tool for the job
3. Pattern #2: Pipeline = value stream
4. Pattern #3: Recover quickly
@manupaisable | manuelpais.net 44

45. what’s missing?
@manupaisable | manuelpais.net 45

46. what’s missing?
we tend to care about
technical bits only…
@manupaisable | manuelpais.net 46

47. Integration
Tests
Functional
Tests
Performance
Tests
Operational
Acceptance
PROD
@manupaisable | manuelpais.net 47

48. @manupaisable | manuelpais.net 48

49. @manupaisable | manuelpais.net 49

50. painful bottlenecks
in team interactions
@manupaisable | manuelpais.net 50

51. bottlenecks = silos
time is lost in queues,
not execution
@manupaisable | manuelpais.net 51

52. automating hand-offs
requires trust… which in
turn requires time
@manupaisable | manuelpais.net 52

53. breaking down silos is
HARD!
@manupaisable | manuelpais.net 53

54. we need to design pipelines
with our users in mind,
just like with our apps
@manupaisable | manuelpais.net 54

55. Amy, the Product Owner
@manupaisable | manuelpais.net 55

56. @manupaisable | manuelpais.net 56

57. Amy, the Product Owner
@manupaisable | manuelpais.net 57

58. non-technical stakeholders
are part of delivery too
@manupaisable | manuelpais.net 58

59. pipeline = (real) workflow
highlight bottlenecks => build bridges
trust requires time => automate gradually
not just technical => include everyone
@manupaisable | manuelpais.net 59

60. pipeline = (real) workflow
highlight bottlenecks => build bridges
trust requires time => automate gradually
not just technical => include everyone
@manupaisable | manuelpais.net 60

61. pipeline = (real) workflow
highlight bottlenecks => build bridges
trust requires time => automate gradually
not just technical => include everyone
@manupaisable | manuelpais.net 61

62. Today
1. DevOps integration: two camps
2. Pattern #1: Best tool for the job
3. Pattern #2: Pipeline = value stream
4. Pattern #3: Recover quickly
@manupaisable | manuelpais.net 62

63. Resilient delivery system
that enables
fast(er) feedback loops
63@manupaisable | manuelpais.net

64. time between failures
time to repair
old view of the world
@manupaisable | manuelpais.net 64

65. time between failures
time to repair
new view of the world
@manupaisable | manuelpais.net 65

66. resilient systems also fail
but…
have short time to repair
@manupaisable | manuelpais.net 66

67. time to repair
=
time to diagnose + time to fix
@manupaisable | manuelpais.net 67

68. time to repair
=
time to diagnose (80) + time to fix (20)
@manupaisable | manuelpais.net 68

69. reducing time to repair
requires ownership
@manupaisable | manuelpais.net 69

70. owning delivery system
>
owning errors and logs
@manupaisable | manuelpais.net 70

71. 111.222.333.123 HOME -
[01/Feb/1998:01:08:39 -0800] "GET
/bannerad/ad.htm HTTP/1.0" 200 198
"http://www.referrer.com/bannerad/ba_in
tro.htm" "Mozilla/4.01 (Macintosh; I;
PPC)"
@manupaisable | manuelpais.net 71

72. the problems with logs
@manupaisable | manuelpais.net 72

73. the problems with logs
too much information
information hiding
@manupaisable | manuelpais.net 73

74. the problems with logs
too much information
information hiding
@manupaisable | manuelpais.net 74

75. @manupaisable | manuelpais.net 75

76. log aggregation won’t
help if there’s
no correlation
@manupaisable | manuelpais.net 76

77. https://www.infoq.com/articles/why-test-logging
@manupaisable | manuelpais.net 77

78. information hiding
should not be the reason
why you can’t diagnose
@manupaisable | manuelpais.net 78

79. @manupaisable | manuelpais.net 79

80. without correct error
information you can’t
diagnose/improve
@manupaisable | manuelpais.net 80

81. log aggregation
own your logs
quickly find correlated events
reduce time to diagnose/repair
@manupaisable | manuelpais.net 81

82. Today
1. DevOps integration: two camps
2. Pattern #1: Best tool for the job
3. Pattern #2: Pipeline = value stream
4. Pattern #3: Recover quickly
@manupaisable | manuelpais.net 82

83. More practices for resiliency
•Pipeline infrastructure-as-code
•Pipeline immutable infrastructure
•Blue-green pipeline deployments
•Pipeline configuration-as-code
•Monitoring & alerting
•Auto-scaling
@manupaisable | manuelpais.net 83

84. More practices for fast feedback
•Dynamic test environments
•Dynamic tests scheduling (fail quickly)
•Measure activity times/deviations
•Focus on flow efficiency (remove
largest bottleneck first)
•Parallelize whenever possible
@manupaisable | manuelpais.net 84

85. Take away
delivery system needs to
support evolution of
practices and tools
@manupaisable | manuelpais.net 85

86. Take away
visualize real workflow
to break down silos
@manupaisable | manuelpais.net 86

87. Take away
reducing time to repair
requires ownership
@manupaisable | manuelpais.net 87

88. Sources
• Manuel Pais and Matthew Skelton, 2017, ‘Continuous
Delivery Anti-Patterns’, https://dzone.com/articles/continuous-delivery-
anti-patterns
• Betty Zakheim, 2017, ‘How Difficult Can It Be to Inte-
grate Software Development Tools? The Hard Truth’,
https://www.infoq.com/articles/tool-integration-hard-truth
• Damon Edwards, 2014 ‘DevOps is an Enterprise
Concern’,
https://www.infoq.com/interviews/interview-damon-edwards-qcon-2014
• Don G. Reinertsen, 2009, ‘The Principles of Product Development
Flow: Second Generation Lean Product Development’
https://www.youtube.com/watch?v=L6v6W7jkwok
@manupaisable | manuelpais.net 88

89. @ReleasabilityY
releasabilitybook.com
Team Guide to
Software Releasability
by Chris O’Dell & Manuel Pais
@manupaisable | manuelpais.net 89

90. Resilient delivery system
that enables
fast(er) feedback loops
90@manupaisable | manuelpais.net

91. thank you
Manuel Pais
@manupaisable
me@manuelpais.net
@manupaisable | manuelpais.net 91