Session held at DDD 13 about deployment patterns

1. AM I DOING DEPLOYMENTS
RIGHT?
MATTEO EMILI
MATTEO.EMILI@LIVE.COM || @MATTVSTS ||
HTTP://MATTVSTS.BLOGSPOT.COM

2. WHO AM I?
SYSTEMS ENGINEERING ADVISOR AT ONE IDENTITY
MICROSOFT MVP – VISUAL STUDIO AND DEVELOPMENT TECHNOLOGIES
PROFESSIONAL SCRUM MASTER 1

3. IS DEPLOYMENT TAKEN FOR
GRANTED?

4. CONTINUOUS DEPLOYMENT AND
DELIVERY OPENED THE
FLOODGATES FOR FASTER AND
BETTER RELEASES
Build Server
Automation
Deployment
Orchestrator

5. TWO SIDES OF THE SAME COIN

6. BROWNFIELD = HARD
RESISTANCE TO CHANGES TO THE DEPLOYMENT PIPELINE
YEARS OF LEGACY (ARCHITECTURE, TECHNOLOGY, ETC.)
DIFFICULT TO DEAL WITH

7. GREENFIELD = EASY
PIPELINE DESIGNED TO BE ALMOST A COPYCAT OF SOMEONE ELSE’S
NO LEGACY
RELATIVELY EASY TO CHANGE ONCE IT IS OUT THERE

8. “WE ARE NOT <OTT OF CHOICE>,
WE CANNOT DO THAT…”
“IT IS TOO EXPENSIVE TO REVAMP
IT ALL!”
“EVERYBODY IS USED TO IT, IT IS
NOT AN ACTUAL CONCERN.”

9. THE REALITY IS THAT THE
DEPLOYMENT PIPELINE IS A CRITICAL
ASSET AND IT BEARS INCREDIBLE
VALUE TO THE COMPANY

10. SOURCE: Ulrichulrich, Wikipedia.org

11. CIVIL ENGINEERING V SOFTWARE
ENGINEERING
PIPES
VALVES
INTERMEDIATE
STATIONS
FUEL
ORCHESTRATOR
APPROVALS
ENVIRONMENTS
OUR SYSTEM!

12. “OK I’M SOLD! HOW DO I START
TREATING MY SYSTEM AS FUEL?”

13. THE REAL CORNERSTONE OF
DEPLOYMENT IMPROVEMENT

14. SOURCE: Alan Chia, Wikipedia.org

15. A PACKAGE SHOULD CONTAIN ALL
THE REQUIRED BINARIES TO MAKE
(A PORTION OF) A SYSTEM WORK

16. BUT… HOW? MY APPLICATION IS
LEGACY/RUNS ON PREMISE/IFS-
BUTS…

17. THERE ARE MANY PACKAGING
SOLUTIONS OUT THERE
(MSDEPLOY?)

19. YOU ALSO NEED AUTOMATION
(OR THE CONCEPT OF IT)

20. IT IS EXTREMELY IMPORTANT FOR
TWO REASONS

21. AUTOMATIONS ARE AT THE
FOUNDATION OF A PIPELINE
INFRASTRUCTURE

22. AUTOMATION ENSURES NO
CHANGES DURING THE PIPELINE
EXECUTION

23. ONCE SOMETHING IS BUILT OUT
OF VERSION CONTROL (WHICH
INCLUDES A PR BY SOMEONE ELSE)
IT’S IMMUTABLE

24. PIPELINES RELY ON PIPES AND
VALVES (APPROVALS) TO ENSURE
THE DESIRED FLOW

25. HOW YOU USE SUCH VALVES
OFTEN AFFECTS THE DEGREE OF
DEPLOYMENT MATURITY

26. APPROVALS CAN BE AUTOMATED
GIVEN CERTAIN CONDITIONS OR
TARGETS

27. THOSE ARE ALL THE
PREREQUISITES

28. LET’S TAKE A LOOK AT SOME
PATTERNS

29. DISCLAIMER!
THE SESSIONS WILL FEATURE WEB APPLICATIONS, BUT EVERYTHING
SHOWN IS GOING TO BE APPLICABLE TO ALL SORTS OF APPLICATIONS
(INCLUDING MOBILE)

30. ROLLING DEPLOYMENTS
THE FIRST STEP TOWARDS DEPLOYMENT MATURITY

31. ROLLING DEPLOYMENT IS THE
FIRST EXAMPLE OF ‘ADVANCED’
DEPLOYMENT

32. ITS MAIN PURPOSE IS TO MAINTAIN
THE SERVICE AVAILABLE DURING
DEPLOYMENTS

33. TOPOLOGY
NODE 1
v1
NODE 2
v1
LOAD BALANCER

34. TOPOLOGY
NODE 1
v1
NODE 2
v1
LOAD BALANCER

35. TOPOLOGY
NODE 1
v1
NODE 2
v2
LOAD BALANCER

36. TOPOLOGY
NODE 1
v2
NODE 2
v2
LOAD BALANCER

41. IT IS PRETTY MUCH WHAT WE DO
WITH AZURE APP SERVICE
DEPLOYMENT SLOTS…

43. DOWNTIME IS LIMITED TO THE
SWITCHOVER

44. WHAT ARE THE DRAWBACKS HERE?

45. YOU ONLY HAVE *ONE*
PRODUCTION ENVIRONMENT TO
DEAL WITH

46. TEMPORARILY REDUCED
AVAILABILITY AND HIGHER THAN
AVERAGE LOAD ON A NODE

47. FULL ROLLBACK IMPLIES FULL
REBUILD

48. IT WORKS VERY WELL WITH
CONTROLLED SCENARIOS
(WHERE YOU CAN HANDLE THE
LOAD)

49. BLUE-GREEN DEPLOYMENTS
THE LAND OF (ALMOST) NO DOWNTIME!

50. BLUE-GREEN DEPLOYMENTS
FEATURE A CONTROLLED
DEPLOYMENT ON A CLONE OF THE
LIVE ENVIRONMENT

51. TOPOLOGY
NODE 1
v1
NODE 2
v1
BLUE
NODE 1
v1
NODE 2
v1
GREEN
LOAD BALANCER

52. TOPOLOGY
NODE 1
v1
NODE 2
v1
BLUE
NODE 1
v1
NODE 2
v1
GREEN
LOAD BALANCER

53. TOPOLOGY
NODE 1
v2
NODE 2
v2
BLUE (TESTED AND
READY)
NODE 1
v1
NODE 2
v1
GREEN
LOAD BALANCER

54. TOPOLOGY
NODE 1
v2
NODE 2
v2
BLUE
NODE 1
v1
NODE 2
v1
GREEN
LOAD BALANCER

55. TOPOLOGY
NODE 1
v2
NODE 2
v2
BLUE
NODE 1
v2
NODE 2
v2
GREEN
LOAD BALANCER

64. SAME NO-DOWNTIME
APPROACH…

65. … BUT DOUBLE THE
INFRASTRUCTURE

66. SLIGHTLY EASIER TO ROLLBACK

67. CANARY DEPLOYMENTS
SILENT LAUNCHES, PROGRESSIVE EXPOSURE, TESTING IN
PRODUCTION

68. CANARY DEPLOYMENTS ARE LIKE
BLUE-GREEN DEPLOYMENTS, BUT
AN ADDITIONAL ENVIRONMENT

69. THIS ENVIRONMENT (‘PRE-PROD?’)
SHOULD BE USED BY A PRE-
DEFINED GROUP OF USERS (‘BETA
TESTERS’?)

70. TOPOLOGY
NODE 1
v1
NODE 2
v1
REGULAR USERS
NODE 1
v2
NODE 2
v2
EARLY ADOPTERS
LOAD BALANCER

71. THESE USERS SHOULD GET NEW
FEATURES BEFORE THE GENERAL
AVAILABILITY OF YOUR PRODUCT

72. WINDOWS INSIDERS PROGRAM? 
(THEY USE RINGS, BUT IT IS
CONCEPTUALLY THE SAME)

73. HOW DO I GET THE MOST VALUE
OUT OF THIS APPROACH?

74. FEATURE FLAGS!

75. SELECTIVELY ENABLE/DISABLE
FEATURES

76. TOPOLOGY
NODE 1
v1
NODE 2
v1
REGULAR USERS
NODE 1
v2
NODE 2
v2
EARLY ADOPTERS
LOAD BALANCER

77. PROVIDE A KILLSWITCH IN CASE OF
ISSUES

78. TOPOLOGY
NODE 1
v1
NODE 2
v1
REGULAR USERS
NODE 1
v2
NODE 2
v2
EARLY ADOPTERS
LOAD BALANCER

79. PERFORM A/B TESTING

80. TOPOLOGY
NODE 1
v1
NODE 2
v1
REGULAR USERS
NODE 1
v2.1
NODE 2
v2.2
EARLY ADOPTERS
LOAD BALANCER

81. SHADOW DEPLOYMENTS
THE MOST EXTREME OF ALL!

82. PERFORM SILENT RELEASES OF
NOT-FULLY-WORKING
COMPONENTS

83. TOPOLOGY
NODE 1
v1
NODE 2
v1
REGULAR USERS
NODE 1
v1
New DB
schema
Beta API
NODE 1
v1
New DB
schema
Beta API
REGULAR USERS
LOAD BALANCER

84. WHAT SHOULD I DO WITH THESE
BITS?

85. RUN REAL WORLD TESTS

86. CREATE A MOCK SERVICE THAT
CONSUMES THE NEW BITS IN
PRODUCTION

87. HOW IS THAT ANY DIFFERENT THAN A
TESTING ENVIRONMENT?

88. YOU ARE USING A SYSTEM UNDER
REAL LOAD SO YOU CAN IMMEDIATELY
SEE IF THERE ARE LOAD OR STABILITY
ISSUES

89. ON THE DOWNSIDE, IT IS THE MOST
EXPENSIVE AND THE MOST COMPLEX TO
SETUP

90. THE SYSTEM’S ARCHITECTURE IS
DRIVEN BY THE PIPELINE

91. HANDLING BREAKING CHANGES
BECOMES EXTREMELY DIFFICULT

92. AND NOW, FOR SOMETHING
COMPLETELY DIFFERENT…
ADDITIONAL NON-TECH SUGGESTIONS TO IMPROVE YOUR PROCESS

93. AS OF NOW WE ONLY LOOKED AT
TECHNOLOGY…

94. …IT’S ACTUALLY *PEOPLE* THAT
MAKE SYSTEMS WORK!

95. A FEW THINGS TO KEEP IN MIND TO
MAKE YOUR LIFE EASIER

96. HAVE CONSISTENT ACLS
ACROSS YOUR STACKS
WE CAN’T ADVOCATE AUTOMATION AND THEN WASTE HOURS
ASSIGNING RESOURCES TO PEOPLE…

97. USING GROUPS MAKES ACCESS
MANAGEMENT BEARABLE WITHOUT
DEALING WITH A SINGLE USER

98. USE THEM AS A SINGLE SOURCE OF
TRUTH

99. KEEP PRODUCTION AS A
SEPARATE STAGE
IF YOU HAVE A LINEAR PIPELINE PRODUCTION SEPARATION COMES IN
HANDY FOR ALL SORTS OF REASONS

100. PRODUCTION SHOULD BE THE PLACE
WHERE MANUAL ACCESS IS A LAST
RESORT

101. KEEP IT SEPARATE FROM THE REST
SO IT ACTS AS A FURTHER CUSHION
STAGE

102. USE AN ALTERNATE STAGE AS
PLACE TO KICK THE TYRES OF
YOUR SYSTEM BEFORE THE
ACTUAL SWITCHOVER
‘ALTERNATE PROD’ IS A THING – YOU CAN USE IT FOR QUICK
ROLLBACKS, TESTING IN PRODUCTION, ETC.

103. ALTERNATE AND PRODUCTION
SHOULD BE CONCEPTUALLY THE
SAME

104. WITH PRODUCTION AS A SEPARATE
STAGE, PRE-PROD IS WHERE THE
MAGIC HAPPENS

105. PARAMETRISE SETTINGS AND
POPULATE AT DEPLOYMENT
SETTINGS SHOULD NOT BE IN THE REPOSITORY!

106. YOU WANT TO BE ABLE TO EDIT
YOUR SETTINGS VIA THE PIPELINE

107. CONSISTENCY IS GRANTED, AND
THERE IS NO MANUAL ACCESS TO
ENVIRONMENTS

108. AS YOU CAN SEE, IT IS NOT JUST FOR
UNICORNS, OVER-THE-TOP
COMPANIES OR GREENFIELD
PROJECTS

109. REGARDLESS OF WHAT YOUR
APPLICATION DOES, IT IS HOW YOU
SHIP IT THAT MATTERS!

110. PACKAGE, AUTOMATE AND GO
TOWARDS THE DESIRED LEVEL OF
DEPLOYMENT MATURITY 

111. QUESTIONS?

112. THANKS!