This is the true fairytale of a company who decided to end its death spiral of monthly release cycles, three hour deployment meetings, expensive hardware, and falling market share.
We describe how we went to near-real-time zero-downtime deployment with lower costs and happier customers. We explain our choices, tools, changes and expose our ugly “before” and “after” pictures. Most importantly, we show you our process of change, born into a new, exciting, competitive culture.
The frog of legacy became the princess of modern agile development+operations.
Contact us to help your transformation: http://blog.atomicinc.com/contact/
11. Who Am I?
Life in Tech Business
Mission-Critical IT Consulting
10 yrs 12+ yrs
S t a r t u p s
Avid (if not very good) ice hockey player
Long-time lover of great engineering… when used to make a real difference
Avi Deitcher avi@atomicinc.com
12. Who Am I?
Life in Tech Business
Mission-Critical IT Consulting
10 yrs 12+ yrs
S t a r t u p s
Avid (if not very good) ice hockey player
Atomic Inc: Generalist Practitioner | Product : engineering : operations | Incentives and culture
Long-time lover of great engineering… when used to make a real difference
Avi Deitcher avi@atomicinc.com
15. Avi Deitcher avi@atomicinc.com
What Tales do We Weave?
The definition of a fairy-story—what it is, or what it should be—does not, then, depend on any
definition or historical account of elf or fairy, but upon the nature of Faërie: the Perilous Realm
itself, and the air that blows in that country.
-“On Fairy-Stories”, J.R.R. Tolkien
16. Avi Deitcher avi@atomicinc.com
What Tales do We Weave?
The definition of a fairy-story—what it is, or what it should be—does not, then, depend on any
definition or historical account of elf or fairy, but upon the nature of Faërie: the Perilous Realm
itself, and the air that blows in that country.
-“On Fairy-Stories”, J.R.R. Tolkien
A fairy tale is a story for children involving magical events and imaginary creatures; a highly
improbable account.
-Collins English Dictionary
17. Avi Deitcher avi@atomicinc.com
What Tales do We Weave?
The definition of a fairy-story—what it is, or what it should be—does not, then, depend on any
definition or historical account of elf or fairy, but upon the nature of Faërie: the Perilous Realm
itself, and the air that blows in that country.
-“On Fairy-Stories”, J.R.R. Tolkien
A fairy tale is a story for children involving magical events and imaginary creatures; a highly
improbable account.
-Collins English Dictionary
18. Avi Deitcher avi@atomicinc.com
What Tales do We Weave?
The definition of a fairy-story—what it is, or what it should be—does not, then, depend on any
definition or historical account of elf or fairy, but upon the nature of Faërie: the Perilous Realm
itself, and the air that blows in that country.
-“On Fairy-Stories”, J.R.R. Tolkien
A fairy tale is a story for children involving magical events and imaginary creatures; a highly
improbable account.
-Collins English Dictionary
A quest in which hero/heroine overcomes external and internal challenges to achieve a goal of
individual and societal growth, a happy ending.
-Common definition
19. Avi Deitcher avi@atomicinc.com
What Tales do We Weave?
The definition of a fairy-story—what it is, or what it should be—does not, then, depend on any
definition or historical account of elf or fairy, but upon the nature of Faërie: the Perilous Realm
itself, and the air that blows in that country.
-“On Fairy-Stories”, J.R.R. Tolkien
A fairy tale is a story for children involving magical events and imaginary creatures; a highly
improbable account.
-Collins English Dictionary
A quest in which hero/heroine overcomes external and internal challenges to achieve a goal of
individual and societal growth, a happy ending.
-Common definition
47. Avi Deitcher avi@atomicinc.com
Balrog Processes
• 4+ hour deployment meetings!
• Deployment artifacts: war files, multi-page instructions
• “edit application.properties line 25 to enter the IP of the environment-specific DB server”
• “add the username and password on lines 26 and 28 of application.properties”
• “restart app servers in the following order”
48. Avi Deitcher avi@atomicinc.com
Balrog Processes
• 4+ hour deployment meetings!
• Deployment artifacts: war files, multi-page instructions
• “edit application.properties line 25 to enter the IP of the environment-specific DB server”
• “add the username and password on lines 26 and 28 of application.properties”
• “restart app servers in the following order”
• 3 sides * 3 sites = 9 deployments for prod alone!
• QA, UAT, etc.?
49. Avi Deitcher avi@atomicinc.com
Balrog Processes
• 4+ hour deployment meetings!
• Deployment artifacts: war files, multi-page instructions
• “edit application.properties line 25 to enter the IP of the environment-specific DB server”
• “add the username and password on lines 26 and 28 of application.properties”
• “restart app servers in the following order”
• 3 sides * 3 sites = 9 deployments for prod alone!
• QA, UAT, etc.?
• Anyone want to run it locally?
50. Avi Deitcher avi@atomicinc.com
When Black Holes Collide
Monolithic Software
• Buggy deployments
• Impossible to test
• Slow development
Manual Deployment
• Errors
• Long nights
• Impossible to verify
• Delayed and riskier deployments
51. Avi Deitcher avi@atomicinc.com
When Black Holes Collide
Monolithic Software
• Buggy deployments
• Impossible to test
• Slow development
Manual Deployment
• Errors
• Long nights
• Impossible to verify
• Delayed and riskier deployments
Unsustainable!
52. Avi Deitcher avi@atomicinc.com
Where is Neverland?
• Independent services
• Development
• Deployment
• Scale
• Fully automated testing
• Monthly Daily Hourly (lim → 1/∞) release cycle
• Identical artifacts+config across all environments laptop through to prod
• Self-healing
53. Avi Deitcher avi@atomicinc.com
Where is Neverland?
• Independent services
• Development
• Deployment
• Scale
• Fully automated testing
• Monthly Daily Hourly (lim → 1/∞) release cycle
• Identical artifacts+config across all environments laptop through to prod
• Self-healing
Mission: Customers work and employees sleep
58. Avi Deitcher avi@atomicinc.com
Ownership
Infrastructure
• Manage / maintain underlying
infrastructure
• AWS-centric
• Infrastructure as Code… only
• CLIs and consoles
• No app deploys!
• Expose app orchestration service to
developers
Application Development
• Engineer & operate services
• Testing, config files
• Individual services
• Leverage app orchestration service
provided by infrastructure
• 99.99% agnostic where orchestration
runs
Clear lines of responsibility
Everyone engineers and operates their parts
74. Avi Deitcher avi@atomicinc.com
Who’s Afraid of the Big Bad Wolf?
• When one project gets killed…
… how happy are you that you designed it correctly?
… analytics to the rescue! (or the other way ‘round?)
75. Avi Deitcher avi@atomicinc.com
Who’s Afraid of the Big Bad Wolf?
• When one project gets killed…
… how happy are you that you designed it correctly?
… analytics to the rescue! (or the other way ‘round?)
• Infrastructure
• Modular infrastructure prep
• 3 terraform modules – vpc, vpn, kubernetes
• Makefiles and scripts to build/leverage CAs, ssh, keys, FQDNs
• We had to change nothing
76. Avi Deitcher avi@atomicinc.com
Who’s Afraid of the Big Bad Wolf?
• When one project gets killed…
… how happy are you that you designed it correctly?
… analytics to the rescue! (or the other way ‘round?)
• Infrastructure
• Modular infrastructure prep
• 3 terraform modules – vpc, vpn, kubernetes
• Makefiles and scripts to build/leverage CAs, ssh, keys, FQDNs
• We had to change nothing
• Application
• Same templates, same concepts
• Same CI/CD
83. Avi Deitcher avi@atomicinc.com
Proof is in the
PodA
PodB
PodC
PodD
External R
External S External T
Hard to deploy
Hard to deploy in cluster
Hard to configure for cluster services
Brutal to configure for simultaneous
internal and external access
84. Avi Deitcher avi@atomicinc.com
Proof is in the
PodA
PodB
PodC
PodD
External R
External S External T
Hard to deploy
Hard to deploy in cluster
Hard to configure for cluster services
Brutal to configure for simultaneous
internal and external access
Easier to deliver short-term as standalone
85. Avi Deitcher avi@atomicinc.com
Proof is in the
“I want just another cluster service!”
PodA
PodB
PodC
PodD
External R
External S External T
Hard to deploy
Hard to deploy in cluster
Hard to configure for cluster services
Brutal to configure for simultaneous
internal and external access
Easier to deliver short-term as standalone
93. Avi Deitcher avi@atomicinc.com
Takeaways
• Number one thing you are changing is culture
• Number two thing you are changing is culture
• Number three thing you are changing is lives
• Take your time, build your way up
• Never expect your team to be where you are, or
to be all on the same page
• Learn their pain, learn the existing technology
and business, gain respect by respecting them
• Involve the team in the process; it matters as
much as the outcome
• Make sure people really get the “why” (“First 90
Days”)
• Get good sponsors
• Find your champions (allies)
• Choose your battles, accept that some things
have to be “good enough”
94. Avi Deitcher avi@atomicinc.com
Takeaways
• Number one thing you are changing is culture
• Number two thing you are changing is culture
• Number three thing you are changing is lives
• Take your time, build your way up
• Never expect your team to be where you are, or
to be all on the same page
• Learn their pain, learn the existing technology
and business, gain respect by respecting them
• Involve the team in the process; it matters as
much as the outcome
• Make sure people really get the “why” (“First 90
Days”)
• Get good sponsors
• Find your champions (allies)
• Choose your battles, accept that some things
have to be “good enough”
Technology takeaways?
• Choose your toolset well
• Expect issues; the technology is not turnkey or fully mature… but what technology is?
• Architecture matters
• Details?