Presented at LAST 2019:
So you walk into a new company, get the lay of the land and then realise, crap! Their development processes are like they were design by a bunch of first-year uni students doing a group project.
There is no DevOps to speak of. There are snowflake servers everywhere. Their git branching strategy is unmanageable. They run tests only every 3 or 4 releases. Their deployment is manual and different for each release. The have no real alerting.
Ok. Take a deep breath! Calm down.
So much to do, but where to start? The business has produced a list of improvement actions, but those actions are focussed around fixing the symptoms of the problems, not solving the root cause. The business does not understand that the path to DevOps improvement is complex and each task has many inter-relations and dependencies.
This is the problem that I faced about a year ago. To overcome this, we went through a process of defining all of the DevOps tasks we could think of and mapped them into a dependency diagram. This diagram was useful to communicate both internal and external to the team.
In this case study, I’ll go through the process to design the dependency diagram, but also our progress through the diagram one year later.
3. Hear an interesting story:
Learn something:
Get ideas:
What you’ll (hopefully) get out of this talk
DevOps = 0 → DevOps > 0
DevOps Dependencies
Planning DevOps Progression
4. Background: History
● 5 year old company
○ Produces SaaS Project Management Software
● Development outsourced
○ Insourcing development (owning core IP)
● Moving from startup to scaleup
○ Industrialisation
5. Background: Strengths and Weaknesses
Strengths
Company
● Product validated in the market
○ Multiple customers who are strong
advocates of the product
● Positive culture
○ Ego-less, highly experienced, willing to
listen
● Committed and willing to invest in
development
Development
● Using git
● Have some unit tests
● Have extensive functional tests
● Scripts to manage deployment
Weaknesses
Company
● Nobody in the company has any Software
Product Development experience
● Communication difficulties to Vietnam, so
require very prescriptive requirements
Development
A bit more extensive...
6. Background: Weaknesses (Development)
● Git: Unmanageable and divergent branching
● Testing
○ No CI
○ Unit tests: run once a month. Don’t all pass
○ Functional tests: 30% fail rate (i.e. useless)
● Deployment
○ Snowflake servers (no IaC)
○ Changes are run directly on production
○ Each environment has completely different
script, and it changes each release
○ Deployments: SSH to prod, git pull, restart
apache
● No monitoring or alerting
● Code:
○ General quality / maintainability
○ DRY = DO Repeat Yourself
Git branching strategy
7. Background: Challenges
● Transition from outsourced to insourced
○ Knowledge transfer difficult with communication barrier
● Commitments requiring premature maturity
○ AWS Well Architected Review
■ Requires autoscaling
○ SOC2
● Company full of Project Managers (need a plan)
● No experience with software development
○ E.g. Currently have a linear view of software development (no understanding of dependencies)
● Monolith Architecture
● Multiple production environments
8. Background: Challenges: SOC2
● SOC2: Service and Organization Controls
○ Provide an organization a way to demonstrate that security practices are in place and
operating effectively
● Some SOC2 Controls relating to development
○ Security: E.g.
■ Pen testing
■ Vulnerability scanning
■ Intrusion detection
■ Security incident analysis
○ Change Control: E.g.
■ Minimal admin access
■ Configuration monitoring
■ Segregation of duties (dev from ops)
■ Strict approval and Change Management process
9. Idea: design a DevOps dependency diagram
Different areas of Devops. E.g.
● Build automation
● Testing
● Deployment
● End-to-end testing
● Monitoring
● Infrastructure
● Configuration management
● Reporting
● etc
Automated
build
framework
Tasks with dependencies. E.g.
Unit tests run
automatically
Continuous
Integration
Continuous
Integration
Milestones. E.g.
Continuous
Deployment
10. What we did
Full version is here:
https://drive.google.com/file/d/1zJRATfrzF20uoN4
Z2zuCBCL8UmdYXUWV
12. Limitations of the diagram
● Iteration / improvement of progress of certain tasks
○ E.g. IaC or CI may be “done” once in place, but constantly improve over time
● Any remediation that does not have a definitive goal:
○ Code tech debt
○ Performance
○ Splitting out of services
○ Quality of testing