The talk is about a real experience of organization transformation with 15 years old legacy monolith project into DevOps oriented, microservice-based modern company. On our way, we have grown from 40 to 80 engineers for the last two years. We have x20 increased delivery of the old main project and implemented CD for projects and processes.
4. 4
3y 4y 5y 3y
Developer Developer & Founder Team Lead & Architect VP of Engineering
@ Various companies @ Startup @ Yandex @ WorldAPP
15 years in practical engineering
80 people in the department
10+ projects from initial commit to production
5. Form.com in 2015
● Leading inspection & audit platform in US
● 12 years on the market
● 40 engineers
● 2M lines of legacy code
5
6. Goals
● Improve delivery quality & customer retention
● Release more features for same period of time
● Create a template for department scaling
● Standardize and organize processes
6
7. Form.com early years
7
Full regression 5 engineers, 4-5 months
Unit/integration tests run 10115pcs, 1h26m
App design Monolith / monorepo
Departments organization Silos
Uptime 99.57% (down 3h8m/month)
8. What do we know about downtimes
● According to Gartner an hour of downtime costs ~300k$
● Our estimated cost is ~6000$/hour (direct revenue)
● 99.5% uptime is a 3h 39m downtime monthly
● The direct loss is about 250k$ yearly
8
10. Tackle long feedback cycle issue
● Hire more engineers
● Reduce QC scope
● Isolate merges by code split
● Improve tests run performance
● Automate manual work
10
16. SOA design
● Modules inside legacy project
● New services for new parts
● Strategy for old modules extraction
● Common middleware approaches
● Common design and guidelines
16
18. Organizations which design systems ... are constrained to
produce designs which are copies of the communication
structures of these organizations.
18
— Melvin Conway
19. A team
19
TEAM
knows what
needs to be done
knows how
to develop
can impartially assess
the quality
knows how it would work
in real world
Team is responsible for what and when should be delivered.
20. Application template
An application
java -jar app.jar - to run application
:8080 - for API calls
/health - for health check
local SYSLOG - for logging
local collectd - for monitoring
20
24. CD can bring some peace
● No single person who knows how to deploy
● No need to waste someone's time on complicated task
● Deployment to the same environments guarantee
● No scare of new releases
24
25. How all of that helped
● Faster onboarding and issues triangulation
● Less time on operational tasks
● Less “own way” decisions in application design
● True agility in teams redesign
25
26. Where we are after 3 years
26
Full regression 5 engineers, 4-5 months 3 engineers, 5 days 3000%
Unit/integration tests run 10115pcs, 1h26m 14192pcs, 40m 294%
App design Monolith / monorepo REST SOA
Departments organization Silos 6 Service teams
Uptime
99.57% (down
3h8m/month) 99.92% (down 35m/month) 537%
27. Let’s recap
● Find allies
● Look for timesavers
● Wisely consider your options
● Define measurable goals
27