How to scale continuous delivery by thinking holistically about organization, tools, process and culture to be fast and successful.

1. Scaling Continuous Delivery to
minimize risks

2. Why Continuous Delivery?

3. For Speed

4. Speed is a lever to mitigate risk

5. Typical cycle of software delivery
Hypothesis
Generation
Development Experimentation
Validation

6. How do you scale it?

7. Foundational elements to scale it
O
R
G
A
N
I
Z
A
T
I
O
N
T
O
O
L
S
P
R
O
C
E
S
S
C
U
L
T
U
R
E
CONTINUOUS DELIVERY

8. Organization

9. Spotify Org Model

10. Takeaways
● Tribes : Optimize team composition for autonomy
● Alliances : Organize team groupings for alignment
● Guilds : Foster channels for knowledge sharing to break silos
● Tribes mapped to metrics : Have explicit accountability by
assigning team metrics

11. Story of our Aha! moment

12. Real story anyone?

13. Series of aha!, aha!, aha!, aha! ...

14. There were 4 phases

15. The startup phase
● 1 location
● 50+ engineers
● Teams mapped to features

16. Teams = Features

17. The expansion phase
● 1 engineering hub in Stockholm
● 2 small engineering offices in NYC and Gothenburg
● 100+ engineers
● Teams still mapped to features but shared ownership across
geographies

18. The autonomy phase
● Global company with 2 engineering hubs in NYC and Stockholm
● Multiple smaller engineering offices
● 250+ engineers
● Geo local missions and full local autonomy

19. Teams Tribes = Missions

20. The alignment phase
● Global company with multiple engineering offices
● 500+ engineers
● 10+ autonomous missions
● Many missions working towards same or similar metric

21. Teams Tribes Alliances =
Related Missions

22. One more thing ...
Spotify Org model Universal Org Model

23. Tools

24. Make pipelines fast and low-touch

25. Enable ease of deployment
● Think Containers to enable easy deployments
● Standardize production and test setup to enable easy build
progression

26. Make tests visible
Git
PR
Pre-merge
Tests
Pass
Post-merge
Tests
Log to TRS
View Test
Results

28. Optimize build times
● Minimize recompiles of the whole dependency tree
● Pre-compile common dependencies and store as binaries
● Think hermetic builds

29. Maximize Automation, Monitoring
and Alerting

30. Make experimentation part of the delivery flow

31. Feature Flags rollouts AB Tests

32. Process

33. Standardize Release
Management

34. Feature
Complete
Beta Testers & Apple /
Incremental Rollout to
Android
Master Branch
Release Branch
Release Blocking Bug
Fix(es)
2 week client release cycle

35. Enable Speedy Reviews

36. Review flow
Software Design
Review
Software
Development
Code
Review
Pre-release
Review
Software
Design
Software
Release

37. Speedy Reviews at Spotify
● Code merges require +1 during code review but aren’t blocked
● Changes are tracked for auditing purposes
● Software review is front-loaded to avoid delivery blockers.
Encourages thinking through completely about architectural setup,
complexity and dependencies

38. Have a company operating rhythm

42. Culture

43. Quality is owned by everyone but
managed by QA

44. Agile is not a process but a
mindset

45. Be explicit about beliefs and
expectations

46. Thank You!
Email : kinshuk@spotify.com
Twitter : @_kinshukmishra