We’ve been told the future is automated, orchestrated, service-oriented applications built on top of dynamic, on-demand, massively scalable infrastructure (serverless anyone?). But how do you go from here to there? At Betterment we built a platform that goes beyond “lift and shift”. In this presentation, I’ll share how we built a platform for tomorrow, with support for today’s legacy apps. I’ll also share how we designed the tooling to encourage developers to build for the future, while ensuring security and reliability, going deep into how the tooling ultimately guided our own path, and why the principles behind everything helped us stay the course.

1. Subverting the
Monolith
With Principles and Tooling
@girlnamedsophia

2. We’ve been told the
future is…
Automated
Serverless
In the cloud
Distributed
Scalable
Flexible
Resilient
Secure

4. How do you go from here
to there?
Here There

5. What’s here?
Here

6. …where were we?

7. • Apps aren’t 12 factor
…where were we?

8. • Apps aren’t 12 factor
• Not all hosts are ephemeral
…where were we?

9. • Apps aren’t 12 factor
• Not all hosts are ephemeral
• High deploy latency - 90 minute deploys
…where were we?

10. • Apps aren’t 12 factor
• Not all hosts are ephemeral
• High deploy latency - 90 minute deploys
• Low reliability in CI/CD
…where were we?

11. • Apps aren’t 12 factor
• Not all hosts are ephemeral
• High deploy latency - 90 minute deploys
• Low reliability in CI/CD
• High onboarding cost for new apps
…where were we?

12. • Apps aren’t 12 factor
• Not all hosts are ephemeral
• High deploy latency - 90 minute deploys
• Low reliability in CI/CD
• High onboarding cost for new apps
…where were we?

13. • Apps aren’t 12 factor
• Not all hosts are ephemeral
• High deploy latency - 90 minute deploys
• Low reliability in CI/CD
• High onboarding cost for new apps
…where were we?

14. What’s 12 factor?

15. • Code in VCS
• Dependencies declared and
isolated
• Config stored in the
environment
• Backing services communicated
over an API
• Internal Services available
via an API
• Concurrency and scaling
through the process model
• Ephemeral
• Environment agnostic
• Logs as event stream
• Admin tasks in production

16. So what is “there”?
There

17. Serverless!
Containerized!
Environment Agnostic!
Declarative Dependencies!
Backing services
communicated over an API!

18. Automated Goats

19. Consumer Contract
CI pipeline
CD pipeline
Onboard new apps

20. And when do we want it?

21. ...Eventually!

22. How do we want it to
happen?

23. ...Gradually!

24. Why don’t we have it
already?

26. How we built the platform
today, thinking of tomorrow
Solved one problem at a time

27. Solved one problem at a time
Implemented Configuration as Code
How we built the platform
today, thinking of tomorrow

28. Solved one problem at a time
Implemented Configuration as Code
Prioritized an engineers-first approach
How we built the platform
today, thinking of tomorrow

29. Solved one problem at a time
Implemented Configuration as Code
Prioritized an engineers-first approach
How we built the platform
today, thinking of tomorrow
…okay, engineers second, Principles first

30. Legacy System
App

31. Which problem to solve
first?

32. Consumer Contract
CI Pipeline
CD Pipeline
Onboard new apps

33. Generic solution

34. Generic solution
Varying needs

35. Generic solution
Varying needs
Usable Value right off the
bat

36. GitHub Jenkins GitHub Jenkins
Slack
AWS
JenkinsJenkinsJenkinsJenkins
Sad engineers
shake fists
CI/CD not in VCS
Onboarding new
apps is
difficult
Sad
Engineer
CI fumbles
Not Atomic!

37. GitHub Jenkins GitHub Jenkins
Slack
AWS
JenkinsJenkinsJenkinsJenkins
Sad
Engineer
CI fumbles

38. Configuration As Code

39. Configuration As Code
Automation

40. Configuration As Code
Automation
Opinionated AF

41. As engineers, what’s your go to
language of choice? What do you
write most of your tools things in?
1. Ruby
2. Elixir
3. Golang

42. You’re wrong!
It’s YAML.

43. So what did we do?
We wrote YAML…
WITH RUBY

44. Enforces a set of conventions
What’s the tool do?

45. What’s the tool do?
Enforces a set of conventions
Defines a predetermined contract

46. Enforces a set of conventions
Defines a predetermined contract
Automates configuration generation
What’s the tool do?

47. Why did we do it?

48. Externalized Configuration
Why a contract?

49. Externalized Configuration
Automation
Why a contract?

50. Externalized Configuration
Automation
Scalability
Why a contract?

51. Why a contract?
Externalized Configuration
Automation
Scalability
Standardization

52. Why a contract?
Externalized Configuration
Automation
Scalability
Standardization
Predictability

53. Why a contract?
Externalized Configuration
Automation
Scalability
Standardization
Predictability

54. Why a contract?
Externalized Configuration
Automation
Scalability
Standardization
Predictability

55. RSpec
Rubocop
Integration Tests
Error Screenshots
Standardized Builds
Conventionally stored artifacts

56. Next, Enforce the
Contract

57. How’s it work?
ruby_app java_appjava_library
…
Coach CLI

58. How’s it work?
ruby_app
ruby_library
java_library

59. How’s it work?
ruby_app
ruby_library
java_library
ruby 2.6.1
ruby 2.4.2
java 1.8.0

60. How’s it work?
ruby_app
ruby_library
java_library

61. How’s it work?

63. Consumer Contract
CI Pipeline
CD Pipeline
Onboard new apps

64. The Structure

65. The Structure

66. Tight coupling! Nooooo!

67. Welcome to software
development

68. The Code

69. The Code

70. The Code

74. The Legacy

80. The Code

81. The Code

82. The Code

83. The Code

84. The Code

85. #portability

86. …But I digress

87. …where are we now?

88. …where are we now?
• Apps start to look a lot more 12 factor-y

89. …where are we now?
• Apps start to look a lot more 12 factor-y
• Not all hosts are ephemeral

90. …where are we now?
• Apps start to look a lot more 12 factor-y
• Not all hosts are ephemeral
• Low deploy latency - 25-35 minute deploys

91. …where are we now?
• Apps start to look a lot more 12 factor-y
• Not all hosts are ephemeral
• Low deploy latency - 25-35 minute deploys
• High reliability in CI, not yet CD

92. …where are we now?
• Apps start to look a lot more 12 factor-y
• Not all hosts are ephemeral
• Low deploy latency - 25-35 minute deploys
• High reliability in CI/CD, not yet CD
• Low onboarding cost for new apps

93. …where are we now?
• Apps start to look a lot more 12 factor-y
• Not all hosts are ephemeral
• Low deploy latency - 25-35 minute deploys
• High reliability in CI/CD, not yet CD
• Low onboarding cost for new apps

94. …where are we now?
• Apps start to look a lot more 12 factor-y
• Not all hosts are ephemeral
• Low deploy latency - 25-35 minute deploys
• High reliability in CI/CD, not yet CD
• Low onboarding cost for new apps

95. How our tooling Encouraged
Developers to Build for the Future

96. Consumer Contract
CI Pipeline
CD Pipeline
Onboard new apps

97. From legacy to
new world

98. What’s the smallest
deployable unit?

99. An app

100. An app is an app is an app

101. Helping the migration

102. What is but isn’t an app?
• Database migrations for a shared
database
• Database monitoring that needs to run
on its own host

103. Running the code in
the cloud

104. Rudder guides the boat

105. The tools reinforce
each other

106. What language or tool do you use
to keep infrastructure in code?
1. YAML
2. Terraform
3. Ansible

107. Terraform with Ruby
Helm with Ruby

110. The evolution of the platform
ruby_app
database_
monitoring
_app
database_
migration_
app
ruby_app_
database

111. Attachable resources
ruby_app
database_
monitoring
_app
database_
migration_
app
ruby_app_
database

112. The engineers and the
principles guide the tools

113. Consumer Contract
CI Pipeline
CD Pipeline
Onboard new apps

114. Define the contract
Enforce the contract
Onboard

115. Onboarding apps

116. Distributing it all

117. Ease of distribution
means ease of adoption

118. Did we subvert the
monolith?

119. Yes!

120. • Apps are 12 factor!
• All hosts are ephemeral, we can even
support one-off ecosystems with ease
• Low deploy latency - 10-25 minute deploys
• High reliability in CI/CD
• Low onboarding cost for new apps
…where are we now?

121. GitHub Jenkins GitHub Jenkins
Slack
AWS
JenkinsJenkinsJenkinsJenkins
Sad
Engineer

122. GitHub CircleCI Coach Web Slack
Coach CLI
GitHub
Happy
Engineer

123. GitHub CircleCI Coach Web Slack
Coach CLI
GitHub
Happy
Engineer
Jenkins
AWS EKS
Rudder

124. Takeaways

126. Sometimes the solution can be
well-designed and knowingly
temporary

127. Automation now is better
than manual later

128. Delivering value as quickly as
possible without sacrificing
quality is critical to growth

129. With clearly defined principles
and goals, know that you never
need to stop iterating

130. Our present is…
Automated
Serverless
In the cloud
Distributed
Scalable
Flexible
Resilient
Secure

131. The Path Forward

132. Thank you