My talk for DevOps at Box.net about "scaling": scaling your development, testing, deployment, and monitoring infrastructure.
Sorry about the slide notes, it's screwed up a bit. I'll see if I can fix them.
5. Disclosures:
https://www.wealthfront.com/benefits/performance
https://www.wealthfront.com/research/mutual-fund-fees-overview
Learn about us at
quora.com/wealthfront
6. Disclosure: Learn about us at
https://www.wealthfront.com/legal/disclaimer quora.com/wealthfront
7. Managing close to $180M*
Processing over $2M/day
Highly regulated: FINRA, SEC, Member SIPC
We’re a technology company
No ops, no QA
* ~$30M AUM ~$150 AUA
Learn about us at
quora.com/wealthfront
9. Ideas
LEARN BUILD
Data Code
MEASURE (Eric Ries)
Learn about us at
quora.com/wealthfront
10. Ideas
LEARN BUILD
“validated learning”
Data Code
MEASURE (Eric Ries)
Learn about us at
quora.com/wealthfront
11. Ideas
LEARN BUILD
This is what
“validated learning”
you scale.
Data Code
MEASURE (Eric Ries)
Learn about us at
quora.com/wealthfront
12. with
LEARN
This is what
Ideas
“validated learning”
BUILD
code
you scale.
Data Code
MEASURE (Eric Ries)
Learn about us at
quora.com/wealthfront
13. Development Stage QA Fix P1 Bugs Release
Cut a Release Integrate Patches
1‐4 Weeks
Learn about us at
quora.com/wealthfront
14. Development Stage QA Fix P1 Bugs Release
Cut a Release Integrate Patches
Learn about us at
quora.com/wealthfront
15. Development Stage QA Fix P1 Bugs Release
Cut a Release Integrate Patches
Learn about us at
quora.com/wealthfront
16. Development Stage QA Fix P1 Bugs Release
Cut a Release Integrate Patches
Learn about us at
quora.com/wealthfront
17. Development Stage QA Fix P1 Bugs Release
Cut a Release Integrate Patches
Learn about us at
quora.com/wealthfront
18. Development Stage Automated QA Fix P1 Bugs Release
TesKng
Cut a Release Integrate Patches
Learn about us at
quora.com/wealthfront
19. Development Stage Automated QA Fix P1 Bugs Release
TesKng
Cut a Release Integrate Patches
Learn about us at
quora.com/wealthfront
20. Development Stage Automated QA Fix P1 Bugs Release
TesKng
Cut a Release Integrate Patches
Learn about us at
quora.com/wealthfront
21. Development Stage Automated QA Fix P1 Bugs Release
TesKng
Cut a Release Integrate Patches
Learn about us at
quora.com/wealthfront
22. Deployment
Development Stage Automated QA Fix P1 Bugs Release
TesKng
Cut a Release Integrate Patches
Learn about us at
quora.com/wealthfront
23. Deployment
Development Stage Automated QA Fix P1 Bugs Release
TesKng
Cut a Release Integrate Patches
Experiments
Learn about us at
quora.com/wealthfront
24. Deployment
Development Stage Automated QA Fix P1 Bugs Release
TesKng
Cut a Release Integrate Patches
Experiments Monitoring
Learn about us at
quora.com/wealthfront
25. Deployment
Development Stage Automated QA Fix P1 Bugs Release
TesKng
Cut a Release Integrate Patches
Experiments Monitoring
5‐10 Minutes
Learn about us at
quora.com/wealthfront
31. Culture
ConKnuous
Test Driven
IntegraKon
Development
Learn about us at
quora.com/wealthfront
32. Immune
System
Culture
ConKnuous
Test Driven
IntegraKon
Development
Learn about us at
quora.com/wealthfront
33. ConKnuous Deployment
Immune
System
Culture
ConKnuous
Test Driven
IntegraKon
Development
Learn about us at
quora.com/wealthfront
34. every problem only once
stop the line if anything fails
fast response vs. prevention
(Eric Ries)
Learn about us at
quora.com/wealthfront
35. e nt
pm
ve
lo trunk‐stable
de
small commits
fwd/back compaKbility
trivial rollbacks
code experiments
broken build →
Learn about us at
quora.com/wealthfront
36. e nt
lo pm
ve
de
commit messages can
do stuff:
#release:um
#cc:david
#pleasereview
Learn about us at
quora.com/wealthfront
37. ti ng eng ⊃ qa
t es
only automated tesKng
forbidden calls
bad code snippets
stuff‐not‐tested‐test
hitchfs (fs stub)
LessIOSecurityManager
Learn about us at
quora.com/wealthfront
38. ti ng
t es eng ⊃ qa
only automated tesKng
hudson
selenium
capybara
other fe magic that
I don’t know jack about
Learn about us at
quora.com/wealthfront
39. de
eng ⊃ ops plo
y me
Running, nt
accepting prod
traffic
zk ftw! Automated
Unannounce self-test rollback monitoring
fail fail
Clients stop Accept prod
Self test
requesting traffic
Start new Announce
Shut down zk ftw!
version
(canaries +
exponential deploys) Learn about us at
quora.com/wealthfront
42. !release <service>
!restart <service>
!buildstatus
!alerts
!markfixed <alertid>
!jirafy <alertid>
...
“Your most talented engineers should
be working on your tools, and your
culture must reflect this priority.”
- Yishan Wong (Facebook)
Learn about us at
quora.com/wealthfront
43. shell ikq curl http+json services
ad-hoc power tools
Learn about us at
quora.com/wealthfront
44. shell ikq curl http+json services
ad-hoc power tools
Learn about us at
quora.com/wealthfront
45. Thanks!
@arosien ⊂ @wltheng
http://eng.wealthfront.com
8
You can open an IRA as late
1
as April 18th 2011
(really, google it)
and it apply to 2010.
Learn about us at
quora.com/wealthfront
Notas do Editor
\n
Big market &#x201C;As of October 2007, there are 8,015 mutual funds that belong to the Investment Company Institute (ICI), a national trade association of investment companies in the United States, with combined assets of $12.356 trillion&#x201D; (from Wikipedia)\n\nBusiness model\nCut of management fees\nMargin on trading commissions\n
Big market &#x201C;As of October 2007, there are 8,015 mutual funds that belong to the Investment Company Institute (ICI), a national trade association of investment companies in the United States, with combined assets of $12.356 trillion&#x201D; (from Wikipedia)\n\nBusiness model\nCut of management fees\nMargin on trading commissions\n
Big market &#x201C;As of October 2007, there are 8,015 mutual funds that belong to the Investment Company Institute (ICI), a national trade association of investment companies in the United States, with combined assets of $12.356 trillion&#x201D; (from Wikipedia)\n\nBusiness model\nCut of management fees\nMargin on trading commissions\n
Our goal is to connect investors with outstanding investment managers.\n\nRetail customers (net worth less than $1M) are underserved. The only actively managed investment vehicles available are mutual funds.\n\nSchwab created Schawb One Source which is THE marketplace for mutual funds.\n\nWe want to create the equivalent for individually managed accounts, essentially bringing wealth management to retail.\n\nOur foundational values are transparency and aligning interests. You can know everything about the investment manager you are entrusting. His portfolio holdings, his past transaction history, his rational, philosophy. And we use all this data to objectively vet managers. We are creating a platform for win-win situations.\n\nIn 6 months, we&#x2019;ve attracted over $11M and are growing very fast.\n\nWe&#x2019;re an SEC regulated company, en route to being FINRA regulated as a broker/dealer.\n
Our goal is to connect investors with outstanding investment managers.\n\nRetail customers (net worth less than $1M) are underserved. The only actively managed investment vehicles available are mutual funds.\n\nSchwab created Schawb One Source which is THE marketplace for mutual funds.\n\nWe want to create the equivalent for individually managed accounts, essentially bringing wealth management to retail.\n\nOur foundational values are transparency and aligning interests. You can know everything about the investment manager you are entrusting. His portfolio holdings, his past transaction history, his rational, philosophy. And we use all this data to objectively vet managers. We are creating a platform for win-win situations.\n\nIn 6 months, we&#x2019;ve attracted over $11M and are growing very fast.\n\nWe&#x2019;re an SEC regulated company, en route to being FINRA regulated as a broker/dealer.\n
\n
our context is extreme uncertainty\nif we can reduce the time to learn which way to go, we can increase our odds of success before we run out of money\n
eric ries&#x2019; graphic\n
eric ries&#x2019; graphic\n
eric ries&#x2019; graphic\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
\n
stuff on the shelves = waste\n
if you wait two weeks to deploy, there&#x2019;s so much that can go wrong\nso if you&#x2019;re testing the failure cases all the time you are much safer\n
bucket: heavy, imprecise, can spill\nhose: precise, controllable, flexible\n
Culture: takes a lot of discipline to achieve CD, it&#x2019;s about realizing we are human and do mistakes, risk mitigation everywhere\n\nContinuous integration: engineers must be aware of test success or failures, important to have physical reminders &#x201C;build monitor&#x201D;\n\n\n
Culture: takes a lot of discipline to achieve CD, it&#x2019;s about realizing we are human and do mistakes, risk mitigation everywhere\n\nContinuous integration: engineers must be aware of test success or failures, important to have physical reminders &#x201C;build monitor&#x201D;\n\n\n
Culture: takes a lot of discipline to achieve CD, it&#x2019;s about realizing we are human and do mistakes, risk mitigation everywhere\n\nContinuous integration: engineers must be aware of test success or failures, important to have physical reminders &#x201C;build monitor&#x201D;\n\n\n
Culture: takes a lot of discipline to achieve CD, it&#x2019;s about realizing we are human and do mistakes, risk mitigation everywhere\n\nContinuous integration: engineers must be aware of test success or failures, important to have physical reminders &#x201C;build monitor&#x201D;\n\n\n
the continuous deployment philosophy (eric ries)\n
\n
our continuous deployment system is hooked into the build system\nso in our commit message we can say: if the build passes then commit service UM\n\n
every new engineer pushes to production their first day\nno deep training, the tests give \n
every new engineer pushes to production their first day\nno deep training, the tests give \n
\n
\n
This wasn&#x2019;t a good day for us. You can see two rollbacks in the bottom right hand corner.\n\nWe missed issues in testing which got out to production. Because of our investment in production infrastructure, we had those issues for about 45 seconds total, before our code automatically rolled back our bad code.\n\nI can also say that because of our process, we won&#x2019;t have either of those issues again.\n\nWhat could have been a very bad day, became mediocre at worst. And the things we learned will make us more stable going forward.\n
the continuous deployment philosophy (eric ries)\n
the continuous deployment philosophy (eric ries)\n
