Continuous Deployment of Architectural Change
•
6 likes•1,357 views
Continuous deployment has proven to be a successful and even addicting part of Etsy's engineering culture. See where it's applicable, some of the tools that make it easy, and the kind of architectural change that it makes possible.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Continuous Deployment of Architectural Change
Similar to Continuous Deployment of Architectural Change (20)
Recently uploaded
Recently uploaded (20)
Continuous Deployment of Architectural Change
- 1. Continuous Deployment of Architectural Change Matt Graham Core Engineer @ Etsy Continuous Deployer #iasany September 27, 2011
- 2. A Brief History of Deployment
- 3. The Internet
- 5. Agility
- 6. Continuous
- 7. Database Deployment @ Etsy Thursday
- 8. What it's all about ● Broadly Applicable ● Tools Help ● Enables the Unfeasible
- 9. MTTR vs MTTF
- 10. MTTR vs MTTF ● Cheaper for electrons
- 11. MTTR vs MTTF ● Cheaper for ● Cheaper for electrons protons & neutrons
- 12. MTTR vs MTTF ● Cheaper for ● Cheaper for electrons protons & neutrons ● Cost prohibitive for humans
- 13. MTTR vs MTTF ● Cheaper for ● Cheaper for electrons protons & neutrons ● Cost prohibitive for humans ● Necessitates low MTTD
- 14. The Case of Electrons ● Consider a project that has 6 bugs ● continuous deployment testing misses 4 & deploys fixes in 4 hours ● monthly release testing misses only 2 & deploys fixes in 24 hours ● 16 hours of broken < 48 hours of broken
- 15. Good Excuses ● Infrequent Changes
- 16. Good Excuses ● Infrequent Changes ● Infrequent Executions
- 17. Good Excuses ● Infrequent Changes ● Infrequent Executions ● Life and Death
- 18. Good Excuses ● Infrequent Changes ● Infrequent Executions ● Life and Death ● Physical Investment
- 26. Deployment Tools @ Etsy
- 27. Culture Before Tools ● No planned deploys; only on demand
- 28. Culture Before Tools ● No planned deploys; only on demand ● A positive change is ready & tested
- 29. Culture Before Tools ● No planned deploys; only on demand ● A positive change is ready & tested ● Software is stable & supported
- 30. Jenkins ● Unit Tests
- 31. Jenkins ● Unit Tests ● Functional Tests
- 32. Jenkins ● Unit Tests ● Functional Tests ● Manual Testing
- 34. tail -f | grep
- 36. IRC
- 37. Key Tools Communication Tools Graphs Feature Flags
- 38. Feature Flags Deploy != Product Launch
- 39. Dark Launch def get_feature_a_link(): return ...
- 40. Dark Launch def get_feature_a_link(): if enabled('NewFeatureA'): return new_feature_a_link() else: return old_feature_a_link()
- 41. Dark Launch application_config - NewFeatureA: admin - NewFeatureB: off - NewFeatureC: on
- 42. Ramp Up application_config - NewFeatureA: 1% - NewFeatureB: off - NewFeatureC: on
- 43. Ramp Up application_config - NewFeatureA: 5% - NewFeatureB: off - NewFeatureC: on
- 44. Ramp Up application_config - NewFeatureA: 25% - NewFeatureB: off - NewFeatureC: on
- 45. Ramp Up application_config - NewFeatureA: 100% - NewFeatureB: off - NewFeatureC: on
- 46. AB Testing ● Prove success of interface changes ● Prove interest in new features
- 47. Graphs!!!
- 48. Ganglia
- 49. Graphite
- 50. Event Overlay
- 51. StatsD if ($success) { StatsD::timing('feature.runtime', $time); } else { StatsD::increment('feature.failure'); } github.com/etsy/statsd
- 55. Forums / Message Boards
- 56. etsystatus.com
- 59. Deployment is First Class Deployment is a First Class Feature
- 60. Engineers are Users Too
- 65. Premature Architecture What now?
- 66. Passing Time => Change ● Scale ● Product ● Technology ● Engineering Team
- 67. Passing Time => Change ● Scale ● Product ● Technology ● Engineering Team ● The Correct Architecture Changes
- 68. Examples ● Images From Twisted to PHP ● From PostgreSQL to MySQL Shards
- 69. From Twisted to PHP ● Run Apache/PHP on a new port
- 70. From Twisted to PHP ● Run Apache/PHP on a new port ● Implement one service in PHP
- 71. From Twisted to PHP ● Run Apache/PHP on a new port ● Implement one service in PHP ● Ramp up users on new service
- 72. From Twisted to PHP ● Run Apache/PHP on a new port ● Implement one service in PHP ● Ramp up users on new service ● Repeat for remaining services
- 73. From Twisted to PHP ● Run Apache/PHP on a new port ● Implement one service in PHP ● Ramp up users on new service ● Repeat for remaining services ● Shut down Twisted version
- 74. PostgreSQL to MySQL Shards ● Migrate table by table
- 75. PostgreSQL to MySQL Shards ● Migrate table by table ● Teed writes to MySQL
- 76. PostgreSQL to MySQL Shards ● Migrate table by table ● Teed writes to MySQL ● Backfill old data from PostgreSQL
- 77. PostgreSQL to MySQL Shards ● Migrate table by table ● Teed writes to MySQL ● Backfill old data from PostgreSQL ● Verify data matches in both DBs
- 78. PostgreSQL to MySQL Shards ● Migrate table by table ● Teed writes to MySQL ● Backfill old data from PostgreSQL ● Verify data matches in both DBs ● Ramp up users reading from MySQL
- 79. PostgreSQL to MySQL Shards ● Migrate table by table ● Teed writes to MySQL ● Backfill old data from PostgreSQL ● Verify data matches in both DBs ● Ramp up users reading from MySQL ● Stop Postgres writes & drop tables
- 80. Continuous Deploy Pattern ● Change in small steps
- 81. Continuous Deploy Pattern ● Change in small steps ● Dark launch placeholder via config
- 82. Continuous Deploy Pattern ● Change in small steps ● Dark launch placeholder via config ● Deploy to production while dark
- 83. Continuous Deploy Pattern ● Change in small steps ● Dark launch placeholder via config ● Deploy to production while dark ● Maintain old & new in parallel
- 84. Continuous Deploy Pattern ● Change in small steps ● Dark launch placeholder via config ● Deploy to production while dark ● Maintain old & new in parallel ● Ramp up to new architecture
- 85. Continuous Deploy Pattern ● Change in small steps ● Dark launch placeholder via config ● Deploy to production while dark ● Maintain old & new in parallel ● Ramp up to new architecture ● Remove old architecture
- 86. Review ● Broadly applicable ● Key Tools: Feature Flags, Graphs, Communication ● Enables Architectural Change ● It's really fun!!!
- 87. Continuous Deployment of Architectural Change Matt Graham http://twitter.com/lapsu http://lapsu.tv Core Engineer @ Etsy Continuous Deployer http://codeascraft.etsy.com We're Hiring! http://bit.ly/etsywantsawesome