Mapping Etsy's Front-end
•
4 gostaram•1,431 visualizações
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1smUK82. Daniel Espeset talks about how Etsy built an incremental compiler for the JavaScript modules, and used it to see how static assets are compiled, being deployed, and loaded. Filmed at qconlondon.com. Daniel Espeset is a Software Engineer working on the Performance team at Etsy. He enjoys hacking on build processes, improving front-end architecture and pursuing moonshot experiments in code instrumentation and static analysis.
Recomendados
Recomendados
Mais conteúdo relacionado
Semelhante a Mapping Etsy's Front-end
Semelhante a Mapping Etsy's Front-end (20)
Mais de C4Media
Mais de C4Media (20)
Último
Último (20)
Mapping Etsy's Front-end
- 1. Friday, March 7, 14
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /javascript-compiler-deploy
- 3. Presented at QCon London www.qconlondon.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
- 4. Mapping Etsy’s Frontend Tools and Techniques for Managing Complexity at Scale @danielespeset Friday, March 7, 14
- 5. @danielespeset Seth Walker Allison McKnight Jonathan Klein @sethwalker @aemcknig @jonathanklein Daniel Espeset @danielespeset Brought to you by Friday, March 7, 14
- 7. @danielespeset A global marketplace for buying and selling unique goods. Etsy Friday, March 7, 14
- 11. @danielespeset 50 deploys a day Friday, March 7, 14
- 12. @danielespeset Deployinator IRC Push Queue Friday, March 7, 14
- 13. @danielespeset Anybody can touch anything. Friday, March 7, 14
- 14. @danielespeset Anybody can touch anything. try not to break the site Friday, March 7, 14
- 16. @danielespeset Engineers own their code through the entire stack. Friday, March 7, 14
- 18. @danielespeset Our frontend is made up of 300 different “pages” Friday, March 7, 14
- 19. @danielespeset At any given time we may be running dozens of experiments per page Friday, March 7, 14
- 21. @danielespeset 2 12 300 = 1,228,800x Friday, March 7, 14
- 22. @danielespeset OMG, how do actually you do anything? Friday, March 7, 14
- 23. @danielespeset 12 month growth of CSS code last year 992 files 197,812 lines last week 1,653 files (1.67x) 319,152 lines (1.61x) Friday, March 7, 14
- 24. @danielespeset 12 month growth of JavaScript code last year 1,451 files 425,915 lines last week 2,394 files (1.65x) 639,484 lines (1.5x) Friday, March 7, 14
- 27. @danielespeset html * #buttons button.button { modules/buttons.css Friday, March 7, 14
- 28. @danielespeset html * #buttons button.button { wtf? modules/buttons.css Friday, March 7, 14
- 29. @danielespeset $css/> grep “@import modules/buttons” Friday, March 7, 14
- 30. @danielespeset $css/> grep “@import modules/buttons” a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css Friday, March 7, 14
- 31. @danielespeset $css/> for ... grep “@import $1” ... a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css base-responsive base base_after_inline Friday, March 7, 14
- 32. @danielespeset $css/> for ... grep “@import $1” ... a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css /base-responsive /base /base_after_inline c/template_editor c/mini a/scout/dashboard Friday, March 7, 14
- 33. @danielespeset $templates/> for ... grep “{% css=$1” ... a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css /base-responsive /base /base_after_inline c/template_editor c/mini a/scout/dashboard wholesale/modules/vendor_avatar wholesale/modules/address_from_array wholesale/pages/purchase_order/modules/uneditable_fields/payment_method wholesale/pages/purchase_order/modules/uneditable_fields/payment_terms wholesale/pages/purchase_order/modules/uneditable_fields/ship_date wholesale/pages/purchase_order/modules/uneditable_fields/billing_address wholesale/pages/purchase_order/modules/uneditable_fields/shipping_address wholesale/pages/purchase_order/modules/product_grid modules/time_utils base Friday, March 7, 14
- 34. @danielespeset $templates/> for ... grep “{% css=$1” ... a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css /base-responsive /base /base_after_inline c/template_editor c/mini a/scout/dashboard wholesale/modules/vendor_avatar wholesale/modules/address_from_array wholesale/pages/purchase_order/modules/uneditable_fields/payment_method wholesale/pages/purchase_order/modules/uneditable_fields/payment_terms wholesale/pages/purchase_order/modules/uneditable_fields/ship_date wholesale/pages/purchase_order/modules/uneditable_fields/billing_address wholesale/pages/purchase_order/modules/uneditable_fields/shipping_address wholesale/pages/purchase_order/modules/product_grid modules/time_utils base modules/deferred_clientlogger modules/jquery modules/async_analytics_init notice/experiment_notice modules/locale/dismissible_link pages/recommendations/friends/v2/modules/gift-ideas-reminder header pages/translator/_modules/md5_overlay modules/breadcrumbs pages/browse/_modules/footer footer Friday, March 7, 14
- 35. @danielespeset $templates/> for ... grep “{% css=$1” ... a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css /base-responsive /base /base_after_inline c/template_editor c/mini a/scout/dashboard wholesale/modules/vendor_avatar wholesale/modules/address_from_array wholesale/pages/purchase_order/modules/uneditable_fields/payment_method wholesale/pages/purchase_order/modules/uneditable_fields/payment_terms wholesale/pages/purchase_order/modules/uneditable_fields/ship_date wholesale/pages/purchase_order/modules/uneditable_fields/billing_address wholesale/pages/purchase_order/modules/uneditable_fields/shipping_address wholesale/pages/purchase_order/modules/product_grid modules/time_utils base modules/deferred_clientlogger modules/jquery modules/async_analytics_init notice/experiment_notice modules/locale/dismissible_link pages/recommendations/friends/v2/modules/gift-ideas-reminder header pages/translator/_modules/md5_overlay modules/breadcrumbs pages/browse/_modules/footer footer Some/Controller/Class.php Other/Controller/Class.php Foo/Controller/Class.php Bar/Controller/Class.php Baz/Controller/Class.php Batman/Controller/Class.php ... Friday, March 7, 14
- 36. @danielespeset $templates/> for ... grep “{% css=$1” ... a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css /base-responsive /base /base_after_inline c/template_editor c/mini a/scout/dashboard wholesale/modules/vendor_avatar wholesale/modules/address_from_array wholesale/pages/purchase_order/modules/uneditable_fields/payment_method wholesale/pages/purchase_order/modules/uneditable_fields/payment_terms wholesale/pages/purchase_order/modules/uneditable_fields/ship_date wholesale/pages/purchase_order/modules/uneditable_fields/billing_address wholesale/pages/purchase_order/modules/uneditable_fields/shipping_address wholesale/pages/purchase_order/modules/product_grid modules/time_utils base modules/deferred_clientlogger modules/jquery modules/async_analytics_init notice/experiment_notice modules/locale/dismissible_link pages/recommendations/friends/v2/modules/gift-ideas-reminder header pages/translator/_modules/md5_overlay modules/breadcrumbs pages/browse/_modules/footer footer Some/Controller/Class.php Other/Controller/Class.php Foo/Controller/Class.php Bar/Controller/Class.php Baz/Controller/Class.php Batman/Controller/Class.php ... htdocs/my_endpoint.php htdocs/foo.php htdocs/other_endpoint.php htdocs/baz/index.php htdocs/index.php Friday, March 7, 14
- 37. @danielespeset $templates/> for ... grep “{% css=$1” ... a/browse.css a/taxeditor.css a/taxonomies.css a/weddings.css b/modules-responsive.css b/modules.css a/orders/refund.css a/scout/scout.css c/account/credit-card.css c/bill/autobilling.css c/bill/index.css c/bill/payments.css c/shop/registration/billing.css /base-responsive /base /base_after_inline c/template_editor c/mini a/scout/dashboard wholesale/modules/vendor_avatar wholesale/modules/address_from_array wholesale/pages/purchase_order/modules/uneditable_fields/payment_method wholesale/pages/purchase_order/modules/uneditable_fields/payment_terms wholesale/pages/purchase_order/modules/uneditable_fields/ship_date wholesale/pages/purchase_order/modules/uneditable_fields/billing_address wholesale/pages/purchase_order/modules/uneditable_fields/shipping_address wholesale/pages/purchase_order/modules/product_grid modules/time_utils base modules/deferred_clientlogger modules/jquery modules/async_analytics_init notice/experiment_notice modules/locale/dismissible_link pages/recommendations/friends/v2/modules/gift-ideas-reminder header pages/translator/_modules/md5_overlay modules/breadcrumbs pages/browse/_modules/footer footer Some/Controller/Class.php Other/Controller/Class.php Foo/Controller/Class.php Bar/Controller/Class.php Baz/Controller/Class.php Batman/Controller/Class.php ... htdocs/my_endpoint.php htdocs/foo.php htdocs/other_endpoint.php htdocs/baz/index.php htdocs/index.php RewriteRule ^(.*)/ve/(.*)$ $1/voluntary-employee/$2 [L,R=301] RewriteRule ^(.*)/hsa/(.*)$ $1/health-saving-account/$2 [L,R=301] RewriteCond %{REQUEST_FILENAME} !-f # Existing File RewriteCond %{REQUEST_FILENAME} !-d # Existing Directory RewriteRule . /index.php [L] RewriteCond %{HTTP_REFERER} !^$ RewriteCond %{HTTP_REFERER} !^http://(subdomain.)?domain.tld/.*$ [NC] RewriteRule ^.*.(bmp|tif|gif|jpg|jpeg|jpe|png)$ - [F] Friday, March 7, 14
- 38. @danielespeset CC Image courtesy of Mark Skipper (bitterjug) on Flickr Halp! Friday, March 7, 14
- 39. @danielespeset Reasoning about our frontend is hard. Friday, March 7, 14
- 40. @danielespeset Deleting code is scary. Friday, March 7, 14
- 41. @danielespeset Testing changes is difficult. Friday, March 7, 14
- 42. @danielespeset Mental overhead is a performance problem. Friday, March 7, 14
- 43. @danielespeset What are our options? • Static Analysis • Synthetic Testing • Access Logs • Dynamic Analysis Friday, March 7, 14
- 44. @danielespeset What are our options? • Static Analysis • Synthetic Testing • Access Logs • Dynamic Analysis Friday, March 7, 14
- 45. @danielespeset What are our options? • Static Analysis • Synthetic Testing • Access Logs • Dynamic Analysis Friday, March 7, 14
- 46. @danielespeset What are our options? • Static Analysis • Synthetic Testing • Access Logs • Dynamic Analysis Friday, March 7, 14
- 47. @danielespeset What are our options? • Static Analysis • Synthetic Testing • Access Logs • Dynamic Analysis Friday, March 7, 14
- 48. @danielespeset What are our options? • Static Analysis • Synthetic Testing • Access Logs • Dynamic Analysis Friday, March 7, 14
- 49. Builda Friday, March 7, 14
- 50. @danielespeset A brief history of JS loading at Etsy • <script> Friday, March 7, 14
- 51. @danielespeset A brief history of JS loading at Etsy • <script> • Etsy.loader Friday, March 7, 14
- 52. @danielespeset A brief history of JS loading at Etsy • <script> • Etsy.loader • //= require Friday, March 7, 14
- 53. @danielespeset Builda Loading JS in development assets/js <script src="foo.js"> builda.php?name=foo.js processing Etsy.loader // require Friday, March 7, 14
- 54. @danielespeset Building JS in deploys bin/builda Builda Friday, March 7, 14
- 55. @danielespeset Builda Building JS in deploys bin/builda assets/js dist/js processing Etsy.loader // require compiled output Friday, March 7, 14
- 56. @danielespeset Builda dist/js Building JS in deploys bin/builda assets/js processing Etsy.loader // require compiled output Friday, March 7, 14
- 57. @danielespeset Builda dist/js Building JS in deploys bin/builda assets/js processing Etsy.loader // require compiled output Friday, March 7, 14
- 58. @danielespeset Builda Building JS in deploys bin/builda assets/js dist/js Minification Friday, March 7, 14
- 59. @danielespeset bin/builda assets/jsBuilda Building JS in deploys dist/js Minification Friday, March 7, 14
- 60. @danielespeset Building JS in deploys bin/builda assets/js dist/js WWW Builda Friday, March 7, 14
- 61. @danielespeset A brief history of JS loading at Etsy • <script> • Etsy.loader • //= require • AMD (requirejs) Friday, March 7, 14
- 63. define(function() { return function Lightsaber(color) { this.color = color; }; }); lightsaber.js Friday, March 7, 14
- 64. define(function() { return function Lightsaber(color) { this.color = color; }; }); lightsaber.js define(['lightsaber'], function(Lightsaber) { return function Luke() { this.lightsaber = new Lightsaber('green'); }; }); luke.js Friday, March 7, 14
- 65. define(function() { return function Lightsaber(color) { this.color = color; }; }); lightsaber.js define(['lightsaber'], function(Lightsaber) { return function Luke() { this.lightsaber = new Lightsaber('green'); }; }); luke.js require(['plot','luke','vader'], function(plot, Luke, Vader) { var luke = new Luke(), vader = new Vader(); plot.exploreFatherIssues(luke, vader); }); starwars.js Friday, March 7, 14
- 66. @danielespeset define(function() { return function Lightsaber(color) { this.color = color; }; }); lightsaber.js define(['lightsaber'], function(Lightsaber) { return function Luke() { this.lightsaber = new Lightsaber('green'); }; }); luke.js require(['plot','luke','vader'], function(plot, Luke, Vader) { var luke = new Luke(), vader = new Vader(); plot.exploreFatherIssues(luke, vader); }); starwars.js <script type="text/javascript" src="starwars.js"></script> Builda HTTP GET starwars.js Async loading AMD in development Friday, March 7, 14
- 67. @danielespeset require(['plot','luke','vader'], function(plot, Luke, Vader) { define(function() { return function Lightsaber(color) { this.color = color; }; }); lightsaber.js define(['lightsaber'], function(Lightsaber) { return function Luke() { this.lightsaber = new Lightsaber('green'); }; }); luke.js Async loading AMD in development Builda uilda HTTP GET luke.js Friday, March 7, 14
- 68. @danielespeset define(function() { return function Lightsaber(color) { this.color = color; }; }); lightsaber.js define(['lightsaber'], function(Lightsaber) { Builda HTTP GET lightsaber.js Async loading AMD in development Friday, March 7, 14
- 69. @danielespeset Async loading AMD in development Friday, March 7, 14
- 70. Incorporating AMD into Builda, phase 1 Friday, March 7, 14
- 71. @danielespeset Building JS with AMD in deploys bin/builda Builda assets/js dist/js r.jsr.jsr.jsr.js Friday, March 7, 14
- 72. define(function() { return function Lightsaber(color) { this.color = color; }; }); lightsaber.js define(['lightsaber'], function(Lightsaber) { return function Luke() { this.lightsaber = new Lightsaber('green'); }; }); luke.js require(['plot','luke','vader'], function(plot, Luke, Vader) { var luke = new Luke(), vader = new Vader(); plot.exploreFatherIssues(luke, vader); }); starwars.js Friday, March 7, 14
- 73. // more modules... starwars.12345678901234.js define('lightsaber', function() { return function Lightsaber(color) { this.color = color; }; }); define('luke', ['lightsaber'], function(Lightsaber) { return function Luke() { this.lightsaber = new Lightsaber('green'); }; }); require(['plot','luke','vader'], function(plot, Luke, Vader) { var luke = new Luke(), vader = new Vader(); plot.exploreFatherIssues(luke, vader); }); // more modules... Friday, March 7, 14
- 74. This broke dev-prod environment parity Friday, March 7, 14
- 75. Also it was crazy slow Friday, March 7, 14
- 76. Builda Revised Friday, March 7, 14
- 77. @danielespeset AKA taking the A out of AMD Friday, March 7, 14
- 78. @danielespeset assets/js dist/js Building JS in development Builda Friday, March 7, 14
- 79. @danielespeset assets/js Save foo.js Building JS in development Buildadist/js inotify IN_CLOSE_WRITE foo.js "foo": { "dependency_of": ["bar", "baz"] } Friday, March 7, 14
- 80. @danielespeset Builda assets/js dist/js AMD (r.js) Etsy.loader // require foo.js Building JS in development Friday, March 7, 14
- 81. @danielespeset Builda assets/js dist/js AMD (r.js) Etsy.loader // require bar.js Building JS in development Friday, March 7, 14
- 82. @danielespeset Builda assets/js dist/js AMD (r.js) Etsy.loader // require baz.js Building JS in development Friday, March 7, 14
- 83. @danielespeset assets/js dist/js Building JS in development Builda HTTP GET foo.js Friday, March 7, 14
- 84. @danielespeset assets/js dist/js Building JS in development Builda Friday, March 7, 14
- 85. Ranger Friday, March 7, 14
- 86. @danielespeset Jason Huff Aaron Moodie @jsnhff @aaronmoodie Friday, March 7, 14
- 98. @danielespeset Ranger is an API Lodge is a client Friday, March 7, 14
- 103. @danielespeset Ranger rollup linter hinter escomplex scp stuff log host WWW Friday, March 7, 14
- 104. @danielespeset Ranger rollup linter hinter escomplex scp stuff log host WWW Friday, March 7, 14
- 105. @danielespeset Now we know enough to surface dead files with confidence Friday, March 7, 14
- 106. @danielespeset Finding dead code paths is still hard Friday, March 7, 14
- 107. Shrinkray Friday, March 7, 14
- 108. @danielespeset Justin Donato Kelly Norton @justindo @kellegous Friday, March 7, 14
- 109. @danielespeset Friday, March 7, 14
- 110. @danielespeset What’s next? • Adding Templates as a resource Friday, March 7, 14
- 111. @danielespeset What’s next? • Adding Templates as a resource • Shrinkray powered delete button Friday, March 7, 14
- 112. @danielespeset What’s next? • Adding Templates as a resource • Shrinkray powered delete button • Detecting inefficiently grouped code Friday, March 7, 14
- 113. @danielespeset What’s next? • Adding Templates as a resource • Shrinkray powered delete button • Detecting inefficiently grouped code • Feature / User-state asset signatures Friday, March 7, 14
- 114. @danielespeset What’s next? • Adding Templates as a resource • Shrinkray powered delete button • Detecting inefficiently grouped code • Feature / User-state asset signatures • Shrinkray for JavaScript Friday, March 7, 14
- 115. @danielespeset The frontend is getting more complex everywhere. Friday, March 7, 14
- 116. @danielespeset We need new strategies to manage and understand it. Friday, March 7, 14
- 117. @danielespeset The browser is part of your distributed system, not just a client you support. Friday, March 7, 14
- 118. @danielespeset The more extensible a tool the more useful it will be. Friday, March 7, 14
- 120. Friday, March 7, 14
- 121. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/javascript -compiler-deploy