At Etsy we are continuously deploying software with about 50 deploys per day on average. We have about 150 engineers that work together to add features, improve the site, solve problems, and figure out outages. We are constantly working on improving ways to collaborate and have the time to invest into research and new projects instead of fire fighting. However this hasn't been like this from the beginning. In the Dark Ages of Etsy we had a vastly different software architecture, a myriad of silos with a less than ideal amount of communication and a not so stable site. So what happened? How did we fix this? We will briefly go over how we changed the architecture and culture to make site operations and stability better. After that we will go into detail about what it meant for us to maintain that culture of collaboration and trust and things we learned from that. And finally take a look at the state of things in the present and give a comprehensive picture of how we arrived here and where to go from there.

1. DEPLOY
COLLABORATE & LISTEN
Daniel Schauenberg
d@etsy.com
@mrtazz

2. ETSY

3. THE MONOLITH

4. LAMP

6. HOW COMFORTABLE
ARE YOU
DEPLOYING A CHANGE
RIGHT NOW

7. MTTR
TRUMPS
MTBF

8. IF THIS IS YOUR
FIRST DAY AT ETSY
YOU DEPLOY THE SITE

10. THE DARK PAST

11. ❗

12. - HINDSIGHT IS 20/20
- THERE WOULD BE NO ETSY
- I WASN'T AROUND FOR THIS
(the grain of salt disclaimer)

13. DARK
LESS FUN

14. ARCHITECTURE OVERVIEW
- UBUNTU
- POSTGRESQL
- LIGHTTPD
- PHP/PYTHON

15. SINGLE BIG
DATABASE

16. BUSINESS LOGIC
IN STORED
PROCEDURES

18. SILOS

19. DEV ! DBA ! OPS

20. SPROUTER

21. A SOFTWARE
MANIFESTATION
OF SILOS

22. SITE UPTIME
WASN'T GOOD

23. MORE UPTIME > LESS UPTIME

25. Trust
THE
PEOPLE

27. SPROUTER

28. NO MORE
MAINFRAME
MASTER

29. HORIZONTAL
SCALING (A SINGLE BOX ONLY SCALES SO FAR)

30. MASTER-MASTER
REPLICATED
MYSQL SHARDS

31. FLICKR DNA

32. DEPLOYINATOR

33. DEPLOY
!=
RELEASE

34. CONFIG FLAGS

35. if Feature::isEnabled($feature) {
// new hawtness
} else {
// nothing to see here
}

36. METRICS!

38. STATSD "HOW DO YOU KNOW THIS WORKS IN
PRODUCTION?"

39. CHEF

40. <3 CHEF

41. DEVELOPER VMS - KVM
- EVERY ENGINEER HAS ONE
- FULLY CHEF’D WITH THE ETSY STACK
- DIFFERENT SIZES AND CHEF ROLES

42. Try

43. BLAMELESS
POST MORTEMS

46. OVERHAULS
&
ITERATIONS

47. CULTURE & TOOLS

48. HUMANS ARE
AWESOME

49. NOBODY COMES
TO WORK TO DO A
BAD JOB

50. Trust YOUR CO-WORKERS

51. THERE IS A LOT OF KNOWLEDGE IN YOUR
ENGINEERING TEAM

52. DEPLOY (AS OFTEN AS IT MAKES SENSE)

53. COLLABORATE (EVEN IF YOU THINK YOU DON'T HAVE TO)

55. LISTEN (TO PROBLEMS AND EXPERIENCES OF YOUR COWORKERS)

56. CODEASCRAFT.COM
ETSY.COM/CODEASCRAFT/TALKS
ETSY.COM/CAREERS

57. THANK YOU!

58. DEPLOY
COLLABORATE & LISTEN
Daniel Schauenberg
d@etsy.com
@mrtazz