Reproducible builds, fast and safe deployment process together with self-healing services form the basis of stable and maintainable infrastructure. In this talk I’d like to cover, from the Site Reliability Engineering (SRE) perspective, how Dropbox addresses above challenges, what technologies are used and what lessons were learnt during implementation process.

2. Building,
deploying and
running
production code
at Dropbox
Leonid Vasilyev, SRE at Dropbox. United Dev Conf ’17

3. •Intro & Background
•Building Code
•Deploying Packages
•Running Services
•Recap & Conclusion

4. Intro &
Background

6. Dropbox Backend Infrastructure:
Something one might call a “Hybrid Cloud”.
Few datacenters + AWS VPCs + Edge Network (POPs).
Running Ubuntu Server, Puppet/Chef and Nagios.
Rest of the stack is pretty custom.
Dropbox today is not just “file storage”,
but dozens of services,
running on tens of thousands of machines.

8. I. Building Code

9. Early days: few code repos, mostly Python.
No build system.
Period.

10. Why Bother?

11. Ruby Python
Java
C/C++
Node.js
Rust
Go
PHP
Exhibit 1: Runtimes & Packages (ಠ_ಠ)

12. Problems:
Repo is growing, new languages are in use:
Golang, Node.js, Rust.
No way to track dependencies,
dependencies installed in runtime via Puppet.
Global Encap repo deployed via rsync onto the whole fleet.

13. In search of a better build system
What are the requirements?
• Fast
• Reproducible
• Hermetic
• Flexible
• Explicit dependencies

15. A Historical Perspective*
•(2006) Google got annoyed with Make and began “Blaze”
•(2012) Looks like ex-googlers at Twitter were missing “Blaze”, hence
began “Pants”
•(2013) Looks like ex-googlers at Facebook were missing “Blaze”,
hence began “Buck”
•(2014) Google realised what’s going on and released “Blaze” as
“Bazel”
•(2016) Ex-googlers at Thought Machine are still missing “Blaze”,
hence began “Please”, in Go this time :)

16. Bazel Concepts
•WORKSPACE: one per repo, defines external
dependencies
•BUILD files: Python-like DSL for describing build
targets (test is also a build target)
•`*.bzl` files: Macro and extensions
•`//dropbox/aws:ec2allocate` — labels to
specify build targets

17. native.new_http_archive(
name = "six_archive",
urls = [
“http://pypi.python.org/.../six-1.10.0.tar.gz”,
],
sha256 = “…”,
strip_prefix = "six-1.10.0",
build_file = str(Label("//third_party:six.BUILD")),
)
External Dependencies(1)

18. py_library(
name = "six",
srcs = ["six.py"],
visibility = ["//visibility:public"],
)
External Dependencies(2)

19. py_library(
name = "platform_benchmark",
srcs = ["platform/benchmark.py"],
deps = [
":client",
":platform",
"@six_archive//:six",
],
)
External Dependencies(3)

20. Bazel adoption at Dropbox

21. •Migration started in July, 2015
•~6,400 Bazel BUILD files (~314,094 lines)
•~9,000 lines of custom *.bzl code
•Custom rules for: python, golang, rust, node.js
•BUILD file generator for Cmake, Python
•Mostly done, still work in progress …
Migration Status

22. Key Insights
•Robust remote build cache is essential.
•Keep explicit dependencies between
components.
•It is possible to retrofit new build system into
old codebase.
•Bazel, Pants, Buck, Please — pick one, or write
your own :)

23. II. Deploying Packages

24. Deployment System: YAPS

25. Service Configuration: Gestalt
Pystachio is used to specify the following schema:

26. class Project(Struct):
name = Required(String)
owner = Required(String)
deployments = Required(List(Deployment))
class Deployment(Struct):
name = Required(String)
build = Required(Build)
kick = Required(Kick)
dependencies = List(Dependency)
Service Configuration: Gestalt

27. class Build(Struct):
name = Required(String)
bazel_targets = Required(List(String))
timeout_s = Default(Integer, 3600)
class Kick(Struct):
name = Required(String)
package_dir = Required(String)
dbxinit = Required(Program)
host_kick_ration = Default(Float, 0.25)
Service Configuration: Gestalt

28. class Program(Struct):
name = Required(String)
num_procs = Default(Integer, 1)
env = Map(String, String)
cmd = Required(List(String))
limits = Limits # rss, max_fd, nice
logfile = Default(String, DEFAULT_LOG_DIR)
root_fs = String # docker os image
health_check = HealthCheck
Service Configuration: Gestalt

29. •About 500 files and 60,000 SLOC
•Complex evaluation rules
•Configuration tends to become a Turing-complete
language
•Advanced linters and validation needed
•Specifying resource limits is tricky
Gestalt: Challenges

30. YAPS Packages

31. YAPS Packages: Historical
approach
•Install Debian packages via Puppet/Chef
•Use Python’s Virtualenv & PyPI
•Encap — “Bag of rats” dependencies :)
•Blast the whole repo via rsync every few minutes
by CRON

32. YAPS Packages: Current approach
•SquashFS images. Native Linux in-kernel support
•Transparent compression and de-duplication
•Read-only mounts, +1 from security
•Loopback device mounts are fast
•SquashFS image has 1+ Bazel targets and
transitive dependency closure for each target

33. $ cd /srv/aws-tools
$ tree -L 3
.
|-- ec2terminate # <- executable file
`-- ec2terminate.runfiles # <- transitive closure
|-- MANIFEST # <- list of all files
`-- __main__ # <- dependencies
|-- _solib_k8
|-- configs
|-- dbops
|-- devsecrets
|-- dpkg
`-- dropbox
...

34. YAPS Packages: Challenges
•*.pyc files have to be in the package
•Unmountable packages due to open file descriptors
•If code has to be modified on the prod server
(YOLO), special procedure — “Hijacking” is
required
•Full package has be pushed even with a 1 line
change (Xdelta compression might help)

35. III. Running Services

36. Process Manager: Historical
approach
•Using Supervisord and configuration generated by
Puppet
•Update of Supervisord requires tasks to be restarted
•Loosing tasks if Supervisord killed by OOM
•Supervisord is really old, from 2004 (has
XMLRPC?!)

37. Process Manager: Current
approach
•Using Dbxinit: in-house project written in Go
•Keeps local state, thus can be updated without tasks
downtime, can survive OOM
•Supports health-checks for tasks
•Has resource limits: RSS, max fds, OOM score
•Speaks JSON HTTP

38. Configuration Management:
Historical approach
•Puppet 2.x in server mode
•Perf problems with server as fleet grew in size
•No linters or unit tests, caused a lot of errors
•“Blast to the fleet” deployment model
•Single global run via CRON, runs all modules - slow

39. Configuration Management:
Current approach
•Chef 12.x in Zero mode
•Invested heavily into linters and unit-testing
•Easy to test on a single production machine
•Has 3 runs: “platform”, “global” and “service”
•Cookbooks deployed via YAPS
•Generally trying to move service owners out of CM

40. It wouldn’t be 2017 if not … containers!

41. Containers: Runc for stateless
services
•Runc is integrated with Dbxinit, each task runs inside
its own container
•Runc uses minimal Ubuntu Docker image
•Main use case is dependency isolation via mount
namespaces
•Doesn’t use network namespaces yet

42. Containers: Challenges
•Log rotation. Logs should be moved from the box
ASAP, since machine with stateless service can be
shut down without notice
•Looking into ELK stack to solve that problem
•Resource accounting. Currently doesn’t enforce any
resource limits

43. VS
Human Automation

44. VS
Human Automation
BRAINS!
Attitude

45. Ops Automation: Nagios &
Naoru
•Nagios runs on all production machines & AWS
EC2 instances
•Common problems are automatically fixed by auto-
remediation system called “Naoru”
•Its input is a stream of Nagios alerts and output is a
set of remediations that can be executed
automatically

47. Talk Summary
Building:
•Unified build system with clean dependencies
Deploying:
•One deployment system and sound packaging
Running:
•Robust process management and automation of
simple tasks

48. Thank You!
{twttr: @leo_vsl}