a bit.ly tech talk on writing code for a production environment

1. Coding For
“Production”
a bit.ly tech talk 9/29/10
@jehiah

2. • Who am I writing code for?
• What is a sandbox?
• Do you use internal APIs?
• Are exceptions cool?
• Who handles logging?
• How do you design for scale?

3. Who am I writing code for?
A: Everyone Else!
• anyone should be able to jump in and run/
change/fix any piece of your architecture
• make startup easy (even if your app is
useless without data, run without it)
• write simple code (don’t use fancy or
advanced programing features)

4. Who am I writing code for?
• script everything you do; it helps others
learn how to use/manage your code.
• use system wide resources to make
bootstrapping your code the same as
everything else. Avoid making your app
“special”
• don't re-invent start/stop scripts, run paths,
logging, monitoring, data collection,
deployment, upgrades

5. Sandbox
• You should always have a sandbox
environment where your application ONLY
talks to other sandbox’d applications
• Your application should never know which
of the two it’s in (only your settings file
should know this)
• Sandboxes let you break things, throw away
data, and start over with no penalty or risk.

6. Test New
Data Code
New Code
Sandbox Data Production
Data

7. How do you keep your code from knowing
where it is running?
fetch('http://backend.server.com/api?' +
urllib.urlencode(...))
fetch(settings.get('metrics_backend’) + '/api?' +
urllib.urlencode(...))

8. http://gist.github.com/595574
import tornado.options
tornado.options.define("environment", default="dev", help="environment")
options = {
"dev" : {
"metrics_api" : "http://127.0.0.1:8080",
},
"prod" : {
"metrics_api" : "http://remote-server1.com",
}
}
# settings that are the same for each environment
# (ie: they are here just to make them easy to find/change/update)
default_options = {
"num_results" : 15,
}
def get(key):
env = tornado.options.options.environment
if env not in options:
raise Exception("Invalid Environment (%s)" % env)
v = options.get(env).get(key) or default_options.get(key)
return v

9. Exceptions are for
Developers
• exceptions help you find what to do next.
• fail hard and fail fast.
• assert your expectations
• when you find a problem with your code,
always ask what error would have helped
me find this sooner, and then add that
error

10. Exceptions are for
Developers
• if you accidentally pass a string instead of an integer, add
> assert isinstance(var, int)
• scripts that take files as args should test that they exist
> assert os.path.exists(filename)
• don't catch exceptions. they are good! if you do catch
an exception logging.exception() so it doesn't get lost.

11. Be Lazy
Expect something else to handle:
• Logging (and log collection/rotation)
• Running Monitoring Checks (and alerting)
• start/stop scripts
• deploy scripts
• installing dependency libraries
• state

12. Logging
• application logging is for errors and
contexts not for auditing
• don’t get confused: sometimes we call raw
data “logs”. For example: access logs. these
are raw data files not logs.
• use logging module for timestamps, stack
traces (context). don’t use print.

13. Designing for scale:
• Assume 100x of your app/script might be
running simultaneously
• Always single thread
• Use stateless connections (they give you
hooks to scale) http->nginx->load balancer
• Client knows what the data is, the remote
end treats it like a blob or key/value store
• Be as generic as possible and as specific as
possible, never in between.