O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Deploying on the cutting edge

Talk on how we do deployment at Urban Airship from Djangocon 2011.

  • Seja o primeiro a comentar

Deploying on the cutting edge

  1. 1. Safely Deploying on the cutting edge Eric Holscher Urban Airship Djangocon 2011Wednesday, September 7, 2011
  2. 2. Wednesday, September 7, 2011
  3. 3. Talk Contents • Company culture & process • Deployment environment • Tools for deploying • Verifying deploymentWednesday, September 7, 2011
  4. 4. ProcessWednesday, September 7, 2011
  5. 5. Process • Deploy out of git • Standard git-based production/master branch model • Production branch has releases are tagged with timestamp • deploy-2011-08-03_14-37-38 • Feature branches • http://nvie.com/posts/a-successful-git-branching-model/Wednesday, September 7, 2011
  6. 6. Features • Easily allows you to hot-fix production • Keep a stable master • Run CI on the master branch or long-lived feature branchesWednesday, September 7, 2011
  7. 7. Services • Everything that we deploy is conceptualized as a service • Services all live in /mnt/services/<slug> (Thanks ec2) • A service is an instance of a repository on a machine • A repository might have multiple services • eg. Airship deployed into “celery” and “web” services • This maps really well onto Chef cookbooksWednesday, September 7, 2011
  8. 8. QA Environment • Run all of your master branches • Allow you to get a copy of what will become production • Catch errors before they are seen by customers • Spawn new ones for long-lived feature branches • `host web-0` and figure out based on IPWednesday, September 7, 2011
  9. 9. Deployment Design GoalsWednesday, September 7, 2011
  10. 10. Jump machine • Have a standard place for all deployments to happen • Log all commands runWednesday, September 7, 2011
  11. 11. No External Services • Chishop • No external server required to deploy code • All branches are checked out on an admin serverWednesday, September 7, 2011
  12. 12. Services look the same • Python • Java • “Unix”Wednesday, September 7, 2011
  13. 13. Composable • Small pieces that you can build into better things • Useful when trying to do something you didn’t plan forWednesday, September 7, 2011
  14. 14. EnvironmentWednesday, September 7, 2011
  15. 15. Environment • Where code lands on the remote machine • Mimics a chroot • Uses virtualenv & supervisord • Owned by the service-user • Managed by ChefWednesday, September 7, 2011
  16. 16. File Structure • /mnt/services/airship • bin/ • current -> deploy-2011-08-03_14-37-38 • deploy-2011-08-03_14-37-38 • etc/ • var/Wednesday, September 7, 2011
  17. 17. etc/ • supervisord.conf • [include] • files = *.conf • airship.confWednesday, September 7, 2011
  18. 18. bin/ • start • stop • restart • logsWednesday, September 7, 2011
  19. 19. SCRIPT_DIR=$(dirname $0) SERVICE_DIR=$(cd $SCRIPT_DIR && cd ../ && pwd) cd $SERVICE_DIR supervisorctl pid > /dev/null 2>&1 if [ "$?" != "0" ]; then echo "Supervisord not running, starting." supervisord else echo "Supervisord running, starting all processes." supervisorctl start all fi cd - > /dev/null 2>&1Wednesday, September 7, 2011
  20. 20. Bin scripts • All of the process-level binscripts wrap supervisord • bin/start -> supervisordctl start all • bin/start foo -> supervisorctl start foo • bin/stop -> supervisorctl stop all • bin/stop shutdown -> supervisorctl shutdownWednesday, September 7, 2011
  21. 21. var/ • data/ • log/ • run/ • tmp/Wednesday, September 7, 2011
  22. 22. Init.d • All services share a common init.d script • This init.d script calls into the service’s bin/ • /etc/init.d/airship start -> /mnt/services/airship/bin/startWednesday, September 7, 2011
  23. 23. SERVICE_USER=<%= @service %> SERVICE_NAME=<%= @service %> SERVICE_PATH=/mnt/services/$SERVICE_NAME set -e RET_CODE=0 case "$1" in start) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/start RET_CODE=$? ;; stop) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/stop RET_CODE=$? ;; restart) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/restart RET_CODE=$? ;; status) sudo su - $SERVICE_USER -c $SERVICE_PATH/bin/status RET_CODE=$? ;; *) echo "$SERVICE_NAME service usage: $0 {start|stop|restart|status}" ;; esac exit $RET_CODEWednesday, September 7, 2011
  24. 24. ToolsWednesday, September 7, 2011
  25. 25. Tools • Fabric • Rsync • Pip • VirtualenvWednesday, September 7, 2011
  26. 26. Low-level verbs • pull • build • tag • sync • install • rollback • start/stop/restart/reloadWednesday, September 7, 2011
  27. 27. Pull • Update the code from the source repository • Defaults to the “production” branch • def pull(repo=None, ref=origin/production) • Can pass in a specific revision/branch/tag/hashish • local(git reset --hard %s % ref, capture=False)Wednesday, September 7, 2011
  28. 28. Build • Could be called “prepare” • Do local-specific things to get repo into a ready state • Mostly used for compiling in java-land • Useful in Python for running pre-install tasksWednesday, September 7, 2011
  29. 29. Tag • Set a tag for the deploy in the git repo • If the current commit already has a tag, use that instead • git tag --contains HEAD • deploy-2011-08-03_14-37-38 • strftime(%Y-%m-%d_%H-%M-%S)Wednesday, September 7, 2011
  30. 30. Sync • Move the code from the local to the remote box • Uses rsync to put it into the remote service directory • Also places a copy of the synced code on the admin boxWednesday, September 7, 2011
  31. 31. Install • Make the code the active path for code on the machine • This is generally installing code into a virtualenv • Updating the “current” symlink in the service directory • Symlink Django settings file based on environmentWednesday, September 7, 2011
  32. 32. Rollback • When you break things, you need to undo quickly • Reset the repository to the previous deployed tag • git tag | grep deploy| sort -nr |head -2 |tail -1 • Deploy that • Very few moving piecesWednesday, September 7, 2011
  33. 33. Start/Stop/Reload • Allow you to bounce services as part of deployment • Allow reload for services that support itWednesday, September 7, 2011
  34. 34. CLI UI • Have nice wrapper commands that do common tasks • deploy host:web-0 full_deploy:airship ➡ pull, build, tag, sync, install • deploy host:web-1 deploy:airship ➡ tag, sync, install • deploy host:web-2 sync:airship ➡ syncWednesday, September 7, 2011
  35. 35. UI cont. • deploy host:web-0 full_deploy:airship restart:airshipWednesday, September 7, 2011
  36. 36. #!/bin/bash cd ~/airdeploy DATE=$(date +%Y_%-m_%-d-%H-%m-%s) echo "deploy" $@ > logs/$DATE.log fab $@ cd - > /dev/null 2>&1Wednesday, September 7, 2011
  37. 37. Meta-commands • Hard-code the correct deployment behavior • “Make easy things easy, and wrong things hard” • Knows what machine each service is deployed to • deploy airship ➡ deploy pull:airship ➡ deploy type:web deploy:airshipWednesday, September 7, 2011
  38. 38. MagicifyingWednesday, September 7, 2011
  39. 39. Magicifying • Now that we have a solid base, we can automate on top • When you do a meta deploy, it should be a “smart deploy”Wednesday, September 7, 2011
  40. 40. Workflow • Deploy to one web server, preferably with one worker • Restart it • Run it against heuristics to determine if it’s broken • If it’s broken, rollback, otherwise continue onWednesday, September 7, 2011
  41. 41. Heuristics • Any 500s • Number of 200s to non-200s • Number of 500s to 200s • Requests a second • Response time • $$$ (Business metrics)Wednesday, September 7, 2011
  42. 42. How it works • Tell load balancer to take machine out of pool • /take_me_out_of_the_lb -> 200 • Start your code with 1 worker and a different port • supervisorctl start canary • Expose metrics from your services over json • Make sure your load balancer weights it appropriately • Poll your metrics for X time before considering it functionalWednesday, September 7, 2011
  43. 43. Thanks • Alex Kritikos • Erik Onnen • SchmichaelWednesday, September 7, 2011
  44. 44. Questions? • Eric Holscher • Urban Airship (Hiring and whatnot) • eric@ericholscher.comWednesday, September 7, 2011

×