Enabling Hybrid Workflows with Docker/Mesos @Orbitz
•
11 likes•3,151 views
This document discusses using Mesos and Docker together at Orbitz to enable hybrid workflows. It describes three main use cases: 1) Using Mesos and Docker to run a microservices platform with multiple environments and datacenters, 2) Using Mesos and Docker slaves to build a Jenkins build farm, and 3) Using specialized Docker slaves to deploy apps from Jenkins to Marathon and other platforms. It provides examples of Dockerfiles used to create specialized slaves for building, deploying to AWS, and other tasks.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Enabling Hybrid Workflows with Docker/Mesos @Orbitz
Similar to Enabling Hybrid Workflows with Docker/Mesos @Orbitz (20)
More from Steve Hoffman
More from Steve Hoffman (6)
Recently uploaded
Recently uploaded (20)
Enabling Hybrid Workflows with Docker/Mesos @Orbitz
- 1. #mesoscon 2015 Steve Hoffman Senior Principal Engineer @bacoboy Enabling Hybrid Workflows with Docker/Mesos @Orbitz
- 3. #mesoscon 2015 • Multiple Brands • Websites • Webservices • Multiple Backends • 500+ apps / thousands of instances • Deployments Daily (sometimes more) If you haven’t heard of us…
- 4. #mesoscon 2015 Mesos as… • … a Microservices Platform using Docker • … a Jenkins Build Farm • … a Jenkins Deployment Farm • HA Notes • CentOS Notes • Questions
- 5. #mesoscon 2015 Case 1: Docker Microservices Platform • Launch Docker apps in • Multiple environments (dev -> qa -> staging -> production) • Multiple datacenters • Update Docker apps in rolling fashion • Restart anything that needs it
- 11. #mesoscon 2015 - tasks: marathon: … PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … } 1.2.16 1.2.16 1.2.16 Deploy
- 12. #mesoscon 2015 Deploy PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … } - tasks: marathon: … 1.2.16 1.2.16 1.2.16 app = GET /v2/apps/editorial-module if not app then deploy_id = POST /v2/apps { “image”: “orbitz/editorial-module:1.2.17”, “id”: “editorial-module” } else deploy_id = PUT /v2/apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” } end if while GET /v2/deployments contains deploy_id // still deploying end // deploy complete
- 13. #mesoscon 2015 - tasks: marathon: … PUT /apps/editorial-module { “image”: “orbitz/editorial-module:1.2.17” … } 1.2.16 1.2.16 1.2.16 Deploy
- 17. #mesoscon 2015 - tasks: marathon: … /health 200 OK 200 OK 200 OK 1.2.16 1.2.16 1.2.16 1.2.17 1.2.17 1.2.17 Deploy
- 19. #mesoscon 2015 And off to the next environment… 1.2.17 1.2.17 1.2.17 Deploy
- 27. #mesoscon 2015 Case 2: The Build Farm • Existing Solution • Dedicated Jenkins Slaves • Hand created • Snapshotted & Rolled Back to “Clean” state after each Job • Hard to Manage Build Environment for 300+ apps across many OSes, Java versions, Ruby versions, perl versions, python versions, protocol buffer compiler versions, etc…
- 29. #mesoscon 2015 Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Before
- 30. #mesoscon 2015 Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Before
- 31. #mesoscon 2015 Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Clone & Build Push Artifacts Before
- 35. #mesoscon 2015 Master Master Slave Poll or Push Trigger Commit/Push or Pull Request/Merge After
- 36. #mesoscon 2015 Master Master Slave Slave Poll or Push Trigger Commit/Push or Pull Request/Merge After
- 37. #mesoscon 2015 Master Master Slave Slave Poll or Push Trigger Commit/Push or Pull Request/Merge After
- 38. #mesoscon 2015 Master Master Slave Slave Poll or Push Trigger Commit/Push or Pull Request/Merge Clone & Build Push Artifacts After
- 41. #mesoscon 2015 Plugin Configuration Don’t forget to install libmesos on Jenkins master! Point at ZK to find active master Leave framework running
- 42. #mesoscon 2015 In Mesos Console… Framework Registered
- 43. #mesoscon 2015 Plugin Configuration Per Slave Type Jenkins Slave Tag Which Mesos Servers this job can use Jenkins Slave Image 1 job per docker slave short timeout for single use RAM/CPU needed
- 44. #mesoscon 2015 Job Configuration Run on specific Docker slave Mark offline immediately
- 46. #mesoscon 2015 Success! Ephemeral Docker Slave in Mesos its tag Docker IP
- 47. #mesoscon 2015 Case 2+: The Docker Build Farm • I ALSO need to build my jenkins/docker slaves • Need a jenkins docker slave to build docker images… • Can I do Docker-in-Docker-on-Mesos? • Will need to build manually the first time http://kb.sparknearby.com/wp-content/uploads/2015/05/chicken-or-egg-cropped1.jpg
- 48. #mesoscon 2015 The docker-builder • Jenkins Slave (Java - jre) • Docker daemon (running & in supervisor mode!) • Registry credentials provided to slave via Credentials Binding Plugin from Jenkins managed security • https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Binding +Plugin • Reuse docker layers (aka share /var/lib/docker)
- 49. #mesoscon 2015 wrapdocker script • Start Docker daemon and then start jenkins slave • https://github.com/jpetazzo/dind/blob/master/wrapdocker • I had to change (variable substitution wasn’t working): [[ $1 ]] && exec "$@" to: [[ $1 ]] && eval exec $@ https://blog.docker.com/2013/09/docker-can-now-run-within-docker/
- 50. #mesoscon 2015 docker-builder Dockerfile FROM docker.orbitz.net/centos:7 MAINTAINER Steve Hoffman <steve.hoffman@orbitz.com> # Need to override default YUM repos # and DNS resolution RUN rm /etc/yum.repos.d/* ADD src/repos/*.repo /etc/yum.repos.d/ ADD src/dns/resolv.conf /etc/ RUN # load repo metadata from above yum clean all && yum makecache && # install packages (jenkins needs at least # java and git) yum install -y jre1.8.0_51 git docker-engine && # update everything not already newer yum update -y # For git to work in bridged mode, we need to setup user identity ADD src/git/gitconfig /root/.gitconfig # Include helper script ADD src/wrapdocker /usr/local/bin/wrapdocker # Mount docker daemon storage VOLUME /var/lib/docker
- 51. #mesoscon 2015 Plugin Configuration Additions Privileged Mode Docker Builder Image Wrapper script to start Docker daemon then run Jenkins slave Shared Docker layers for reuse Additional Docker daemon options
- 52. #mesoscon 2015 Job Configuration Bind Docker registry credentials to ENV variable Copy to user’s ~/.dockercfg Cleanup! Run on docker-builder Build
- 53. #mesoscon 2015 For Example: The go-builder • Jenkins Slave • Compile GO program • Package as Docker app
- 54. #mesoscon 2015 go-builder Dockerfile FROM docker.orbitz.net/docker-builder:latest MAINTAINER Steve Hoffman <steve.hoffman@orbitz.com> RUN mkdir -p go/{bin,pkg,src} ENV GOPATH /go RUN # install packages yum install -y golang && # update everything not already newer yum update -y && # remove local caching repos yum clean all
- 55. #mesoscon 2015 Case 3: The Deploy Farm • Create single purpose client images with tools baked in • Run corresponding Jenkins work against that slave • Not just talking to Marathon… Talk to anything…
- 56. #mesoscon 2015 The marathon-deployer • Launch/Upgrade/Downgrade Docker apps via marathon • use marathon python module library • https://github.com/thefactory/marathon-python • inside Ansible playbook • http://www.ansible.com
- 57. #mesoscon 2015 marathon-deployer Dockerfile (template) FROM docker.orbitz.net/centos:7 MAINTAINER Steve Hoffman <steve.hoffman@orbitz.com> # Need to override default YUM repos and DNS resolution RUN rm /etc/yum.repos.d/* ADD src/repos/*.repo /etc/yum.repos.d/ ADD src/dns/resolv.conf /etc/ RUN # Need java to run a jenkins slave and git yum install -y jre1.8.0_31 git # For git to work in bridged mode, we need to setup user identity ADD src/git/gitconfig /root/.gitconfig # extra RPMs stored in git RUN mkdir /tmp/packages ADD src/packages/*.rpm /tmp/packages/ RUN yum clean all && yum makecache && yum install -y ansible pythons && rpm -Uvh /tmp/packages/*.rpm RUN pip install{{ with .HTTPS_PROXY }} -- proxy={{ . }}{{ end }} -v marathon && yum install -y python-boto python-requests python-crypto && # update everything not already newer & clean yum update -y && yum clean all && rm -rf /tmp/packages
- 58. #mesoscon 2015 marathon-deployer Jenkins Job Run Template Engine To Apply ENV to Dockerfile
- 59. #mesoscon 2015 Case 2++/3+: AWS Builder/Deployer • Build AMIs using Packer in AWS using amazon-ebs provider • Build with Jenkins from source in Git with Packer shell provisioner • Also perform scaling & rolling upgrades via aws-cli • Needed AWS capable Jenkins Slave…
- 60. #mesoscon 2015 The aws-monkey • Jenkins Slave (jenkins user + java) • AWS CLI, Ansible & Packer (pre-installed) • http://aws.amazon.com/cli/ • http://www.ansible.com/ • http://packer.io • AWS credentials provided to slave via Credentials binding plugin from Jenkins managed security • https://wiki.jenkins-ci.org/display/JENKINS/Credentials+Binding+Plugin http://images.spatiallyadjusted.com/GeoMonkey-AWS.jpg
- 61. #mesoscon 2015 aws-monkey Dockerfile FROM docker.orbitz.net/centos:7 MAINTAINER Steve Hoffman <steve.hoffman@orbitz.com> # Need to override default YUM repos and DNS resolution RUN rm /etc/yum.repos.d/* ADD src/repos/*.repo /etc/yum.repos.d/ ADD src/dns/resolv.conf /etc/ RUN # install java and git yum clean all && yum makecache && yum install -y jre1.8.0_31 git # For git to work in bridged mode, we need # to setup user identity ADD src/git/gitconfig /root/.gitconfig RUN # install ansible, packer and other utils yum install -y ansible python-boto python- requests packer unzip tar pytz python-dateutil && # install aws-cli (no RPMs - but gets latest) curl "https://s3.amazonaws.com/aws-cli/awscli- bundle.zip" -o "awscli-bundle.zip" && unzip awscli-bundle.zip && ./awscli-bundle/install -i /usr/local/aws -b / usr/local/bin/aws && rm -rf ./awscli-bundle awscli-bundle.zip && # update everything not already newer yum update -y ENV LANG en_US.UTF-8 ENV LC_ALL en_US.UTF-8
- 62. #mesoscon 2015 Jenkins Slave Tag Which Mesos Servers this job can use The Jenkins Environment 1 job per docker slave Plugin Configuration
- 63. #mesoscon 2015 Run on specific Docker slave Pass AWS Credentials as ENV vars Job Configuration
- 65. #mesoscon 2015 HA Mesos • Run Multiple Masters so no SPOF • Run Multiple Mesos Clusters for DR/HA • Per Datacenter • Per Environment (Production/Pre-Production)
- 66. #mesoscon 2015 sum(mesos-master) = • Zookeeper (for election and state) • Mesos Master (1 elected leader) • Marathon (1 leader, but all serve UI) • Chronos (1 leader, but all serve UI) • Web Server for Authentication for Marathon and Chronos UIs Master ZK Marathon Chronos httpd
- 70. #mesoscon 2015 Zookeeper Configuration $ cat /etc/zookeeper/zoo.cfg server.1=master-1.foo.com:2888:3888 server.2=master-2.foo.com:2888:3888 server.3=master-3.foo.com:2888:3888 $ cat /var/lib/zookeeper/data/myid 1 (on master-1, 2 on master-2, 3 on master-3) $ echo "stat" | nc localhost 2181 | grep Mode Mode: leader|follower (1 leader, 2 followers)
- 71. #mesoscon 2015 Mesos Master Configuration (mesosphere rpms) $ echo "zk://master-1.foo.com:2181,master-2.foo.com: 2181,master-3.foo.com:2181/mesos" > /etc/mesos/zk $ hostname > /etc/mesos-master/hostname $ hostname -I > /etc/mesos-master/ip $ echo "WARNING" > /etc/mesos-master/logging_level $ echo "2" > /etc/mesos-master/quorum
- 72. #mesoscon 2015 Mesos Slave Configuration (mesosphere rpms) $ echo "zk://master-1.foo.com:2181,master-2.foo.com: 2181,master-3.foo.com:2181/mesos" > /etc/mesos/zk $ hostname > /etc/mesos-slave/hostname $ hostname -I > /etc/mesos-slave/ip $ echo "WARNING" > /etc/mesos-slave/logging_level $ echo "5mins" > /etc/mesos-slave/executor_registration_timeout $ echo "10secs" > /etc/mesos-slave/docker_stop_timeout $ echo "docker,mesos" > /etc/mesos-slave/containerizers $ cat /etc/mesos-slave/attributes service_cluster:jenkins chef_environment:production
- 74. #mesoscon 2015 CentOS Notes • Setup docker bridge yourself and use --bridge flag • http://backreference.org/2010/07/28/linux-bridge-mac-addresses-and-dynamic- ports/ • OUR most stable/performant config for Jenkins docker nodes seems to be: • CentOS 7.1 • kernel 4.1.2 (to get overlayfs) • --storage-driver=overlayfs (on ext4 /var/lib/docker partition) • BUT…Beware https://bugzilla.redhat.com/show_bug.cgi?id=1213602 for DinD Yum operations (will be fixed in CentOS 7.2). Workaround in bug report
- 75. #mesoscon 2015 Summary • Jenkins + Mesos + Marathon = Kick Ass Docker Platform • Can do multiple Jenkins in 1 Mesos Cluster • Can do multiple Mesos clusters with 1 Jenkins • Works with RHEL/CentOS! (also Ubuntu — so I’m told) • Separate Mesos clusters if it makes sense • Make special purpose Mesos Slaves if it makes sense • Mesos Slaves in VMs are OK - its what you get in the public cloud anyway
- 76. #mesoscon 2015 Finally… • Don’t force everybody into the same box • Instead — Manage the boxes (all shapes and sizes) consistently • What goes in the box is always changing!