Julien Rottenberg, CityGrid. Presentation of the tools and workflow for our puppet setup. How puppet helps us managing 500+ servers in a hybrid environment O&O Datacenter and EC2, hands free. Watch the video at http://youtu.be/FPwga7HwomM
PuppetCamp LA, May '12.
14. Toolset
trunk = work in progress - could break but fix
ASAP
production = tested code - do not break
15. Toolset Trunk --> Production (CherryPicking)
We use svnmerge.py
Code in progress = technical debt
It needs to be merged and deployed or
reverted
Fail the build if pending Cherry Picks > 30
Can't CherryPick yourself ~ code review
16. Toolset
Svn even with svnmerge has limitations
Git ?
17. Toolset
Vagrant
Sits on top of Virtualbox
Allows to 'go crazy'
Test, experiment, share configured box
$ vagrant provision == puppet apply
21. Toolset
Man ! that's a lot of tools !1!!
And it's fine, design by contract.
Each application does one thing
and was designed to do that
thing only (so does it well)
Pipe them
Dennis Ritchie
22. Toolset
Boto for
● ec2
● route53
● elb
● sqs
● sns
River dolphin native to the amazon WP
23. Culture cloud
EC2 glitches, expect them
It used to `just work` :
● gmond (split view dns side effect, using
gmetric)
● jmx (need to specify public name at runtime)
● ELB pseudo load balancer
● Bad neighbors
25. Culture cloud
# Build 10 nodes
$ create_ec2_instance.py -e dev -s m1.medium -c thd -n 10
# assign them to the hadoop::datanode class (specific install on top of our vanilla)
$ nv --name aws1devthd --addtonodegroup hadoop::datanode
# send traffic & enable monitoring (configs to be updated on the next puppet run)
$ nv --name aws1devthd --set status=inservice
One AMI to rule them all (map set per region)
27. Culture Collaborate
"If the company's doing well and people
don't hate each other, you're probably
doing ok."
John Allspaw, speaking at Devopsdays 2011 MV.
28. You guys !
Make it better :
- Integrated function for 'secrets' (no get_var or
hiera + gpg)
Something like an encrypted data luggage
- puppet agent stability, puppetd vs cron ?
31. Two last things...
On call rotation side effect
Don't be a human keyboard
32. Shameless plug
We aggregate advertising & content from local
businesses and distribute across our network of
web & mobile sites
● Ingest : Hadoop & Cassandra
● Serve : Solr & MongoDb + Tomcat
● Code : Java mostly
33. Q/A This page : goo.gl/soxEs
puppet doc : http://docs.puppetlabs.com/guides/style_guide.html#puppet-doc
puppet-lint : http://puppet-lint.com/
Jenkins : http://jenkins-ci.org/
svn strategy : http://weblogs.java.net/blog/johnsmart/archive/2009/05/ci_branching_an.html
svnmerge.py : http://www.orcaware.com/svn/wiki/Svnmerge.py
vagrant : http://vagrantup.com/
nventory : http://sourceforge.net/apps/trac/nventory/wiki
foreman : http://theforeman.org/
boto : https://github.com/boto/boto
Don't be an Ahole : http://www.hollenback.net/DevOpsMeansDontBeAnAhole
Developers on call - Mitchell Hashimoto : http://vimeo.com/31367609
Don't be a human keyboard : http://sysadvent.blogspot.com/2010/12/day-13-dont-be-human-keyboard.html
Citygrid is hiring : http://careers.citygrid.com/
Puppet Master LA User Group : http://www.meetup.com/LAPMUG/