From ChefConf 2015.
https://youtu.be/FI5sQQh8aKw
When we first began using chef at Parse, we fell in love with it. Chef became our source of truth for everything. Bootstrapping, config files, package management, deploying software, service registration & discovery, db provisioning and backups and restores, cluster management, _everything_. But at some point we reached Peak Chef and realized our usage model was starting to cause more problems than it was solving for us. We still love the pants off of chef, but it is not the right tool for every job in every environment. I'll talk about the evolution of Parse's chef infrastructure, what we've opted to move out of chef, and some of the tradeoffs involved in using chef vs other tools.
12. Chef the Base System!!
• bootstrapping nodes with knife-ec2
• configuring system packages
• managing deb versions
• ec2 hostname tags from chef node names
• route53 DNS records from hostname tags
• cron jobs, batch jobs
13. Chef the Services!!
• haproxy configs
• generate yaml files
• generate host lists
• manage config files for Parse services
• monitoring and graphing based off roles
14. Chef the Databases!!
• creating/managing mongo replica sets
• provisioning & assembling RAID devices
• assigning cassandra initial tokens
• backups, snapshotting & restores
• community cookbooks for mysql, redis
18. 1) Things we did with
chef badly
2) Things that chef was
not the right tool for
19. mistakes were made …
• Overloading roles with too much work
• Confusion between role vs instantiation of service
• Using definitions instead of providers
• Using lots of data bags
• One attribute per config entry instead of a hash of all
entries
• Using knife search extensively
20. mistakes were made …
• Forking + modifying community cookbooks
• Importing community cookbooks with too many
custom dependencies
• Not using repo-per-cookbook / Berkshelf
• Not investing the time into vagrant, unit tests, staging
environment, versioning
• Where is my source of truth?!
32. Managing software & configs
Developer-owned services
• Do not tie code deploys to system changes
• Perform the minimal set of changes
• Configs *are* software. Version together.
33. Managing software & configs
Internal operations software
• Treat software engineering like software
engineering
• Treat systems-y packages like systems
packages
• Package and version “util” scripts
• Manage package versions with Chef
36. Databases
DBA operations
• Create, tear down replica sets or nodes
• Verify backups
• Rolling version upgrade
• Elect new primary / switch masters
• Enable/disable query killer
• Change schemas or indexes
• Compaction, rotation
• Version replica set state
• Etc
37. Databases
DBA operations
If you don’t have to do a ton of DBA
ops, Chef can manage databases.
Don’t over-engineer in advance of
your actual needs.
38. Databases
Separation of configuration and state
Base system => chef
Detect and publish state changes => chef, zk
Generate monitoring configs => chef
Imperative commands => db tooling
40. We chef for:
• Building base AMIs
• Generating monitoring configs
• Storing encrypted secrets
• Cron jobs (with zk lock)
• Inferring and publishing db state changes
41. Things we still suck at
• Single source of truth (git / chef-server)
• Isolated staging environment
• Full continuous testing for cookbooks
42. • Realtime data
• Internal software packaging & management
• Database administration at scale
Things we don’t chef