Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (20)
Semelhante a The Berkshelf Way
Semelhante a The Berkshelf Way (20)
Último
Último (20)
The Berkshelf Way
- 1. JAMIE WINSOR| SOFTWARE ENGINEER THE BERKSHELF WAY
- 6. ME IN 2009 200 201 201 201
- 7. 200 201 201 201 MY CHEF REPO IN 2010
- 8. 200 201 201 201 AND IN 2011…
- 9. 200 201 201 201 …FROM 2012 AND ON
- 11. WHAT’S BERKSH ELF? ‣ A COMMAND LINE TOOL ‣ A SOURCE CODE MANAGEMENT TOOL ▾ Similar to „Mix‟, „Go‟, or „Leiningen‟ ‣ A PACKAGE MANAGER ▾ Similar to „Gem‟, „Apt‟, or „Yum‟ ‣ REPLACES PORTIONS OF KNIFE ▾ Cookbook Generator, Cookbook Uploader, Cookbook Downloader
- 12. WHY IS IT CALLED BERKSHELF?
- 14. COOKBOOKS ARE FED INTO CHEF NGINX COOKBOO K 1.2 .3 Chef-Client / Chef-Solo
- 15. RECIPES GO IN COOKBOOKS NGINX COOKBOO K 1.2 .3 RECI PE RECI PE RECI PE
- 16. RECIPES CONTAIN RESOURCE COLLECTIONS DEFAU LT PACKAGE NGINX SERVICE NGINX REPO YUM_REPOSI TORY NGINX YUM_KEY NGINX NGINX COOKBOO K 1.2 .3
- 17. EXAMPLE RESOURCE 1 2 3 package “nginx” do action :install end PACKAGE NGINX
- 19. INSTALL BERKSHELF $ gem install berkshelf Successfully installed berkshelf-1.4.0 1 gem installed
- 20. CREATE A NEW COOKBOOK $ berks cookbook myface create myface/files/default create myface/templates/default create myface/attributes create myface/definitions create myface/libraries create myface/providers create myface/recipes create myface/resources create myface/recipes/default.rb create myface/metadata.rb create myface/LICENSE create myface/README.md create myface/Berksfile create myface/chefignore create myface/.gitignore run git init from "./myface" create myface/Gemfile create myface/Vagrantfile Using myface (0.1.0) at path: '/Users/reset/code/riot-cookbooks/myface'
- 21. CONVERT AN EXISTING COOKBOOK $ berks init create Berksfile create chefignore create .gitignore run git init from "." create Gemfile create Vagrantfile Using old_cookbook (0.1.0) at path: '/Users/reset/code/riot-cookbooks/old_cookbook' Successfully initialized
- 22. STUCK? ASK FOR HELP! $ berks cookbook –h Usage: berks cookbook NAME Options: [--foodcritic] # Creates a Thorfile with Foodcritic support to lint test your cookbook [--chef-minitest] # Creates chef-minitest support files and directories, adds minitest-handler … [--scmversion] # Creates a Thorfile with SCMVersion support to manage versions for continuous integration -L, [--license=LICENSE] # License for cookbook (apachev2, gplv2, gplv3, mit, reserved) # Default: reserved -m, [--maintainer=MAINTAINER] # Name of cookbook maintainer -e, [--maintainer-email=MAINTAINER_EMAIL] # Email address of cookbook maintainer [--no-bundler] # Skips generation of a Gemfile and other Bundler specific support [--skip-vagrant] # Skips adding a Vagrantfile and adding supporting gems to the Gemfile [--skip-git] # Skips adding a .gitignore and running git init in the cookbook directory -c, [--config=PATH] # Path to Berkshelf configuration to use. -F, [--format=FORMAT] # Output format to use. # Default: human -q, [--quiet] # Silence all informational output. -d, [--debug] # Output debug information Create a skeleton for a new cookbook
- 23. A COMPLETE EXAMPLE CHEF- SERVE RCOOKBOO K 1.2 .3 ├── Berksfile ├── CHANGELOG.md ├── CONTRIBUTING.md ├── Gemfile ├── LICENSE ├── README.md ├── Thorfile ├── Vagrantfile ├── attributes │ └── default.rb ├── chefignore ├── libraries │ ├── dev_helper.rb │ └── omnitruck_client.rb ├── metadata.rb ├── recipes │ ├── default.rb │ └── dev.rb └── templates └── default └── chef-server.rb.erb Berksfile metadata.rb
- 25. COOKB OOK METADATA IS IMPORTANT CHEF- SERVE R THE NAME OF THE COOKBOOK
- 26. COOKB OOK CHEF- SERVE R METADATA IS IMPORTANT V2.0.0 THE VERSION OF THE COOKBOOK FOLLOW SEMVER: HTTP://SEMVER.O
- 27. METADATA IS IMPORTANT COOKBO OK YUM V1.2.3 COOKBO OK CHEF- SERVER V1.2.3 COOKBO OK APT V1.2.3 Red Hat CentOS Ubuntu SET YOUR SUPPORTED PLATFORMS CentOS Ubuntu Red Hat
- 28. METADATA IS IMPORTANT COOKBO OK YUM V1.2.3 SET WHAT COOKBOOKS YOU DEPEND ON COOKBO OK CHEF- SERVER V1.2.3 COOKBO OK APT V1.2.3 SUPPORTED PLATFORMS SUPPORTED PLATFORMS SUPPORTED PLATFORMS
- 29. CHEF-SERVER COOKBOOK’S METADATA name "chef-server" maintainer "Opscode, Inc." maintainer_email "cookbooks@opscode.com" license "Apache 2.0" description "Installs and configures Chef Server" long_description IO.read(File.join(File.dirname(__FILE__), 'README.md')) version "2.0.0" depends "apt", ">= 1.7.0" depends "yum", ">= 0.5.0" %w{ ubuntu redhat centos fedora amazon scientific oracle }.each do |os| supports os end
- 30. IT STARTS WITH A BERKSFILE ├── Berksfile ├── CHANGELOG.md ├── CONTRIBUTING.md ├── Gemfile ├── LICENSE ├── README.md ├── Thorfile ├── Vagrantfile ├── attributes │ └── default.rb ├── chefignore ├── libraries │ ├── dev_helper.rb │ └── omnitruck_client.rb ├── metadata.rb ├── recipes │ ├── default.rb │ └── dev.rb └── templates └── default └── chef-server.rb.erb Berksfile site :opscode metadata group :dev do cookbook 'git' end Berksfile
- 31. INSTALLING COOKBOOKS Using chef-server (2.0.0) at path: '/Users/reset/code/riot-cookbooks/chef-server' Installing git (2.5.0) from site: 'http://cookbooks.opscode.com/api/v1/cookbooks' Installing apt (1.9.2) from site: 'http://cookbooks.opscode.com/api/v1/cookbooks' Installing yum (2.2.0) from site: 'http://cookbooks.opscode.com/api/v1/cookbooks' Installing dmg (1.1.0) from site: 'http://cookbooks.opscode.com/api/v1/cookbooks' Installing build-essential (1.4.0) from site: 'http://cookbooks.opscode.com/api/v1/cookbooks' Installing windows (1.8.8) from site: 'http://cookbooks.opscode.com/api/v1/cookbooks' Installing chef_handler (1.1.4) from site: 'http://cookbooks.opscode.com/api/v1/cookbooks' Installing runit (1.1.2) from site: 'http://cookbooks.opscode.com/api/v1/cookbooks' $ berks install
- 32. INSTALLED TO WHERE? $ ls ~/.berkshelf/cookbooks apache2-1.6.0 chef_handler-1.1.4 known_host-0.1.3 nginx-1.3.0 riot_chef-client-2.0.7 tsf4g-0.0.14 apt-1.9.0 couchdb-2.4.0 liquibase-0.0.1 not_system_ruby-0.1.7 riot_coherence-0.10.12 ucspi-tcp-1.0.0 ark-0.0.11 cron-1.0.4 logrotate-0.8.2 ntp-1.1.8 riot_ejabberd-0.0.152 ulimit-0.1.3 artifact-0.10.10 daemontools-1.0.0 logrotate-1.1.0 ohai-1.1.6 riot_ejabberd-0.0.188 users-1.1.4 artifact-0.10.2 database-1.3.12 lvm-0.8.6 openssl-1.0.0 riot_geoip-0.1.0 vim-1.0.2 artifact-1.2.0 database-1.3.4 magic_shell-0.2.0 postgresql-1.0.0 riot_java-0.1.4 windows-1.8.2 artifact-1.3.1 dmg-1.1.0 man-0.7.0 pvpnet_core-2.3.123 riot_java-0.1.8 xfs-1.0.0 artifact-1.4.0 editor-0.2.0 mysql-1.2.4 pvpnet_core-2.3.131 riot_mysql-2.0.1 xml-1.1.2 aws-0.100.0 erlang-1.2.0 mysql-1.3.0 python-1.2.2 riot_rtview-1.0.1 yum-2.1.0 bluepill-1.1.0 gecode-2.0.0 mysql-2.1.0 rbenv-1.3.2 riot_team-0.9.41 zlib-2.0.0 build-essential-1.1.2 git-2.2.0 mysql-2.1.2 riot_activemq-0.9.28 riot_tomcat-1.0.18 build-essential-1.3.4 hornetq-0.0.27 networking_basic-0.0.5 riot_base-2.7.32 runit-0.16.2 chef-client-2.1.4 java-1.5.0 nexus-1.1.0 riot_base-2.7.37 runit-1.0.4 chef-server-1.1.0 java-1.9.0 nginx-1.2.0 riot_base-2.7.49 sudo-1.2.0
- 33. UPLOADING COOKBOOKS Using chef-server (2.0.0) at path: '/Users/reset/code/riot-cookbooks/chef-server' Using git (2.2.0) Using apt (1.9.0) Using yum (2.1.0) Using dmg (1.1.0) Using build-essential (1.3.4) Using windows (1.8.2) Using chef_handler (1.1.4) Using runit (0.16.2) Uploading chef-server (2.0.0) to: 'https://api.opscode.com:443/organizations/vialstudios' Uploading git (2.2.0) to: 'https://api.opscode.com:443/organizations/vialstudios' Uploading apt (1.9.0) to: 'https://api.opscode.com:443/organizations/vialstudios' Uploading yum (2.1.0) to: 'https://api.opscode.com:443/organizations/vialstudios' Uploading dmg (1.1.0) to: 'https://api.opscode.com:443/organizations/vialstudios' Uploading build-essential (1.3.4) to: 'https://api.opscode.com:443/organizations/vialstudios' Uploading windows (1.8.2) to: 'https://api.opscode.com:443/organizations/vialstudios' Uploading chef_handler (1.1.4) to: 'https://api.opscode.com:443/organizations/vialstudios' Uploading runit (0.16.2) to: 'https://api.opscode.com:443/organizations/vialstudios' $ berks upload
- 38. SERVICES ARE MADE UP OF COMPONENTS 1 app servers 2 cache servers 3 database servers 4 background workers 5 load balancers 6 etc
- 39. APPLY A DESIGN PATTERN
- 42. COMPONENTS OF MYFACE 1 load balancer 2 application server proxy 3 caching service 4 application server 5 background worker 6 database server
- 45. CRACK OPEN MYFACE::DATABASE_SERVER node.set[:mysql][:port] = 3309 node.set[:mysql][:tunable][:max_connections] = 1000 include_recipe "myface::_common_system" include_recipe "mysql::server"
- 46. PUBLIC/PRIVATE RECIPES PUBLIC RECIPE ‣ What you put in the run_list of a node ‣ Documented in the README ‣ Documented in the metadata PRIVATE RECIPE ‣ Not exposed to end user ‣ Should never be put in the run_list of a node ‣ Always included into other recipes ‣ Documented in the code for other developers
- 47. WHAT’S IN MYFACE::_COMMON_SYSTEM? include_recipe "ntp::default" # . . . MAYBE SETUP SOME USERS . . . # . . . MAYBE SETUP SOME ADDITIONAL REPOS . . .
- 48. WORK IN VERTICALS FROM THE OUTSIDE-INFAVOR DATA- DRIVEN COOKBOOKS THE BERKSHE LF WAY
- 50. TUNABLES OF MYFACE 1 listen on a different port or bind address 2 set memory usage of workers 3 any app config 4 set services to started or stopped states
- 51. 3 WAYS TO CONFIGURE ATTRIBUTES DATA BAGS ENCRYPTED DATA BAGS
- 52. FAVOR CONFIGURING WITH ATTRIBUTES ‣ Path of least resistance for cookbook consumers ‣ Provide sensible defaults in the Attribute files ▾ Consider creating an attribute file for each recipe ‣ Can configure your application on an environment level
- 53. WHEN TO USE DATA BAGS USERS GROUPS YUM/APT THINGS CONFIGURED BY "BASE" COOKBOOK (IF YOU HAVE ONE) ORGANIZATION LEVEL CONFIGURATIONS
- 54. USE ENCRYPTED DATA BAGS WHEN STORING SENSITIVE DATA
- 55. WORK IN VERTICALS FROM THE OUTSIDE-INFAVOR DATA- DRIVEN COOKBOOKS WRITE WELL- ENCAPSULATED COOKBOOKS THE BERKSHE LF WAY
- 58. NOT VERSIONED CAN’T BE PACKAGED / DISTRIBUTED NOT NAMESPACED STAHP USING ROLES THEY ARE ORGANIZATION LEVEL DATA ADDITIONAL SETUP COMPLEXITY FOR THE USER
- 60. USE DATA BAGS WHERE THEY MAKE SENSE DO YOU NEED TO STORE ORGANIZATION-WIDE DATA? DO YOU NEED TO STORE SENSITIVE DATA? Use one Use an encrypted one
- 61. USE DATA BAGS WHERE THEY MAKE SENSE MOST IMPORTANTLY… VALIDATE YOUR DATA BAGS!
- 62. 5 include_recipe "myface::database” 6 include_recipe "myface::app_server” 7 include_recipe "myface::worker_pool” 8 include_recipe "myface::cache_server” 9 include_recipe "myface::app_proxy" DEFAULTS RECIPE IS SPECIAL
- 63. LIGHTWEIGHT RESOURCES AND PROVIDERS (LWRP) LIBRARIES DEFINITIONS COOKBOOKS ARE MORE THAN RECIPES COOKBOOKS DON’T NEED TO HAVE RECIPES
- 65. LIBRARY COOKBOOKS ‣ Probably doesn‟t have recipes ‣ Contain LWRPs, Definitions, or Libraries useful to other cookbook authors ‣ Database Cookbook is a perfect example. Has HWRP‟s for ▾ Managing mysql, postgres, and sql server databases ▾ Managing database users in mysql, postgres, and sql server
- 67. WRAPPER COOKBOOKS ‣ Super lightweight ‣ Contain recipes ‣ Contain attribute overrides ‣ Riot-Java is a perfect example ▾ Contains a Oracle Java 6 recipe ▾ Contains a Oracle Java 7 recipe ▾ An abstraction on top of the Java cookbook for Riot engineers ▾ Overrides the default “install flavor” for Java ▾ Overrides the default location of the Java artifacts to an internal location ▾ NOT A FORK OF THE JAVA COOKBOOK
- 68. WORK IN VERTICALS FROM THE OUTSIDE-INFAVOR DATA- DRIVEN COOKBOOKS WRITE WELL- ENCAPSULATED COOKBOOKS HAVE SHORT ITERATION LOOPS THE BERKSHE LF WAY
- 69. OLD WORKFLOW EDIT COOKBOOK UPLOAD COOKBOOK PROVISION MACHINE BOOTSTRAP MACHINE RUN CHEF CLIENT SSH AND VALIDATE 5-30
- 70. NEW WORKFLOW EDIT COOKBOOK VAGRANT UP / VAGRANT PROVISION TEST / SSH AND VALIDATE 5 seconds – 1 minute
- 72. BERKSHELF VAGRANT PLUG-IN $ vagrant plugin install vagrant-berkshelf Installing the 'vagrant-berkshelf' plugin. This can take a few minutes... Installed the plugin 'vagrant-berkshelf (1.2.0)'!
- 73. Bringing machine 'default' up with 'virtualbox' provider... [Berkshelf] Updating Vagrant's berkshelf: '/Users/reset/.berkshelf/vagrant/berkshelf-20130424-40671-t43soz’ Generating chef JSON and uploading... Running chef-solo... [2013-04-25T03:01:31+00:00] INFO: *** Chef 10.14.2 *** [2013-04-25T03:01:32+00:00] INFO: Setting the run_list to ["recipe[minitest-handler::default]", "recipe[myface::default]"] from JSON [2013-04-25T03:01:32+00:00] INFO: Starting Chef Run for myface-berkshelf #### CHEF RUN GOES HERE #### [2013-04-25T03:02:10+00:00] INFO: Chef Run complete in 38.330025825 seconds [2013-04-25T03:02:10+00:00] INFO: Running report handlers Run options: -v --seed 41224 # Running tests: minitesthandler::default#test_0001_runs_no_tests = 0.00 s = . myface::default#test_0001_runs_no_tests_by_default = 0.00 s = . Finished tests in 0.002425s, 824.7576 tests/s, 0.0000 assertions/s. 2 tests, 0 assertions, 0 failures, 0 errors, 0 skips [2013-04-25T03:02:10+00:00] INFO: Report handlers complete PROVISION NEW ENVIRONMENT IN MINUTES $ vagrant up 1 minute 20 sec
- 75. Running chef-solo... [2013-04-25T03:01:31+00:00] INFO: *** Chef 10.14.2 *** [2013-04-25T03:01:32+00:00] INFO: Setting the run_list to ["recipe[minitest-handler::default]", "recipe[myface::default]"] from JSON [2013-04-25T03:01:32+00:00] INFO: Starting Chef Run for myface-berkshelf #### CHEF RUN GOES HERE #### [2013-04-25T03:02:10+00:00] INFO: Chef Run complete in 0.520326234 seconds [2013-04-25T03:02:10+00:00] INFO: Running report handlers Run options: -v --seed 41224 # Running tests: minitesthandler::default#test_0001_runs_no_tests = 0.00 s = . myface::default#test_0001_runs_no_tests_by_default = 0.00 s = . Finished tests in 0.002425s, 824.7576 tests/s, 0.0000 assertions/s. 2 tests, 0 assertions, 0 failures, 0 errors, 0 skips [2013-04-25T03:02:10+00:00] INFO: Report handlers complete CONVERGE IN SECONDS $ vagrant provision 7 seconds
- 76. WORK IN VERTICALS FROM THE OUTSIDEFAVOR DATA- DRIVEN COOKBOOKS WRITE WELL- ENCAPSULATED COOKBOOKS HAVE SHORT ITERATION LOOPS THE BERKSHE LF WAY
- 78. CONTRIBUTE GITHUB.COM/RIOTGAM ES/BERKSHELF HOW CAN YOU HELP? OR JOIN US WWW.RIOTGAMES.COM/CAREERS
Notas do Editor
- Hey everyone, I’m Jamie Winsor and I love video games. I’ve been playing for a while; I’ve been enamoured with video games since I was a little older than two. I distinctly remember wanting to learn how to read because games like Zelda were too difficult and I learned how to type so I would be able to communicate with people in Everquest more easily.In an open letter to my third grade teacher I told her I’d be a “video game maker” by 2013. Here it is.*NEXT*What’s up now Mrs. Hoey? I might not have made 1 million cartridges like I promised, but I’m working on it.I also apparently didn’t know what Thanksgiving was all about when I was in kindergarten*NEXT*But you didn’t come to hear about what my League of Legends ELO is, or about the time I camped Raster of Guk for 34 hours straight to get my Monk’s epic in Everquest. You’re probably here for something more Chef related.But really, my love for games is what brought me to this stage in the first place. Specifically my passion for persistent, always-on, online games like World of Warcraft, Everquest, or League of Legends, is why I am here today. You see, persistent online games have a unique problem to solve that has a lot more in common with other online services like Amazon, Facebook, or online banking software. They need to be available 24 hours a day and 7 days a week. And these games can get big.I work for a company in located in Santa Monica called Riot Games and we are the creators of one of my all time favorite games; League of Legends.
- LoL is one of these always-on, online games. It came out in 2009 and since then has grown quite a bit. We are one of the most played games in the world with 32 million active monthly players, 12 million of which login and play every day and at a peak time during the day there is 5 million players playing at once.And players have high expectation for the availability of these online games. If my online banking software goes down (and it does ALL the time) I’ll just come back later. If I go home, crack open a beer and try to play some a game with some of my friends and IT’S down?*NEXT**PAUSE FOR LOLS**NEXT*I get mad and, I work on, these things and understand the space. Dedicated players have high expectations and that bar is continually getting raised. It’s no longer acceptable to have “launch day” problems or minimal content for players to experience. That newness factor of simply being able to get into a game and shoot a rocket at your friend has long worn off for a lot of us. We demand not only triple A content but also a triple A online service.
- At Riot, we aspire to be the most player focused game company in the world. Everything we do at Riot is driven by us trying to do what is best for our players.For the last 4 years I have been on a quest to help build a reliable platform that software engineers and operations people alike could use to deploy and configure their software with hopes that the gaming community, my friends, and myself could enjoy higher uptimes and more rapid delivery of new content. And by using and contributing to the open source world, my dream might actually become a reality.
- At Riot we are using Chef to bring league of legends to players around the world and it is the very core component to our publishing platform. And I love Chef. I love almost everything about Chef; the people making it, the software, the language it's written in, the community.But for as much as I love Chef, there’s a learning curve that goes with it.
- And that curve can be steep. This particular graph represents the difficulty level of popular massively multiplayer online games. It’s the time spent playing versus the level of skill required as you go.Now Chef definitely isn’t this bad, but it doesn’t matter if you have a background in software engineering or a background in operations, it can be rough man.It's been about four years now since the first time I used Chef. I still remember my first time, “ahh back in 2009 if I recall correctly”
- This was me. Sitting there wondering what the in the hell is a cookbook? The clever names of the Chef primitives, and just the sheer number of them that you will learn in your first hours of working with Chef can be daunting to a new user.Once I had a handle on the terminology I was well on my way to automating my infrastructure. I was creating cookbooks, roles, I think data bags came out somewhere in there and I was exploring those as well.I had built up quite the Chef repository and I was getting shit done.
- This is what my Chef repository looked like when I had a “strong grasp” of Chef. For anyone who might not know what a “Chef repository is”; it’s a collection of all the things that you put in your Chef server and it’s generally stored using a source control manager like Git.My cookbooks were stored in the “site-cookbooks” directory, anything that I had downloaded from the community site was in the “cookbooks” directory and then any changes I made to those cookbooks were just committed right on top of the master branch. I had a pretty large repository with a lot of cookbooks and with the tangled mess of my own cookbooks and my changes to cookbooks I was given, it was nearly impossible to receive updates from those cookbooks that I had downloaded from the community site.I did a lot of experimenting to try and figure out how I could clean this up. I was really surprised that I was having such a hard time since this was the “recommended way” of doing things among the community.The next experiment that I ran was pretty much the last before I hit a good stride. I thought, “I know…”
- …I’ll use GitSubmodules to maintain the cookbooks in my repository and clean that right up.This was a dark time. Git can be confusing to even the beardiest of technologists and while there’s nothing complicated about submodules on the surface, they can be a beast to maintain. There’s a sort of ritual that you’ve got to remember to perform to keep everything up to date.It wasn’t until I decided to stop keeping cookbooks inside of a Chef repository until things really started to take off
- By not storing cookbooks in the chef repository, they were just more portable.I could package one up and give it to somebody else. I could take one from somebody else and use it in our infrastructure and it would just work. It also encouraged less long lived forks. Instead of treating the community NGINX cookbook as a basis for a “Riot NGINX” cookbook by creating a long lived fork of it in our Chef repository, we instead were helping to fix or make the NGINX community cookbook more generic so it was useful to more people.There were some rules that we had to establish to make sure these cookbooks would be portable and re-usable. We also needed to create a bit of tooling to make it all possible.
- The tool we set out to build started as a spike, or an experiment.Initially we had a knife plugin that did the job of getting all of the dependencies that a Chef repository had.Things went in a very different direction once we proved that the spike was a good idea but it needed more of the team’s attention. We started to follow README driven development: This is when you write down exactly how you want to explain to somebody how to use your software before you actually write your software.Through this exercise we realized what we really needed to build was a mix between the rubygem’s “Gem” command and Bundler, for cookbooks. We eventually landed on what we call today, Berkshelf.
- Berkshelf is a standalone binary*NEXT*It’s a command line tool*NEXT*It’s a source code management tool similar to Mix from Elixir, Go from Go, or Leiningen from Clojure*NEXT*It’s a package manager similar to Gem, Apt, or Yum*NEXT*And it reproduces the Cookbook generation, packaging, and distribution portions of KnifeEven after I tell people what Berkshelf is, there’s still an elephant in the room, an unanswered question left on everyone’s mind, so I’ll just get that out of the way.
- Why is it called Berkshelf?If you go through the commit history you’ll see that it wasn’t always called Berkshelf. When we finally landed on the idea to store cookbooks on your own local machine we wanted to call it Bookshelf. It fit with the the naming theme Chef already had going for it: It’s where you store cookbooks in your house, but that’s just as easy to google as Chef and Knife.My team and I sit in a jabber chat room throughout the day where we share image macros of extremely important things. One of those was this particular image:*NEXT*This picture of a little girl telling you just how much she loves Gersberms, her fravrit berks.
- I know a lot of you here at ChefConf already know all about cookbooks, but let’s take a quick minute to explain what a cookbook is for anyone watching the VHS after the conference or anyone in the audience whose a newcomer.
- Cookbooks contain instructions for how to install and configure a thing or components of a thing. These cookbooks are fed into Chef-Client or Chef-Solo along with some information about your computer.In this example we’re feeding the NGINX cookbook into Chef
- Every cookbook contains at least one recipe (called default.rb), but it could contain more with more verbose names. These recipes are where the instructions are held for configuring your machine.
- And these instructions are represented by collections of resources.Resources are an abstraction on top of your operating system to manipulate things like * services * files * directories * packages * yum repositories
- Here is one of the resources from the default recipe in the NGINX cookbook. It’s a package resource which will install the “NGINX” package on your computer.The really cool thing here is that this resource doesn’t make any mention about WHAT operating system we’re running on. It just knows that you want to install the package called NGINX and it knows how to do that depending on which operating system this recipe is executed on.This is why I love Chef. Chef gives us the power to make configuring machines easy, regardless of what platform. In the very near future I envision a world where people use Chef to configure their machines not just because it’s a repeatable and reliable way, but just because it’s easier to do it then it is to manually do it.One of the ways that we’re going to get there is by providing Chef developers with tools that allow them to be productive as possible.
- And we think that Berkshelf is that tool. Let’s take a quick 5 minutes to all get familiar.
- The first thing you want to do is get Ruby on your machine.Assuming that you already have Ruby, all you need to do is install the Berkshelf gem with this command here.This will give you access to the “berks” command. With the berks command you can do things like generate a new cookbook
- The cookbook generated will include everything that you need to get moving.And if you already have an existing cookbook that you want to add Berkshelf support to
- You can just run the Berks init command
- And if at any time you get stuck you can always just ask Berks for help by passing the –h flag to berks or any of it’s sub commands. Like any good unix utility, it will give you a description of all the commands and the option flags that they support.
- To speed things up let’s look at a cookbook that already exists. The Chef-Server community cookbook. This is the cookbook which you would use to build an open source Chef Server in your infrastructure.*NEXT*Here is the directory structure of the Chef-Server cookbook. It pretty standard stuff. There are two files in the root of the cookbook which are very important to the operation of Berkshelf that we need to highlight.*NEXT*The Berksfile and metadata files
- One of the most important yet, most neglected files in a cookbook is the metadata file. It describes a lot of very important pieces of information.
- It contains the name of the cookbook. In the current version of Chef, this field is automatically populated by the name of the directory that the Cookbook is in.This isn’t the behavior that you want: the Chef-Server cookbook is still the Chef-Server cookbook even if it’s in a directory called Cheese-Steak. This will be fixed in a future version of Chef, but for now PLEASE explicitly set the name of your cookbook.
- It also contains the version of the cookbook.Version numbers in Chef are represented using a versioning scheme called Semantic Versioning or SemVer. You can read the RFC at semver.org if you want to study up, but you really just need to know that there are three numbers and two dots.If the first number changes then there’s probably breaking changes between the last released version that you should be aware of and there’s probably a bunch of awesome features in it. It’s called the Major version.When the second number changes it means there’s some new features, enhancements to old features, and cumulative bug fixes in the new version. It’s called the Minor version.A cookbook whose third number has been changed indicates bug fixes only. This is called the patch version. It’s almost always a good idea to accept the latest patch version of a cookbook.None of these numbers should ever decrement unless the number to their left is incremented. For example: 1.2.3 is a lower version than 2.0.0.
- The metadata also included is a list of supported platforms, like CentOS, Red Hat, or Ubuntu.When authoring a cookbook, be sure to include all of the platforms you know that cookbook to work on. Please don’t leave it blank or list EVERY platform if you haven’t tested the cookbook on them all. People will hate you for wasting their time.This attribute will become more important as the adoption rates for automated testing tools, like test kitchen, increase. These tools will reflect on the supported platforms to know what platforms to automatically test your cookbook against.
- And the last important thing that you can describe in a Cookbook’s metadata is any dependencies upon other Cookbooks.Anytime you leverage an LWRP, library, definition, or recipe from another cookbook, that cookbook should be listed as a dependency in your metadata.The purpose of this is two fold:1. it will ensure that your Chef run will fail early if the Chef-Client doesn’t have all of the necessary cookbooks to successfully converge2. It will also give Berkshelf a list of Cookbook versions that it needs to satisfy and retrieve
- This is what the actual contents of the Chef-Server metadata looks like
- Now, the Berksfile is what is actually given to Berkshelf.It defines a list of cookbook sources that you want to retrieve. The Berksfile for the Chef-Server cookbook is pretty basic containing only about 5 lines of code.The first line says, “look on the Opscode community site for any cookbooks we define without an explicit location”.The second line instructs Berkshelf to look at the metadata file found in the current working directory. Pretty much every cookbook that contains a Berksfile will have this line in it. Berkshelf will read the metadata and retrieve any dependencies listed in the metadata and also their dependencies, and their dependencies, and so on.
- Now let’s go and get all the cookbooks we need to operate the Chef-Server cookbook. We do that with the Berks install command.*NEXT*Run Berks install*NEXT*And then Berkshelf will reflect on the metadata of the Chef-Server cookbook and go and download all of it’s dependencies (and their dependencies) from the Opscode community site.
- If you’ve been using Chef you might ask yourself, “where did those cookbooks go?”. Afterall, I haven’t made mention of a Chef repository with a “cookbooks” or a “site-cookbooks” directory at all.*NEXT*That’s because, just like Ruby’s Gems, they are installed into a common location on your local disk. We call this “the berkshelf”.And unlike a “cookbooks” directory in a Chef repository, you’ll notice that there can be multiple versions of a cookbook. This is because, like Rubygems, we store every version of a cookbook that you’ve installed.
- In order to get this behavior we couldn’t just write some software and call it a day. We also needed to make some changes to the way we we’re writing cookbooks and managing our Chef repository. We call this paradigm shift “The Berkshelf Way”.It’s a growing set of principles and best practices that we’ve discovered through creating, and teaching others on how to create cookbooks. Our goal for Berkshelf and The Berkshelf Way is to get an engineer managing their application in a production environment using Chef as soon as possible.
- One of the first things we stress is to work in verticals from the outside-in by starting on the surface. I love the idea of behavior driven development. I think that things are easier to accomplish if you have a clear and attainable goal.Think about what our ACTUAL goal is when we implement Chef in our organizations? You wouldn’t be wrong if you said it’s to configure a massive amount of computers on your network, but let’s pull back even further.What is it exactly that your organization is providing to it’s customers?
- Let's take Riot Games for example. What do we provide to our players?Is it NGINX?Is it CentOS?No, it’s League of Legends.
- So create a cookbook for that!Every application or service that you are a provider for should have it's own cookbook.In the case of League of Legends we have a lot of different services which make up what our players perceive as the game. Each one of these services has it’s own cookbook which is developed and maintained by the engineering team who wrote the service. And this cookbook lives not in a Chef repository, but next to the service’s codebase and in a repository of it’s own.
- And each one of these services is made up of components.This is a common list of components that you might see. And when a software engineer has a commonly occurring problem he can often remedy it by applying something called a Design pattern.
- A design pattern is a solution to a commonly occurring problem within a given context in the software world.
- We can map a service in our infrastructure to a cookbook of it’s own which will allow us to work in verticals and from the outside-in. We call this the Application Cookbook Pattern.This idea isn't something that comes with the Chefdocumentation. There's no choice to generate or designate a new cookbook as an “Application Cookbook”. It’s just a normal cookbook which follows a simple pattern that deploys and configures one of your services.In Riot’s case this is League of Legends or the services surrounding it.
- Let's take a look at an application cookbook for my fictional social networking application "Myface”.Let's imagine that this is the only product that my company provides and it's a large app that rivals Myspace and Facebook. It's comprised of a couple of different components and it's distributed across a number of machines in a datacenter.
- After mapping the components to their own recipe, we end up with these 7Anyone of these recipes can be put in the run list of a node and it should converge successfully.Each recipe configures/installs only that one component and nothing more.
- Let’s open up the database server recipe.In it we are setting the mysql port to listen on 3309 instead of 3306 and the max connections to 1000 instead of 800.We’re also including a few other recipes, common system from the myface cookbook and the server recipe from the mysql cookbook.Look at what we were able to do here with just 4 lines of code. We hid the implementation detail that myface uses mysql by wrapping it in this database server recipe. We made things easier for the operator or another engineer by just having this neat little package that can be included in the run list of a machine to get a database server for myface.Now what about that _common_system recipe. That wasn’t even in the list of 7 recipes that I showed you earlier and it doesn’t map to any of myface’s components.
- That’s because _common_system is what I like to call a “private recipe”.These are never exposed to the end user. They’re never even put in the run list of a node. They’re always included in other recipes to get some sort of shared and encapsulated behavior between recipes. These should be well documented in the code for other developers to take advantage of.Public recipes are what you actually put in the run list of a node and they are well documented in the README and metadata.
- And if we look into common system this is what you see
- By exposing the ‘max_connections’ and ‘port’ tunables in the mysql cookbook we were able to customize what a MyFACE database server looked like without ever opening up the MySQL cookbook and editing it ourselves.This is because the MySQL cookbook is a data driven cookbook.
- Writing data-driven cookbook’s allows us to change it’s behavior at runtime. That dream of having a single, high quality, cookbook for any major service that can be installed on a machine is both real and possible.
- We can also make our applications easier to manage by exposing a set of tunables.Let’s go back to our MyFACE cookbook. Operators could make the application:Listen on a different port or bind addressChange the memory usage of workersAny application configuration, reallyEven set services to started or stopped states
- The way a cookbook behaves can be changed without ever forking and changing the actual cookbook with these 3 primitives. They act as a sort of parameter list to your cookbook.By reflecting on attribute, or data bag data, we can create high quality, portable, re-usable cookbooks. This should be the goal of any of the cookbooks that you seen on the community site.Really, there shouldn't be a reason that we all have our own MySQL or NGINX cookbook.
- Out of the three given primitives my first choice is always attributes.They often require no additional work on the cookbook consumer because you can provide sensible default values within an attribute file.As my cookbook grows I like to create an attribute file for each recipe.Attributes are also pretty bad-ass because you can configure components of your application on an environment level like, “all of production”.All of production has this feature disabled or this rate limit set or this max connections set.
- Data bags have their place, but you put a bit more burden on the consumer of your cookbook when you require the use of a data bag because it, and any items need to be created and keyed into the server without any sort of validation.It also requires a bit more finesse as a cookbook author. What happens if the data bag hasn’t been created and a chef run is executed? Well, the user gets this cryptic 404 message unless you handled the loading and validations in your recipe.However, they are particularly useful if you want to set stuff on an organization level. You can do things like:Setup a set of users to be put on every machineOr groupsYour organization specific yum or apt repositoriesThings configured by your “base” cookbook (if you have one)
- They are also useful if you have any kind of sensitive data to store. Right now there is no built-in primitive found in Chef to store sensitive data as an attribute so you need to rely on encrypted data bags or hack it in yourself.If you have a password or a private key that you need to store, put that in an encrypted data bag.
- The community site is an amazing ideal. Cookbooks which perform a purpose, only that purpose, that can be used in anyone’s infrastructure. A big collection of ready to use easy bake oven cookbooks.The problem is that a lot of these cookbooks are tied to the Chef Repository they were written around. If you package up a cookbook and send it over to me, I shouldn’t need parts of, or all of, your Chef Repository.This is a big reason why we wanted cookbooks to be developed in seclusion away from a Chef Repository. Writing cookbooks away from everything else in a project of their own produces well encapsulated and portable cookbooks.
- You’re probably asking yourself them, “how do I use roles if they aren’t packaged with my cookbooks”?
- Stahp it. Don’t. Don’t use em.
- Roles aren’tVersionedThey can’t be packaged and distributedThey aren’t namespaced. We’re called Riot Game(s). Think of how awesome it is when somebody names a role “game”They are organization level data. They aren’t tied to an environment, they are tied to an organization. If I have a role called “game server” and it includes two recipes in a run list and this role is used in production and also another QA environment which tests the latest version of the game. What do you think happens when we change that role for the QA environment? Well it changes what the signature of a game server looks like in production. We could be breaking something or releasing something when we aren’t ready!They add more onto the cookbook user who now also not only needs to understand the primitive, but they also need to create that role on their Chef server.
- Ok so you might say, “whoa… what about data bags?”They’re organization wide data too. Not only that, they’ve got a structure too them so they are even easier to break.
- Well… use them where they make sense.I mentioned that they hold organization level data. If you need to store organization-wide data, then use one.If you need to store sensitive data, then use an encrypted one.But because data bags aren't packaged with cookbooks, I try my best to avoid using them in favor of attributes.
- You should always validate that any data bag your cookbook requires has been created on the Chef server and is in the expected format as early as possible.
- Another key to writing well encapsulated and easy to use cookbooks is to have an entry point, a sort of main() function.The default recipe is special. It’s that main() function for a cookbook. Even if you have never read the README of a cookbook, you should still be able to put the default recipe in the run list of a node and have a node become a thing that the cookbook represents.And this will work because all we are doing is including the recipe of each component; in the proper order; that we provide, in the default recipe.
- Encapsulation and re-usability doesn’t only apply to recipes in a cookbook. A cookbook can provide more than that.A cookbook can also provide:Lightweight resources and providersLibrariesAnd DefinitionsCookbooks don’t even NEED to have recipes. They can just exist and contain these other things.
- We call these cookbooks, “Library Cookbooks”.
- These cookbooks might have recipes – but often times they don’t.They exist for the sole purpose to make your life easier when writing Chef recipes.The Database Cookbook is a perfect example
- And there’s one last pattern that I want to talk about: the Wrapper Cookbook Pattern.
- These are super lightweight cookbooks that provide an abstraction on top of another cookbook.They usually contains recipes and attribute overrides and nothing more.We have an internal cookbook called Riot-Java which is a perfect example of this pattern. It contains two recipes. One for Oracle Java 6 and one for Oracle Java 7.It’s an abstraction on top of the community Java cookbook for Riot Engineers. It automatically sets the install flavor to Riot’s taste and overrides the default location of the Java artifacts to an internal location.It is NOT a Fork of the Java cookbook. It’s just a thin wrapper around the Java cookbook so these things common to nearly all projects at Riot Games don’t need to be duplicated.
- Are you still with me? There’s just one last principle that we have to cover.Even if Chef was so easy that if everyone in the world picked up Chef and immediately became a master, all of this still wouldn’t matter.It wouldn’t because we have essentially been working developing in the dark for the last four years.
- Testing minor changes: 7 seconds
- Testing minor changes: 7 seconds
- It was a slow start when we set out to teach the other engineers at Riot Games how to use and develop for Chef.With Berkshelf and The Berkshelf Way, we were able to train over 30 Riot engineers how to effectively author cookbooks of production environment quality in just four weeks. These classes were held in house and taught by one member of my team. Each class was three days long and included more detailed explanations of the material that I’m going to show you today.