1.
Proprietary andConfidentialAutomating JoyentSmartMachines with ChefChef on SmartOSEric Saxby@sax @ecdysone @sax

2.
Who am I?Proprietary andConfidential■ Application developeroperational experience with manytechnologies, project by project■ BSD/AIX/UbuntuSolaris in 2002, but I was very muchout of my element■ Switched to DevOps-y team 18 months agoMultiple back end services for a large e-commerce site,transitioning to SmartOS■ Now I’m at Wanelo

3.
From a certain point of view...Proprietary andConfidential

4.
What is Wanelo?Proprietary andConfidential■ Wanelo (“Wah-nee-lo” from Want, NeedLove) is a global platform for shopping.

5.
Proprietary andConfidentialMarketing-free shopping across 100s ofthousands of unique stores

6.
Proprietary andConfidentialPersonal feed of products from anystore on the internet

7.
Technology overviewProprietary andConfidential■ MRI Ruby 1.9.3 & Rails 3.2■ PostgreSQL 9.2.4, Solr 3.6■ Joyent Cloud, SmartOSZFS, ARC, raw IO performance, SmartOS, CPU bursting, dTrace■ Circonus, Chef + OpscodeMonitoring, graphing, alerting, automation■ Amazon S3 + Fastly CDN■ NewRelic, statsd, Graphite, nagios

8.
What’s SmartOS?Proprietary andConfidential■ Illumos branch optimized for cloudcomputing■ Developed by Joyent for their publiccloud

9.
What’s Illumos?Proprietary andConfidential■ It’s what OpenSolaris became after Oraclekilled the project■ Umbrella for various distributions, eachcommitted to pushing their improvementsupstream■ http://wiki.illumos.org/display/illumos/About+illumos

10.
What does SmartOS lookProprietary andConfidential■ Compute Node — physical server■ Global Zone — host OS (SmartOS)■ Non-Global Zone — like a virtual machine, withnative system calls (no fake hardware layer)■ Very secure■ Can run KVM for guest OS (Ubuntu, Centos)

11.
How is it deployed?Proprietary andConfidential■ Can manage from global zone (imgadm,zoneadm)■ Tools provide APIs■ Smart Data Center (Joyent’s tools, can be licensed)■ Project FIFO (SDC API in free package)■ Joyent Public Cloud■ Many compute nodes working in a cluster,PXE booted from a head node

12.
■ Service Management Facility (SMF)If init.d and monit and god were one thing, andactually awesomeWhy should I care?Proprietary andConfidential■ Visibility toolsdtrace, kstat, snoop, truss■ ZFSFile system built for speed and data integrity■ Application LatencyZones are OS virtualization, so fasterProcesses are scheduled in global zone kernel,not in a hardware virtualization layer

13.
■ # cores, RAM required =~# processesLower latency == less costProprietary andConfidential■ # processes required =~requests/second of site■ Requests/second of single process =~request latency$$$

14.
On to Chef!Proprietary andConfidential

15.
TerminologyProprietary andConfidential■ Image / Dataset — OS at a particular version,snapshotted at base state■ Flavor / Package— RAM, CPU shares■ API URL — Each data center has its own URL■ Server ID / Zonename — Each zone gets aUUID

16.
knife-joyentProprietary andConfidential

17.
Installation/ConfigurationProprietary andConfidential■ Update knife.rb■ Add to Gemfileknife[:joyent_username] = saxknife[:joyent_keyname] = EricSaxbyknife[:joyent_keyfile] = "#{ENV[HOME]}/.ssh/id_rsa"knife[:joyent_api_url] = https://us-sw-1.api.joyentcloud.com/■ Add first public key in cloud APIhttps://my.joyentcloud.comgem knife-joyent

18.
Managing keysProprietary andConfidential■ No role based access, but at least you canmake each user upload their own keyknife joyent key add -f ~/.ssh/id_rsa -k KeyNameknife joyent key delete KeyName■ Passphrase protected keys are annoyingEach API request includes data signed with the privatekey. Ruby does not have a good way of signing privatekeys with ssh-agent.

19.
Creating servers!Proprietary andConfidential■ See what images are availableknife joyent image listcf7e2f40-9276-11e2-af9a-0bad2233fb0b base64 1.9.1 smartosf4bc70ca-5e2c-11e1-8380-fb28785857cb smartosplus64 3.1.0 smartosda144ada-a558-11e2-8762-538b60994628 ubuntu-12.04 2.4.1 linux■ base / base64 — minimal install, you add whatyou need■ smartosplus — many more things pre-installed, but can get in the way13328c9a-9173-11e2-a9a5-2ff43d306c21 ws2008ent-r2-sp1 2.0.2 windows

20.
Creating servers!Proprietary andConfidential■ See what flavors are availableknife joyent flavor listName RAM Disk SwapExtra Small 512 MB 0 GB 15 GB 1 GBSmall 1GB 1 GB 30 GB 2 GBMedium 2GB 2 GB 60 GB 4 GBMedium 4GB 4 GB 120 GB 8 GBLarge 8GB 8 GB 240 GB 16 GBLarge 16GB 16 GB 480 GB 32 GB■ Custom networking can be done in a customflavor (ie public or private VLAN, routes)

21.
Creating servers already!Proprietary andConfidentialknife joyent server create--image cf7e2f40-9276-11e2-af9a-0bad2233fb0b--flavor Medium 2GB-N server.domain.com-E environment-d distro-r run_list■ No Omnibus, so you have to provide your owndistro bootstrap templatehttps://gist.github.com/sax/5457464

22.
knife joyent server listSee whats there...Proprietary andConfidentiala597a3a7-3fdf-481f-af08-e7c1e0ae7dca admin.prod running smartmachinesdc:sdc:base64:1.8.1 8.19.1.1 10.100.1.1 8 GB 240 GB5c066e6e-8af2-4d4f-a81e-c8e2691ae8a0 demo.dev running smartmachinesdc:sdc:base64:1.8.1 10.12.1.1 165.225.1.1 8 GB 240 GBb3370d52-3bed-462e-857a-e17eba15ab06 app010.c1.prod running smartmachinesdc:sdc:base64:1.8.1 10.100.1.2 165.225.1.2 8 GB 240 GB■ ID / zonename■ Name■ Run state■ Type■ Image■ IP addresses■ RAM■ Disk

23.
Other managementProprietary andConfidentialknife joyent server delete <server_id>knife joyent server start <server_id>knife joyent server stop <server_id>knife joyent server reboot <server_id>knife joyent server resize <server_id> -f <flavor>knife joyent snapshot create <server_id> <snapshot_name>■ Snapshots are full ZFS snapshotsCopy-on-write snapshot of local file system.Each snapshot is locally mounted in zone at/checkpoints

24.
So now you have asmartmachine...Proprietary andConfidential

25.
Whats different?Proprietary andConfidential■ Things you expect in /usr/local are in /opt/local■ For historical reasons■ If youre used to Linux, this can be annoying■ Joyent is working on a more Linux friendly image■ For now, add /opt/local/bin to PATH■ Many configs are in /opt/local/etc instead of /etc■ Some utilities are different■ This is not the grep youre looking for....■ Symlink your "correct" version into /opt/local/bin■ Add /opt/local/lib to CFLAGS and LDFLAGS

26.
Caveats?Proprietary andConfidential■ Zones inside of zones inside of...■ Vagrant does not currently work with SmartOS■ VirtualBox only works in Bridged network mode■ Local integration tests do not work

27.
Where are all the things?Proprietary andConfidential■ Services■ svcs -a■ svcadm < enable | disable | clear > service■ Packages■ pkgin search packagename■ pkgin -y install packagename

28.
Public vs. Private IPProprietary andConfidential■ ipaddr_extensions gem■ Adds privateaddress attribute to ohai■ Useful to add this to bootstrap■ Smartmachines may have a public IP and aprivate IP■ Recipes can be configured to use ipaddress orprivateaddress

29.
System preparationProprietary andConfidential■ smartos cookbook■ https://github.com/modcloth-cookbooks/smartos■ fixes chef providers■ smartmachine_functions■ links nicer utils into /opt/local/bin■ https://github.com/higanworks-cookbooks/smartmachine_functions■ fixes chef providers■ provides access to Joyent metadata APIor

30.
Useful LWRPsProprietary andConfidential

31.
SMFProprietary andConfidential■ https://github.com/modcloth-cookbooks/smf■ Chef knows how to use SMF, not how to configure it■ Uses nokogiri, which requires libxsltsmf postgres douser postgresgroup postgresproject postgresstart_command postgres-service.sh startstop_command postgres-service.sh stopworking_directory /var/pgsql/dataenvironment PATH => /opt/postgres/binend

32.
SMF (cntd)Proprietary andConfidentialsmf postgres douser postgresgroup postgresproject postgresstart_command postgres-service.sh startstop_command postgres-service.sh stopstop_timeout 120restart_command postgres-service.sh restartrefresh_command postgres-service.sh reloadworking_directory /var/pgsql/dataenvironment PATH => /opt/postgres/binendservice postgres dosupports :status => true,:restart => true, :reload => trueend

33.
Resource Control /Proprietary andConfidential■ https://github.com/wanelo-chef/resource-control■ configure max file descriptors, shared memory, etc■ Bunch up master/worker processes to view inprstat -Jresource_control_project "postgres" docomment "PostgreSQL 9.2"users "postgres"project_limits "max-shm-memory" => 12000000,"max-lwps" => 6process_limits "max-file-descriptor" => {"value" => 32768, "deny" => true}action :createend

34.
Role Based Access ControlProprietary andConfidential■ https://github.com/modcloth-cookbooks/rbac■ Allows delegation of authority without sudo■ Implementation currently too simple, only useful for SMFdelegationrbac solr douser waneloaction :add_management_permissionsend

35.
Contributing to cookbooksProprietary andConfidential■ ~95% just require SMF, correct package names■ ~5% of those need a special init script■ The rest usually require custom compile`postgres -D /path/to/data` not granular enough`pg_ctl -D /path/to/data < start | stop | reload | refresh >`--with-libraries=/opt/local/lib--with-includes=/opt/local/includeLDFLAGS=-R/opt/local/lib -L/opt/local/lib

36.
Comments? Questions? Findme.https://github.com/wanelohttps://github.com/wanelo-chefhttps://github.com/wanelo-chef/smartos-chef-repoProprietary andConfidential@sax @ecdysone @sax