When most people talk about automating infrastructure, they focus on things like consistency, scalability, and flexibility. While fine goals, we recently converted several projects to Chef for both systems AND application deployment, and found that, with a little work, these tools could also help you enable better software quality assurance, load modeling, and even improve resource allocation. By sharing cookbooks across projects, we were able to standardize practices and eliminate arbitrary differences, while using parameterization to perfectly isolate the special needs of each project. This allowed us to transfer knowledge among staff much more quickly. Pulling in and parameterizing application state – database contents, website assets, uploaded content – allowed us to spin up new environments with as much or as little state as needed. Integrating with Vagrant and Jenkins, we were then able to use chef to treat the entire image – system and application – as a test fixture. As each engineer (ops or dev) has visibility into the whole stack, we can more easily move people between dev and ops, or between projects.

1. The Unintended
Benefits of Chef
Standardization FTW

2. whoami
Clinton Wolfe, Web Architect @ OmniTI
mod perl developer since 1996
cheffin' since 2012
@clintoncwolfe
github: clintoncwolfe
LEGO group

3. $JOB
omniti.com
custom web development - UX to hosting
we make OmniOS, an Illumos based OS distro

4. our dev clients
mostly brownfield work
new features
oh no it won't scale
my nephew built it

5. client scale
some have hundreds of nodes
most clients have < 10 nodes
some have many, want fewer
some have few, need more
we specialize in doing
more with less
mijasper/brickshelf

6. process : anti-process
each customer has "unique" problems
some are, most aren't
we're flexible, urging gradual change

7. TMTOWTDI....
dev environments
asset state
cron
self-referential urls
CPAN/PECL/gem/egg/npm builds
“You Only Deploy Once!"

8. clients: YODO!!!
except for disaster recovery

9. clients: YODO!!!
except for disaster recovery
except for onboarding devs

10. clients: YODO!!!
except for disaster recovery
except for onboarding devs
except for CI

11. clients: YODO!!!
except for disaster recovery
except for onboarding devs
except for CI
except for scaling out

12. clients: YODO!!!
except for disaster recovery
except for onboarding devs
except for CI
except for scaling out
except for load testing

13. clients: YODO!!!
except for disaster recovery
except for onboarding devs
except for CI
except for scaling out
except for load testing
except for upgrade testing

14. clients: YODO!!!
except for disaster recovery
except for onboarding devs
except for CI
except for scaling out
except for load testing
except for upgrade testing
except for datacenter moves

15. QA is hard
can't make new QA environments
QA env is precious
state hard to manage
If testing is hard,
no testing is done
mijasper/brickshelf

16. surely we can do better
We want standardization
Client wants deployability (scale-out, resilience)
Chef FTW

17. The Plan!
Focus on deployability
Integrate systems and app deployment
Custom Recipes – choosing your battles
Automate testing early

18. cheffing as discovery
Exploring the existing app for on-boarding
Even rsyncing a golden
image is OK
capture it in chef
Get deployable (in Vagrant ) ASAP
mijasper/customminifigs.co.uk

19. The Plan!
Focus on deployability
Integrate systems and app deployment
Custom Recipes – choosing your battles
Automate testing early

20. app deployment in chef?
most app deployment tools partially dictate
layout, push process, etc
who knows what we have been handed
but chef can do pretty much anything!

21. cheffing an app
identify and decouple roles
setup user auth, systems stuff
code checkout
templatize config files
service definitions
Iterate, standardize, simplify!

22. The Plan!
Focus on deployability
Integrate systems and app deployment
Custom Recipes – choosing your battles
Automate testing early

23. finding variances
understand existing variances
standardize the arbitrary
justify the ones that need to remain
writing a custom
cookbook is a red flag
jokeith/brickshelf

24. Our shared cookbooks
Over 20 cookbooks used internally at the
company, cross-client
Attribute-driven default recipes
Exposes project variances in clear relief... yet
we can easily extend them if needed
Shared roles/handlers/databags, too

25. it's still OK to punt
We will never be fully standardized
As old projects mature, new projects arrive
known technical debt
can be healthy
mijasper/brickshelf

26. The Plan!
Focus on deployability
Integrate systems and app deployment
Custom Recipes – choosing your battles
Automate testing early

27. cheffing: ci with jenkins
static checks on chef config
scratchbuild Vagrant runs
parametric builds and Vagrantfiles
vagrant-test-subject
mijasper/brickshelf

28. system tests
High-level, near-english behavioral tests
Not low-level unit tests against codebase
Supposed to be serving http? Check for it!
Easy to add more later - start with minimum

29. aside: our test harness
vagrant-rspec-ci gem
provision ok?
services running?
connect to port?
Capybara for web UI testing
Junit output for pretty Jenkins graphs

30. Test spec example
describe "TrafficServer Service" do
before(:all) do
@vm = VagrantTestSubject::VM.attach()
end
it "should appear as a healthy service" do
@vm.should have_running_service("trafficserver")
end
it "should be listening on external_ip:80" do
@vm.should be_listening_on_external_ip(80)
end
it "should be the right process name on port 80" do
process = @vm.process_name_listening('127.0.0.1', 80)
process.should match(//opt/ts/bin/traffic_manager/)
end
it "should respond with HTTP 200 to / on port 80" do
@vm.http_get('/').should be_http_not_found # from rspec-http gem
end
end

31. Shiny, Happy Teams
Ops Team Impact
Dev Team Impact
QA Team Impact
Organizational Impact
LEGO Group

32. Ops Team Impact
Chef everywhere! YAY!
Chef for everything, especially wildly divergent
app deployments! BOO!
Direct exposure to dev team deployment needs
& tooling... a more devvy ops team

33. Dev Team Impact
Can get a new dev env anytime, have more
than one - YAY
Same build everywhere – YAY
I have to learn ruby? BOO
Some awareness of practices on other projects,
improved perspective
Much more visibility into how systems are
provisioned – a more opsy dev team

34. QA Team Impact
Use vagrant VMs as test subjects
QA VM provisioning can now be elastic,
responsive to testing/release cycles
Dev and QA exposure to systems BDT
Dev and ops teams more QA-y: devopsqa!

35. the VM as fixture
data loads - vary by test scenario
data loads - vary for scale
asset loads
role permutations
os, vm, env permutations
clusters of related machines (db, web, CDN,
client)

36. aside: loading fixtures
We use Chef roles to define attributes that load
fixture state
We use recipes to read the attributes and
converge to the needed test state
•special purge-all role clears all state
•use env var to pass fixture list into vagrant for
provision

37. i accidentally the whole db
fixture loading must be
used with care
use a safety flag to
enable it
Nelson Yrizarry/brickshelf

38. Organizational Impact
Standardization eases:
actual deployment
on-call ops can support the app more
devs responsible for deployability
ops can participate in app management
reduced ramp-up time onto existing projects

39. Thanks!
Clinton Wolfe
clinton@omniti.com
findmybrick.com