Picture the scene. You're trying to come up with a Chef workflow or toolchain for your team or organization. So you start looking at what other people in the field have been doing - but wait! Some folks insist you should use one tool, while others are vehemently against it. Some advocate using a particular feature of Chef while others claim the world will end if you do. What's do be done? How do you decide what *you* should use? In this session I'll help you with strategies and techniques to filter out the signal from the noise and identify tools, workflows and best practices that will fit with your organisational needs. We'll explore some common workflows and toolchains in use today and look at the importance of understanding the motivations behind their design as an aide for validating their potential usefulness for your organization. We'll also examine the importance of grokking the underpinning architecture and characteristics of Chef's design and implementation, and how to incorporate this information into your workflow and toolchain design process. https://youtu.be/lsupJuAkfwQ

1. Workflow Design
Extracting Signal from the Noise

2. @jonlives
Jon Cowie
Staff Operations Engineer

4. @jonlives
There is no correct way to
use Chef.
workflow_design

5. @jonlives
…but I just want to
get stuff done!

6. @jonlives
Absorb what is
useful.
Discard what is
useless.

7. @jonlives
“I am not smart enough to
build an ontology … that
can encompass all the
variations in infrastructure.
Nobody is, the world
moves too fast.”

8. @jonlives
There is no
magic pill.

9. @jonlives
You are the
expert.

10. @jonlives
Beware Argument Culture!
Kate Heddleston: bit.ly/argument-culture

11. @jonlives
[An] argument culture urges us to
approach the world—and the
people in it—in an adversarial
frame of mind
Deborah Tannen “The Argument Culture”

12. @jonlives
Argument Culture
• Useful in “regulated” environments like law
• Engineering is not one of those environments
• Weed out weak logic, keep strong ideas
• Everybody is on one side or the other
• We attack their position and defend our own
• We can win or lose arguments—just like battles

13. @jonlives
Argument Culture - Effects
• Emotional interactions
• Winning becomes paramount
• The loud dominate
• People are afraid to speak
• Best practice == repeating the loudest arguments
• Sucks for diversity in cultures and orgs

14. @jonlives
What a Workflow is…
workflow
noun ( also work-flow) /ˈwɜːkfləʊ/
the way that a particular type of work is
organised, or the order of the stages in a
particular work process
Cambridge Business English Dictionary

15. @jonlives
What a Workflow is not…
• “The Chef Way”
• “We need to use X tool”
• “We need to be more DevOps”
• “We need our process to be more like Y”
• “Z in Chef is bad. Never do Z.”

16. @jonlives
A workflow is the key to
solving a problem.

17. @jonlives
There is no correct way to
use Chef…

18. @jonlives
There are only lessons
learned by others in the
process of solving their
specific problems.

19. @jonlives
Case Studies

20. @jonlives
Case Studies
• Anonymized
• Companies of varying sizes
• First, a quick summary
• Then we look at their motivations
• Finally we look at their solutions

21. @jonlives
Case Study #1 - Summary
• Small internet video company
• Small Ops team
• EC2 based infrastructure
• Semi-immutable infrastructure
• Packer used to build AMI containers
• Chef run on instances to correct config & service status
• Previously required rolling a new AMI

22. @jonlives
Case Study #1 - Motivation
• They’re trying to make it as easy to get started as
possible
• Chat or comments in PRs since everybody is remote
• They aren't interested in letting some third party
change stuff unexpectedly
• No auto-deploying master - they had an accidental
push

23. @jonlives
Case Study #1 - Solution
• Cookbooks live in chef repo
• Dependencies in Berksfile
• “Pull request” development model
• Merge to master & Jenkins runs foodcritic & rubocop
• Merge into “jenkins” branch, triggers upload to server

24. @jonlives
Case Study #2 - Summary
• One project team inside extremely large company
• On-premise cloudstack infrastructure
• VMs created, then bootstrapped with Chef
• Dev, QA, Staging, Prod environments
• Legacy cookbooks maintained alongside new
• 1 workflow for new cookbooks, one for legacy

25. @jonlives
Case Study #2 - Motivation
• Too easy to make an innocuous change that broke
several features without knowing about it
• Desire to catch syntax errors or undeclared attributes
• Complete test coverage with minitest, kitchen-ci for
easier PR validation, and local cookbook development

26. @jonlives
Case Study #2 - Solution
• 1 monolithic project cookbook
• Add dependencies to Berksfile & “berks install”
• Create recipes & write tests
• “kitchen converge” until it works
• Push to git & chef server
• Test in dev, then on to prod

27. @jonlives
Case Study #3 - Summary
• Mid-sized online marketplace
• Mainly on-premise infrastructure
• Servers often long-lived & not immutable
• Large number of Dev & Ops Chef users
• One main environment

28. @jonlives
Case Study #3 - Motivation
• Many Chefs
• Many small changes by often inexperienced users
• Historical cookbooks make using community
cookbooks difficult & time consuming
• App deployed using separate deploy tool
• Continuous delivery in “DNA”

29. @jonlives
Case Study #3 - Solution
• Cookbooks live in monolithic repository
• Roles == attribs, environments == constraints
• Tooling geared for efficiency & iteration vs testing
• Testing done locally with chef-zero & why-run
• …or in prod with whitelists
• Versions always pinned in prod environment

30. @jonlives
Case Study #4 - Summary
• Mid-sized ecommerce service provider
• Mainly on-premise infrastructure
• Multiple product-based teams
• Multiple environments to support those teams

31. @jonlives
Case Study #4 - Motivation
• Teams based around the Products they build and run
• Different opinions & requirements for workflow
• Most effort was spent on new greenfield projects
• Trying to avoid “veneering tech debt”
• Chef was new to most people
• Pressure to produce visible & demonstrable results

32. @jonlives
Case Study #4 - Solution
• 85% of cookbooks in monolithic repo
• Start with community cookbooks
• Wrap as needed
• Team or Product specific environments
• Roles used to set permissions etc
• Optional cookbook pinning

33. @jonlives
So, how do we create a
Workflow?

34. @jonlives
Workflow Design
• What problem are you solving?
• Who are you solving it for?
• How does your ideal solution look?
• Choose your Tooling
• Implement your workflow

35. @jonlives
What Problem are you Solving?
• Write down a simple problem
statement
• “I want to automate app deploys”
• “I want to automate server builds”
• “I want us to test our cookbooks
more”
• Break that down into simple pieces
What
Who
How
Tooling
Implement

36. @jonlives
Who are you solving it for?
• Understand your stakeholders
• Understand your business priorities
• What do *they* care about most?
• How often will they work with your
solution?
• Conway’s law
What
Who
How
Tooling
Implement

37. @jonlives
Conway’s Law
“Organizations which design systems ...
are constrained to produce designs which
are copies of the communication
structures of these organizations”
Melvin Conway, 1968

38. @jonlives
How does your ideal solution look?
• Have an ideal “end goal” in mind
• Agree with stakeholders
• You won’t get there in one go
• But you’ll know what you’re
working towards
What
Who
How
Tooling
Implement

39. @jonlives
Choose your Weapons!
• Pick candidate tools
• Probe strengths and weaknesses
• Understand their motivations
• Extensibility if tweaking needed?
• Is anybody already using it in the
same way?
What
Who
How
Tooling
Implement

40. @jonlives
…or make your own Weapons
• Sometimes, you have to invent
• Legitimate approach
• Many popular workflow tools
started this way
• Be sure you have time / resources
• Clear mission statement is vital
What
Who
How
Tooling
Implement

41. @jonlives
Implementation
• Have realistic expectations
• Start slowly
• Efficiency-Thoroughness Trade-Off
(ETTO)
• Iterate often
• Learn from failures. They will happen.
• Improve
What
Who
How
Tooling
Implement

42. @jonlives
Don’t Fear the Code!

43. @jonlives
Don’t Fear the Code!
• Understanding == Effectiveness
• If you know how it works, you can figure out
why it breaks.
• Research known bugs & limitations
• Be prepared to report or fix issues

44. @jonlives
It’s OK to try stuff!

45. @jonlives
http://jonliv.es/book
Discount Code: AUTHD
40% off Print
50% off Digital

46. @jonlives
Thanks!
Questions?
@jonlives / http://jonliv.es / jon@etsy.com