This is a 90 min talk with some exercises and discussion that I gave at the DHS Agile Expo. It places DevOps as a series of feedback loops and emphasizes agile engineering practices being at the core.

1. DevOps
Boldly Go Where You
Never Have Gone Before…
Paul Boos

2. DevOps: Episodes
A Tale of Perspectives
Exploring New Worlds: The DevOps System
Feedback & Learning
• Inner System
• Outer System
Achieving Warp
Star Trek and its images are
owned by Paramount Pictures

3. Meet Mr. Spock
Mr. Spock develops software.
He wants to develop software his users
will love find fascinating.
To him, it’s creative problem solving.
(How logical.)
He wants to deploy it often so his users
can be more productive.
He wants tools and techniques to help
him accomplish these things.

4. Meet Mr. Scott
Mr. Scott runs systems.
He wants the systems to run reliably.
Problem-solving almost always means
bad news for him.
Users usually do not love it when he
has to do problem-solving.
Fast deployments hopefully don’t
mean more work and outages.

5. When things don’t go well…
CAPT Kirk, the CIO, shows more emotion
than Mr. Spock and Scotty would like.
You’d thing the Klingons had taken his son
or something…

6. So how do we unite these two
departments successfully?

7. Incoming Transmission Captain…
DevOps is all about increased and improved
feedback:
oTechnical
and
oBusiness
Starfleet Out…

8. Starfleet Academy Exercise #1
Make a list of feedback loops you think you would
want supported by DevOps
Next to those, list the technique you are currently
doing that helps achieve that feedback loop, note
how long it takes
Pair up, discuss these and see what the other is
doing and how long it is taking them
5 min
5 min

9. Debrief

10. Set Course for
DevOps
(Warp Factor 5)
Story in Ready State

11. DevOps System Survey
oA planetary system consisting of 10 planets
in a variety of feedback loops
oInhabited by a civilization w/many
sophisticated practices
oEach planetary system
has a set of advanced
technological bases

12. DevOps System Survey (cont’d)
oInner system has a lens on fulfilling what is
requested.
oThese are focused on agile engineering practices.
oOuter system lens is on learning what is
actually needed.
o Occur as operations
involvement
increases.

13. DevOps-1 Planetary Scan:
Unit Tests
oFeedback loop often developed to ensure one is
meeting a story’s acceptance criteria
oParticularly useful if performing TDD as one writes
code to pass the test
oAn assurance on the
correct functionality
on our local machine.

14. What did our away team learn?
The technical base for unit testing is an appropriate
unit test framework: jUnit, nUnit, pyUnit, etc.
This is code. It should be treated like any other
code and checked into the source code repository.
This is your safety net for refactoring or adding
features.
What is the orbital period of this feedback loop?
≈ minutes to a couple of hours

15. DevOps-2 Planetary Scan: Sync to
∆’s
oBefore we check-in, update to ensure everything
works with other people’s changes.
oCode hoarding tends to create problems as the
long cycles make it more likely a problem was
created.
oIf done often, this
helps prevent
broken builds.

16. What did our away team learn?
Blindly checking in code increases the integration
work required on the many.
Every piece of code not checked in is inventory at
risk: unit tests, code meeting unit tests…
This tension is what creates success as it helps us
focus on small batches of work.
What is the orbital period of this feedback loop?
≈ minutes to a couple of hours

17. DevOps-3 Scan:
✓- in to Build
oWhen we check-in our code, we get the immediate
feedback of whether a problem was introduced.
oWe want to check directly into a trunk; no
extended branches & merges except for
architectural experiments.
oIf the build breaks,
fix immediately.
(That’s you in case it
wasn’t clear.)

18. What did our away team learn?
If code worked locally, but breaks the build, it was OUR
code that broke it. Fix it in collaboration with the other
team members.
Code hoarding (code not checked in) hurts our ability to
integrate easily. Again small batches of work help us.
We’ll need build scripts (e.g. ant for Java) for the CI
server to use.
What is the orbital period of this feedback loop?
≈ minutes to a couple of hours

19. DevOps-4 Planetary Scan:
Automated Tests on ✓- in
oUpon code check-in (which includes our tests), our
next line of feedback comes from the collection of
automated tests that have been created.
oThese tests act as our safety net; similar to what
we know as regression tests.
oIf the tests fail, fix
immediately.
(That’s you in case it
wasn’t clear.)

20. What did our away team learn?
If code worked locally, but then fails tests on check-in, it
was OUR code that caused the failure. Fix it in
collaboration with the other team members.
This often happens when we failed to synchronize with
the source code before a check-in. Again, code hoarding
works against us, small batches help us.
Repeat CodeUnit Test(locally)SyncCheck-in cycle.
What is the orbital period of this feedback loop?
≈ minutes to a couple of hours

21. DevOps- 5 Planetary Scan:
Automated Acceptance Tests on ✓- in
oFollowing automated unit tests is automated
acceptance tests (behavior tests).
oThese ensure we built what was collectively
articulated within the user story and its
acceptance criteria.
oIf the tests fail, we
haven’t finished
fulfilling the story.

22. What did our away team learn?
Behavioral code is fulfilling the user story. We build to
this need.
If we have difficulty, remember the Agile value:
customer collaboration > contract negotiation.
When these tests pass green, we are basically ready to
head to the outer system and exchange with the
outside. (There’s one more waypoint…)
What is the orbital period of this feedback loop?
≈ generally one hour to a day or so

23. DevOps- 6 Planetary Scan: Exploratory &
Accessibility Testing
oExploratory Testing is looking for edge cases and
additional cases that perhaps should become
acceptance criteria. This manual testing is interleaved
to help design additional automated tests.
oAccessibility (508) Testing should be automated
where possible.
oThese test activities
can be performed in
parallel.

24. What did our away team learn?
Exploratory testing is a means of uncovering more complete and
better unit tests. We want skilled test engineers (red shirts).
Accessibility (508) testing can be automated, and it will be very
different than what is typically seen. Requires excellent test design.
Much of the negotiation with the product owner may occur with
these.
There should be a final check before going to the next [production-
like] environment (the outer system).
What is the orbital period of this feedback loop?
continuously

25. DevOps (Agile Development)
Workflow
✓- Out Code for Work
Write Code &
Pass Unit Test
Sync from
Repository
✓ - In & Build
Automated Tests Run
(Unit Test Suite)
Repeat until BDD
Tests Pass
Automated
Behavior-Driven
Tests Run (BDD Suite)
Repeat until all
Unit Tests Pass
Exploratory/Accessibility
Testing
Ready
Story
to the
Outer
System
Unit Test
Loop:
Minutes
To Hours
Behavior
Loop:
Few Hours ± Day
This is continuous!
Consider a last waypoint
before further journeying.

26. What ‘tools’ do we need
onboard to exchange with
external cultures?
Before we head to the outer system…

27. Source Code Repository –
distributed system or
supporting optimistic locking
(Git, Subversion, Vault, etc.)
Unit Test Framework
(Language Dependent)
Behavior Testing Tools (e.g.
Cucumber, Lettuce, Fitness,
etc.)
UI Testing Tools – these need
to be able to be integrated
into the CI pipeline (e.g.
Selenium Web Driver, etc.)
Test-Oriented
Code Management
Documentation ‘Production’
(Language Dependent &
Independent – e.g. javadoc,
pydoc, etc. + UML reverse-
engineering tools)
Static Analysis Tools – examples:
• Coding ‘standards’ tools
• Cyclomatic Complexity Analysis
Other Useful Tools
Continuous Integration (CI)
Server (Jenkins, Bamboo,
Microsoft Team Foundation
Server, etc.) w/build scripts

28. Surprise!
You’ll be writing code to enable most of
these…

29. What Difficulties Have
You Faced to This
Point Spock?

30. DevOps-7 Planetary Scan:
Performance & Security Testing
oIn preparation for production, often performance
and security testing will be used to look for
weaknesses.
oThese help ensure system integrity when used by
actual customers.
oThese tests should be as
close as possible to the
development work
(w/in an iteration).

31. What did our away team learn about
performance testing?
Performance testing requires an environment that
can mimic production.
• Architecture, servers, data, etc.
Tooling depends on the type of application; for web
apps, tools such a Selenium Grid or Jmeter can be
used for load testing with Selenium Web Driver.
What is the orbital period of this feedback loop?
from within iteration to a release hardening period
(shorter is better)

32. DevSecOps
Incorporating Security Practices into DevOps

33. What did our away team learn about
security testing?
There are many types of security tests:
Penetration, vulnerability, etc.
Tools are dependent on the type of testing and sometimes language
Tools like Chaos Monkey can also be used for cloud-like environments
Hardened server images can also be created for use by teams
for their deployments; separates concerns.
Many security tests can be incorporated into behavior tests
through user|dark stories & acceptance criteria.
What is the orbital period of this feedback loop?
from within iteration to a release hardening period
(again shorter is better)

34. ark Stories
• Basically sets of behavioral tests for
this persona…
• Objective is to have all stories FAIL
• Persona is ‘hostile’:
internal or external

35. Integrating Security in a DevOps
Fashion
Coding Deployment
Dev-Test
Environment(s)
Stage
Environment
Deployment
Production
Environment
Penetration/Vulnerability Testing

36. DevOps-8 Planetary Scan:
Automated Deployments
oAutomated deployments ensure stable, repeatable
configurations in an environment.
oNeeds to include an automated rollback strategy.
oTreat infrastructure as code.

37. What did our away team learn?
Treating infrastructure as code requires versioning; thus some
form of repository.
• The continuous delivery fans would say these must be binary images.
• I’ve had success using a CI server and building through to production.
Besides the images, the deployment (and rollback) scripts
should also be versioned.
It’s usually a good practice to have an automated smoke test
following a completed production deployment. Having QA
decision points along the entire build/deploy pipeline.
What is the orbital period of this feedback loop?
when deployment occurs -- often

38. Comparing Deployment Approaches
Deploying Binaries
Pluses
• Faster
• No compilation is necessary
• Easily can avoid downtime
Minuses
• Deployment scripts are
different than build scripts
• More infrastructure is
required
Building Through
Pluses
• Build scripts to every
environment are the same
• Compile to each server
Minuses
• Need to clean-up any build
tools used in production
• Slower to build
• More difficult to avoid
downtime

39. DevOps-9 Scan:
Product Owner/Business Review
oProduct Owner/business ‘approval’ is still desired.
oFrequently the Product Owner or business is tugged
by the gravitational pulls of other stakeholders.
oReview can occur at any
point in the process &
when in any
environment.

40. What did our away team learn?
IT/App Dev does work for the business’ value stream.
Report what passes ‘green’ and what FAILed.
Reviewing with the Product Owner/business can
occur post-Production deployment. How?
• Using toggles coded into the configuration.
• Toggles turn on/off features around stories, epics, or other
groupings of features
What is the orbital period of this feedback loop?
whenever the business desires

41. DevOps-10 Planetary Scan:
Customer Monitoring
oEverything the business says they want on behalf of
a customer-user is a hypothesis.
oIts important to find out what works (or doesn’t).
oMore frequent is
better.
oThis is where the
business learns.

42. What did our away team learn?
Obvious ways to learn from customers:
• Customer surveys
• Net Promoter Score
Less obvious ways:
• Periodic focus groups
• Flows through the system with dwell times, conversions, and
abandoned sessions
• Heat maps of where users click
What is best is entirely dependent on what the application or
system is supposed to do for them. Can cause a rollback!!
What is the orbital period of this feedback loop?
every release or ‘toggle on’

43. Load Testing – multiple
browsers (& browser
engines) simultaneously
(Selenium Grid, Jmeter, etc.)
Flow/Conversions/Abandoned
Sessions (Google Analytics,
Matomo, W3Counter,
MixPanel, Kissmetrics, etc.)
Click Tracking (Open Web
Analytics, ClickHeat, etc.)
Customer Surveys (Survey
Monkey, Foresight, etc.)
Customer Monitoring
Performance & Security Testing
Test Portal/Orchestration
(SonarQube, etc.)
Binary Source Management
(Artifactory, Nexus, etc.)
Deployment Configuration (Chef,
Puppet, etc.)
OS Virtualization/Containers
(Docker, Vagrant, Apache Mesos, etc.)
Pipeline-Oriented
Penetration Testing (Metasploit
Framework, etc.)
Vulnerability Scanning (Nmap,
OpenVAS, etc.)

44. What tool have I not
mentioned?

45. Starfleet Academy Exercise #2
Get into a group of 4-6 (preferably from the same
component); create a mindmap of the types of quality
(e.g. reliability, data integrity, performance)
Now create a mindmap of the stakeholders that care
about quality
And add the quality types to stakeholders
You can now use this technique (even at a story level) to
determine the type of testing to emphasize for
development work in your pipeline.
• Emphasize by stakeholder importance
• Improvement stories can be used to address future needs
5 min
5 min
Since automated testing is crucial for DevOps…

46. Debrief

47. A glimpse
into the
future

48. Captain’s Log Stardate 10.23.2028
Scaling DevOps
• Cadenced delivery  review at scale gives way to continuous
flow of value
• Learn more about scaling with the Manifesto for Scaling Agility
Artificial Intelligence/Machine Learning
• Helps make decisions on criticality of errors and whether to
promote code or not
Refocus on the Human-Side of Agile
• Source Code Repositories Report who has code already checked
out and assist in loading a virtual pairing session
• We realize Scrum Masters (team facilitators) can’t be replaced
with AI, but is an essential skill all team members should have

49. Back to
Now…

50. Captain’s Log Stardate 10.23.2018
Review what feedback loops you want to improve
Write down the ONE next loop that you
immediately want to improve.
Also describe how you will collaboratively work
with the others to make this happen. 5 min

51. Red Alert :: Conference Identified
13-14 November
AMERICAN UNIVERSITY
WASHINGTON COLLEGE OF LAW
4300 Nebraska Ave NW

52. Red Alert :: 2nd Conference Identified
http://glasscon.us/
Government Lean-Agile
Software & Systems Conference

53. Thank You!
Paul Boos
paul.boos@excella.com
@paul_boos

54. Any Questions?

55. Enjoy Your
Voyage
Paul Boos
paul.boos@excella.com
@paul_boos