The past few years have seen a phenomenon of software organizations abandoning traditional release cycles in favor of daily or even hourly deployments. The emergence of a new, rapid software development workflow has raised questions regarding the role of test and QA in a product's life cycle. When there is no QA phase, can there still be QA? Indeed, as the speed of product development changes, risk must be assessed differently. A monthly release cycle provides time to assure a product is reasonably free of defects. In a rapid release cycle, such assurance is impossible. Quality instead comes to mean assuring that the product is capable of recovering from defects. I will discuss how software risk mitigation practice changes as the speed of development increases. And I will explore the idea that recovering from failure is a far more pragmatic goal than preventing failure in the first place.

1. Continuous
Improvement
How rapid release cycles
alter QA and testing.
Noah Sussman
DevopsDay Mountain View, 2012
@noahsussman
#devopsdays

2. The Canonical Agile
Release Cycle
Sprints of two weeks or more in length.
Start deployment process at the end of the sprint.
QA is part of the deployment process.
QA must be complete before new code goes live.

3. The Continuous
Release Cycle
Minimum viable feature set.
Deployment is decoupled from release.
Real-time data on how releases impact revenue.
Constant tweaks to live features.

4. Releasing a feature
is decoupled from
deploying code.
David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122

5. An airport without
an air traffic controller.
—Chad Dickerson

6. This Part Really Is
Different From Agile
Large features are deployed piecemeal over time.
Every feature is part of an A/B campaign.
Dark and Admin-Only launches.
Wire-Offs and Config Flags.
There is no “Done Done.”

7. Observed Behavior
Of Complex Systems
Emergent behaviors require unplanned responses.
Improvements, too are discovered not designed.
Users of the system have complex expectations.
Such systems are never “complete.”

8. QA Happens When?
First of all, what is “Quality Assurance?”
Authoritatively assuring that there are no defects?
That’s impossible.

9. Testing is everyone’s job.

10. Myths About Bug
Detection
There are a finite number of bugs.
There are a finite number of detectable bugs.
All severity one bugs can be found before release
Software is built to specifications.
At some point, software is finished.

11. The Biggest Myth
Bugs have complex, unpredictable causes.
In fact, most errors in software are the results of
incorrect assumptions made by programmers.

12. Many Small
Anomalies Combined
The “Swiss Cheese” model of risk presents us
with a strong case for prioritizing the elimination of
small errors rather than focusing on the mitigation
of large catastrophic failures.
Unit testing is great at eliminating small errors.

13. The whole time I’m
programming, I’m
constantly checking
my assumptions.
—Rasmus Lerdorf

14. Resilience, Not
“Quality”
Readable code.
Reasonable test coverage.
Sane architecture.
Good debugging tools.
An engineering culture that values refactoring.
These are measureable goals.

15. Manual Testing
It doesn’t always look like you think it looks.
Real-Time Monitoring is the new face of testing.

22. Anomaly detection is hard.

23. Watching The Graphs
Etsy collects well over a quarter million metrics.
Deciding which ones matter is a human problem.
Everyone watches some subset of the graphs.
Human vision excels at anomaly detection.

24. QA Happens When??
Exploratory testing can be performed any time.
Rigorous, scientific approach.
Focus on customer satisfaction rather than a spec.
Equally useful before or after a release.

25. Just Quality
“Assurance” is a terrible word. Let’s discard it.
Quality exists, we just can’t assure or prove it.

26. There Is No Such
Thing As A Formal
Proof Of Quality.
Yet most of us would agree it exists.
I propose that “customer experience” is a better
term-of-art-than “quality.”
Though there’s no formal proof for that either.

27. Exploratory Testing
Addresses areas that Developer Testing can’t.
Developer Testing validates assumptions.
The Tester’s job is to invalidate assumptions.

28. Technology Informs
Customer Experience
Exploratory Testing requires an understanding of
the ways in which a whole system is intended to
serve a community of users.
The problem space has as much to do with
technology as it does with product requirements.

29. Most bugs, most of the
time, are easily nailed given
even an incomplete but
suggestive characterization
of their error conditions at
source-code level.
—Eric S. Raymond

30. Source, diffs, logs.
If your QA Analysts don’t look at these, teach them.

31. Customer Support
Your customer support operators spend more time
talking to your users than anyone else.
Customer Support interface with users as
individuals rather than as aggregate data.

32. Keep the feedback loop short.

33. Manage Your Culture.

34. Effeciency To
Thoroughness
Trade-Off
Rapid release cycles have different risks than
slower release cycles.
But nothing about risk itself has changed.

35. Test Everywhere
Foreseeable errors can be worked out in dev.
Unforeseeable errors must be worked out in prod.

36. Fail Forward
Let go of the idea of “last stable release.”
Software exists in context.
Networks, services and people are always in flux.

37. Forget About
Satisfying The
Requirements
Watch your graphs.
Listen to your customers.
Adhere to your protocols.
Improve your product.

38. Questions?
@noahsussman
ns@noahsussman.com
infiniteundo.com

39. Further Reading
“How Google Tests Software,” James Whittaker (especially chapter 5)
“Look At Your Data,” John Rausser
“Optimizing For Developer Happiness,” Chad Dickerson
“Outages, Postmortems and Human Error,” John Allspaw
http://en.wikipedia.org/wiki/Swiss_cheese_model
“What Is Exploratory Testing?,” James Bach
“How Many Eyeballs Tame Complexity,” ESR
“The Timeless Way of Building,” Christopher Alexander