Slides from my talk at the GroupOn office in Palo Alto, CA. http://bit.ly/X4NDxt

1. Continuous Improvement
How Continuous Delivery is changing Quality Assurance.
GroupOn
Palo Alto
January 15 2013
Noah Sussman
ns@noahsussman.com
@noahsussman

2. The canonical Agile release cycle
film still from The Lord of the Rings

3. Sprints of two or more weeks in length
Cocento Tecnologia on Flickr

4. Start deployment once the sprint is over
TheMasonDixon on Etsy

5. QA is part of the release process
SisterDimension on Flickr

6. QA sign-off is required before going live
film still from The Lord of the Rings

7. The Continuous release cycle
evoo73 on Flickr

8. Minimum viable feature set
Travis S. on Flickr

9. Releasing a feature
is decoupled from
deploying code.
David E. Smith on Flickr

10. An airport without
an air traffic controller.
—Chad Dickerson

11. Real-time data on how releases impact revenue
Etsy

12. Default to open access

13. Constant tweaks to live features

14. Large features are deployed piecemeal over time
dogpose on Flickr

15. Every feature is part of an A/B campaign
NASA

16. Dark launches
Joy and Jon

17. Opt-in experiments
NASA

18. Partial rollouts
NASA

19. Config Flags
Wikipedia

20. Wire-Offs
Joe Thomissen on Flickr

21. if ($cfg["new_search"]) {
// new hotness
$resp = search_solr();
} else {
// old busted
$resp = search_grep();
}

22. $cfg = array(
'checkout' => true,
'homepage' => true,
'profiles' => true,
'new_search' => false,
);

23. There is no “done done”
NASA

24. Observed Behavior Of Complex Systems
NASA

25. Emergent behaviors require unplanned responses
NASA

26. Improvements are discovered rather than designed
NASA

27. Users of the system have complex expectations
NASA

28. Complex systems are never “complete”
NASA

29. QA Happens When?
NASA

30. First of all, what is “Quality Assurance?”
NASA

31. QA: assuring that there are no defects?
NASA

32. It is impossible to prove the absence of defects
NASA

33. There will always be bugs in production
Lukjonis

34. Testing is everyone’s job.
Library of Congress

35. Myths About Bug Detection
The Jargon File

36. Myth: there are a finite number of bugs

37. Myth: here are a finite number of detectable bugs
niscratz on Flickr

38. Myth: all severity one bugs can be found before release
NASA

39. Myth: software is built to specifications
Fred Brooks at Etsy

40. Myth: at some point, software is finished

41. Myth: most bugs have complex, unpredictable causes

42. The animistic metaphor of the bug
that maliciously sneaked in while the
programmer was not looking...
disguises that the error is the
programmer's own creation.
— Edsger Dijkstra

43. The whole time I’m programming, I’m
constantly checking my assumptions.
—Rasmus Lerdorf
@loriabys

44. As you're about to add a comment, ask
yourself, “How can I improve the code
so that this comment isn't needed?”
Improve the code and then document
it to make it even clearer.
— Steve McConnell

45. Debugging is twice as hard
as writing the code in the
first place. Therefore, if
you write the code as
cleverly as possible, you
are, by definition, not
smart enough to debug it.
—Brian Kernighan

46. No blame

47. Many Small Anomalies Combined

48. An organization's defenses against failure
are a series of barriers, represented as
slices of swiss cheese. The holes in the
cheese represent weaknesses in
individual parts of the system. Failures
occur when a hazard passes through all
of the holes in all of the defenses.
— Wikipedia

49. John Allspaw

50. Prioritize the elimination of small errors
John Allspaw

51. Focus less on mitigation of large, catastrophic failures

52. Optimize for
recovery rather
than failure
prevention.
Failure is
inevitable.
Richard Avedon

53. Unit testing is great for preventing small errors
John Allspaw

54. Resilience, Not “Quality”
John Allspaw

55. Readable code
NASA

56. Reasonable test coverage
NASA

57. Sane architecture
NASA

58. Good debugging tools
NASA

59. An engineering culture that values refactoring
NASA

60. Measurable goals
NASA

61. Manual Testing.
But probably not the kind you’re thinking of.
NASA

62. Real-Time Monitoring is the new face of testing
NASA

63. Etsy

64. Etsy

65. Etsy

69. Anomaly detection is hard.
Greg and Tim Hildebrandt

70. Watching the graphs
NASA

71. As of 2012, Etsy collected well over
a quarter million real-time metrics
NASA

72. Deciding which metrics matter is a human problem
NASA

73. Everyone watches some subset of the graphs
NASA

74. Human vision is an excellent tool for anomaly detection
NASA

75. QA happens when???
NASA

76. Exploratory testing can be performed at any time
NASA

77. Rigorous, scientific approach
NASA

78. Focus on customer satisfaction
NASA

79. Less focus on product specifications
NASA

80. Exploratory Testing is equally useful
before or after a release
NASA

81. Just Quality
NASA

82. “Assurance” is a terrible word.
Let’s discard it.
NASA

83. Quality exists, but it’s tricky to assure or prove that
NASA

84. There’s no such thing as a formal proof of quality
NASA

85. Most of us would agree that quality exists
NASA

86. “Customer Experience” is a better term of art than “Quality”
NASA

87. Customer Experience.
Though there’s no formal proof for that, either.
NASA

88. Exploratory Testing addresses areas that Developer Testing doesn’t
NASA

89. Developer Testing validates assumptions
NASA

90. The Independent Tester’s job is to invalidate assumptions
NASA

91. Technology Informs Customer Experience
NASA

92. Exploratory Testing requires an understanding of the
whole system
NASA

93. Exploratory Testing requires understanding how the
system serves a community of users
NASA

94. Customer Experience is as much about technology as it
is about product requirements
NASA

95. Most bugs, most of the time, are easily
nailed given even an incomplete but
suggestive characterization of their
error conditions at source-code level.
— Eric S. Raymond

96. Source, diffs, logs.
If your QA Analysts don’t look at these — teach them.
NASA

97. Customer Support
NASA

98. Your customer support operators spend more time
talking to your users than anyone else
NASA

99. Customer Support interface with users as individuals
rather than as aggregate data
NASA

100. Keep the feedback loop short.

101. Manage Your Culture.

102. Effeciency To Thoroughness Trade-Off
NASA

103. Rapid release cycles have different risks than
slower release cycles
NASA

104. Continuous Delivery does not alter the fundamental
nature of risk
NASA

105. Test in both dev and prod
NASA

106. Detectable errors should be caught in dev
NASA

107. Undetectable errors must be worked out in production
NASA

108. Software exists in context
NASA

109. Networks, services and people are always in flux
NASA

110. Small changesets are easier to debug

111. An SCM revert is a changeset
NASA

112. Large changesets are riskier and harder to debug
NASA

113. Fail Forward!
NASA

114. Always deploy the HEAD revision of trunk
scrapnow on Etsy

115. Never roll back to an earlier state.
Always roll forward. When it's
desireable to revert a previous
change, do that as part of a new
commit.

116. Instead of rolling back, fix the problem and move on
NASA

117. Let go of the idea of “last stable release”
NASA

118. Focus less on satisfying the requirements
Scott Holloway

119. Watch the graphs
NASA

120. Listen to your customers
NASA

121. Build a culture of shared responsibility
Kirsten Dunst on the set of Marie Antoinette

122. Low-Ceremony Process
Kirsten Dunst on the set of Marie Antoinette

123. Iteratively improve your product
WSHS Science blog

124. Further Reading
“How Google Tests Software,” James Whittaker
“Look At Your Data,” John Rausser
“Optimizing For Developer Happiness,” Chad Dickerson
“Outages, Postmortems and Human Error,” John Allspaw
http://en.wikipedia.org/wiki/Swiss_cheese_model
“What Is Exploratory Testing?,” James Bach

125. Questions?
@noahsussman
ns@noahsussman.com
infiniteundo.com