Top web companies employ continuous delivery of software to build and deploy systems faster and gain a marked competitive advantage. You can do it, too! Andy Singleton shares the patterns for testing in real time that result in more frequent and more reliable releases. He explains why you will have to invest seriously in automated tests and shares experiences developing the most time-efficient types of automated tests, setting up a social structure to get the tests you need, and employing existing layers of testing and production monitoring. Then, Andy goes on to describe the strategies and techniques within the development/delivery process that enable continuous delivery: developer code review workflows; specific ways to use Git, Perforce, and Subversion for version control; continuous integration approaches; and more. Join a discussion on the ways continuous delivery changes the roles of developers and testers and accelerates the value you deliver to your business.

1. AW12
Concurrent Session
11/13/2013 3:45 PM
"Test (and More) Patterns for
Continuous Software Delivery"
Presented by:
Andy Singleton
Assembla
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888 268 8770 904 278 0524 sqeinfo@sqe.com www.sqe.com

2. Andy Singleton
Assembla
Andy Singleton is the founder of Assembla, a SaaS company providing
tools for distributed agile development teams. Andy moved Assembla from
bi-weekly releases to multiple daily releases. His forthcoming
book Unblock! A Guide to the New Continuous Agile describes workflows
for product management, task management, coding, and testing.
Previously, Andy founded PowerSteering Software, a provider of enterprise
project management software, and Cambridge Interactive, an eBusiness
consultancy. He developed financial information systems for Reuters,
Bloomberg, SNL Kagan, and Thomson. During his career, Andy has
produced more than twenty major software and financial information
products.

3. Be#er%So(ware%East%
From%Andy%Singleton,%h#p://andysingleton.com%
www.assembla.com%
%

4. In%this%session%
! IntroducAon%to%CD,%benefits%of%CD,%and%ways%to%pitch%it%
! MulAple%code%and%test%pa#erns%to%choose%from%
! How%to%get%the%automated%tests%you%need%
! Changes%in%the%role%of%QA%
%

5. Waterfall%to%ConAnuous%
Skip%
Plan%
CI%&%CD%
Program%
End%up%with%
Pull%
Automate%
&%Blend%
Test%
Lag%
Doc%
Releases%
Program%
Measure%
Launch%
Automate%
Deploy%

6. Assembla%in%2011%
! “Scrumban”%with%iteraAve%releases,%but%conAnuous%planning%
to%accommodate%a%distributed%team.%
! Releases%took%longer%as%system%got%bigger%and%there%was%more%
to%test.%2%weeks%V>%3%weeks%
! Bugs%in%producAon.%%2%days%for%fixes.%%Stressful)
! CompeAtors%achieved%faster%velocity%with%conAnuous%delivery%
! Made%a%study%of%conAnuous%methods%with%our%own%team,%
customers,%and%tools.%
! Now%–%releasing%about%250%Ames%per%month.%%Fewer%bugs.%%
Much%less%stress.%

7. You%can%do%it%incrementally%
1. Release%more%frequently%
2. Improve%
%
%
%
%
%
%

8. When%to%use%ConAnuous%Delivery?%
! You%provide%an%online%service.%%CompeAAve%pressure%will%
force%you%to%conAnuous:%Office%365%vs%Google%Docs.%
! You%provide%any%service%with%so(ware%inside%
! Your%release%Ames%are%gedng%longer,%or%the%release%
process%is%stressful%
! You%are%developing%a%new%product%with%lean%startup%and%
MVP%techniques%
! You%have%a%big%project%with%a%lot%of%contributors%
! This%applies%to%hardware,%also%

9. Benefits%for%the%test%process%
! Eliminate%stress%from%the%release%process.%%There%is%never%
a%big%batch%of%stuff%to%test%under%pressure.%
%
! Add%or%remove%test%layers%to%increase%quality%or%increase%
speed%
! ConAnuous%integraAon%removes%the%stress%of%acquiring%
and%fidng%components%from%many%teams%and%suppliers%
! Your%CEO%wonders%why%the%compeAAon%is%releasing%faster%
and%moving%faster%
%

10. ConAnuous%Delivery%Dial%
Business
Dev
Ops
Controlled
SelfService
DEV
TEST
INTEG
CONTINUOUS
DELIVERY
STAGING
UAT
PROD
RELEASE TRAIN
From%Steve%Brodie%and%Rohit%Jainendran%

11. The%big%quesAon:%How%to%test?%
! We%release%so(ware%in%batches%so%that%we%can%test%it.%%
That%is%the%whole%reason%for%doing%it.%%We%test%so(ware%%
“release%candidates”%to%make%sure%everything%works%
together.%
! In%conAnuous%delivery,%we%might%get%as%li#le%at%10%
minutes%to%test%a%release%candidate%
! If%we%can’t%test%the%complete%version,%how%do%we%
guarantee%quality?%%Is%it%magic?%

12. Test%Layering%–%A%Golden%Ticket%
Monitor%your%released%so(ware:%Errors,%Usage%
volume,%usage%pa#erns,%user%feedback%
Switch%new%features%and%architecture%
QA%System%with%Human%test%consultants%
Code%review:%Both%a%manual%test,%and%a%place%to%
ask%for%test%scripts.%
ConAnuous%integraAon:%Run%automated%tests%
before%using%human%review%Ame%
Unit%tests%in%the%development%environment%

13. Quality)
Go%Both%Ways%
Increase)Quality)
Increase)Velocity)
Velocity)

14. Switch%and%Unveil%
Test%
Story%Owner%and%testers%see%the%
change%on%test%systems.%
Beta%
Insiders%see%it%and%use%it.%%Story%
Owner%can%show%it%to%users%for%
feedback%or%A/B%tesAng.%
UNVEIL!%
The%big%event.%%Communicate%with%
all%users.%%Measure%reacAon.%
One%code%version%
No%special%test%builds%
No%longVrunning%branches%
Hidden%
Programmer%sees%a%change%locally.%
Change%is%tested%in%the%%main%
version%but%not%seen.%

16. Code%and%Test%Pa#erns%
! Centralized%
! TradiAonal%process%with%no%branches%finds%and%fixes%
problems%as%early%as%possible%
! Distributed%
! Release%every%change%with%its%own%branch%and%test%
! Temporary%branches%
! Efficient%review,%combines%benefits%of%centralized%and%
distributed%
! MAXOS%
! Use%centralized%conAnuous%integraAon%to%manage%a%
massively%scalable%IT%system%

17. Centralized%CI/CD%
Contributor%Commits%–%“as%early%as%possible”%to%find%problems%
ConAnuous%
IntegraAon%tests%
Fail%V%alarm%
Pass%
Release%Candidate%
Test%System%
QA%TesAng%
Release%

18. Distributed%ConAnuous%Delivery%
Contributor%Commits%
Peer%review%merge%requests%
Branch%
or%Fork%
QA%Consults%
Merge%back%
Current%
Pass%Final%
Auto%Test?%
Deploy%
Deployed%version%
Other%contribuAons%
merged%and%released%
“as%late%as%possible”%

19. Distributed:%MulAple%Test%Systems%
Contributor%1%
Test%System%1%
CI%System%
QA%Team%
Contributor%2%
Test%System%2%
ProducAon%
Revision%
Release%
AnyAme%

20. Assembla%spins%up%test%servers%

21. Test%and%Review%on%Temp%Branches%
Test%before%spending%human%Ame%on%review%
Contributor%Commits%
Auto%tests% Review%
Auto%tests% Review%
Mainline%
• Github%style%–%Pull%requests%or%merge%requests%going%to%CI%system%
• Gerrit%style%–%Temporary%branches%for%each%contribuAon%
• Assembla%git%supports%both%–%merge%requests,%and%temp%branches%%created%%
by%a%push%to%a%protected%branch.%

22. The%Services%Megatrend%
Desktop
Web App
Cloud Services
Service
App
DB
Service
Service

23. Matrix%of%Services%V%MAXOS%
Organize%by%
Each%team%releases%
tesAng%as%one%system%
when%ready%
Service%team%
IntegraAon%
test%env%
ProducAon%
service%
Service%team%
IntegraAon%
test%env%
ProducAon%
service%
Service%team%
IntegraAon%
test%env%
ProducAon%
service%
Current%
Work%
PrioriAzed%
Backlog%
Feedback%on%speed,%errors,%usage,%and%requests%

24. Coordinate%without%big%meeAngs%
ConAnuous%IntegraAon%between%%
latest%dev%version%of%each%service%
Service%team%
IntegraAon%
test%env%
Current%
Work%
Service%team%
IntegraAon%
test%env%
PrioriAzed%
Backlog%
Service%team%
IntegraAon%
test%env%
• ConAnuous%integraAon%
helps%teams%coordinate.%
• See%dependencies%
between%“producers”%
and%“consumers”%
• Errors%and%conflicts%show%
related%team%contact%
info%
• MeeAngs%and%changes%
negoAated%between%two%
teams,%not%many%

26. Role:%Developer%
! Developers%have%more%power%and%responsibility.%
! Developers%have%more%responsibility%for%tesAng.%
! Developers%(not%QA%or%PM)%decide%when%to%release.%This)
is)a)strong)finding.%
! IncenAves%are%correct.%%Developer%might%have%to%come%back%
from%Friday%night%beers%to%fix%a%problem.%%This%provides%a%
moAvaAon%to%make%good%decisions%and%automate%tesAng.%
! Features%can%be%released%but%hidden.%%Product%Managers%
and%Marketers%will%unveil%when%they%are%ready.%Unblock!%

27. Role:%QA%
! QA%is%a%consultant.%
! QA%gets%more%respect.%%Developers%have%to%ASK%for%
service.%
! Developers%do%more%of%the%tesAng%work.%%They%should%
organize%reviews%and%automated%tests%so%that%bugs%don’t%
go%through%into%the%manual%test%process.%
! QA%has%more%Ame%to%invesAgate%usability%
! QA%monitors%producAvity%and%quality%metrics%

28. IncenAves%for%ConAnuous%Flow%
You%don’t%need%culture%change.%%You%only%need%to%release%
more%frequently%
1. If%you%do%code)review,%you%can%get%automated.%No%
browbeaAng%and%cajoling%is%required.%%Developers%will%
ask%for%tests%when%they%review.%
2. If%developers%decide%to%release,%they%will%take%more%
responsibility%for%tesAng%and%automated%tests.%
3. If%you%release%more%frequently,%developers%will%quickly%
learn%not%to%break%the%build%
4. If%PM’s%take%unveil)and)measurement%responsibility,%
they%will%make%be#er%features%

29. Our%Master%Plan%
1. Release%more%frequently%
2. Improve%
%
%
%
%
%
%

30. www.conAnuousagile.com/unblock%