For the last four years myself and my colleagues at RainStor have been evolving a process for testing a structure data archiving system in an Agile development environment. In this talk I will discuss the evolution of a team from a rudimentary Agile implementation on an unreleased product, to our current process which uses the fundamental elements of Specification By Example to successfully deliver software functionality across 30 different platform/backend configurations to a series of high profile and demanding customers. Last year our company was used as a case study for successful implementation in Gojko Adzic's book on Specification By Example.
My report will discuss the lessons learned during the early implementation and the challenges faced in moving away from a compressed waterfall approach. Through a process of incremental change we have identified and tackled the fundamental issues that undermined the development effort as a team. I’ll describe some of the mistakes made in attempting to implement a more formal process of requirements documentation into an Agile implementation and the benefits we uncovered on moving to a more flexible user story based approach. I’ll also discuss some of the issues around trying to implement user stories in a server system with no GUI and very technical and performance based requirements.
Raising the importance of quality and the status of testing both within the development team and the organisation as a whole has allowed the challenges facing the team to be recognised and respected. The result has been a more collaborative approach taken between developers and testers both through “collaborative specification” of user stories and tackling the problems that impact the delivery of value to the customers. I also plan to discuss how we’ve expanded from documenting acceptance criteria for each user story such that we now document Criteria, Assumptions and Risks for each feature and, rather than a ‘Done/Not Done’ approach how we identify the confidence in each of these categories to measure the confidence we have in each new feature being implemented.
Having the test team as an involved and influential team through the entire development process has also allowed us to implement a number of testability features to help to make the product more testable. I will discuss the benefits of having development understand and prioritise testability issues with some illustrative examples.
I will discuss the challenges and benefits of developing our own metadata driven test harnesses as opposed to an off the shelf solution. I’ll detail how having control over these harnesses has allowed us to work towards a self documenting test system using realistic customer examples as “Automated Specifications” of the RainStor system allowing us to explain current behaviour to Product Management in terms of well understood customer scenarios.
'An Evolution Into Specification By Example' by Adam Knight
1. An Evolution Into SpecificationAn Evolution Into Specification
By ExampleBy Example
Adam KnightAdam Knight
Twitter: adampknightTwitter: adampknight
2. Two ConversationsTwo Conversations
• Zachary KnightZachary Knight
““Can we seeCan we see
the picture ofthe picture of
the Monkeythe Monkey
turning into aturning into a
Man?”Man?”
• Gojko AdzicGojko Adzic
““Although theirAlthough their
process has almostprocess has almost
all the elements ofall the elements of
SBE, they just thinkSBE, they just think
of it as theirof it as their
homegrown way ofhomegrown way of
developingdeveloping
software”software”
4. Evolutionary CyclesEvolutionary Cycles
• It takes time to adapt to changeIt takes time to adapt to change
• Methodology is an environmental factorMethodology is an environmental factor
• Faster feedback cycles encourage evolutionFaster feedback cycles encourage evolution
6. • Included Functional DesignIncluded Functional Design
• Abstracted out Customer DetailAbstracted out Customer Detail
• Not maintainedNot maintained
• Not completed in sprintNot completed in sprint
Requirements DocumentsRequirements Documents
7. User StoriesUser Stories
• Developer ReservationsDeveloper Reservations
• Hard to break downHard to break down
• Still Solution BasedStill Solution Based
• Value drivenValue driven
• IndependentIndependent
• RecursiveRecursive
11. Early Test ManagementEarly Test Management
• Poorly selected TCM toolPoorly selected TCM tool
• Excel with Test CaseExcel with Test Case
EstimatesEstimates
Counting test cases is inappropriateCounting test cases is inappropriate
12. Exploratory TestingExploratory Testing
• Session Based ETSession Based ET
Run times too longRun times too long
Can’t parallelise sessionsCan’t parallelise sessions
• Thread Based ETThread Based ET
Estimate chartersEstimate charters
• Reporting ConfidenceReporting Confidence
15. Towards Living DocumentationTowards Living Documentation
• Self DocumentingSelf Documenting
• Reporting in Business TermsReporting in Business Terms
• Specification for RefactoringSpecification for Refactoring
16. Current PositionCurrent Position
• Around100 End Customer installationsAround100 End Customer installations
• Highly Rated Testing/Support TeamHighly Rated Testing/Support Team
• ““The product was installed seamlessly and the customerThe product was installed seamlessly and the customer
was delighted.”was delighted.”
17. RecapRecap
It takes time to adapt to changeIt takes time to adapt to change
FROMFROM TOTO
Formal RequirementsFormal Requirements User Stories, CollaborativelyUser Stories, Collaboratively
Specified using ExamplesSpecified using Examples
Test Case ManagementTest Case Management Exploratory ThreadsExploratory Threads
Bespoke AutomationBespoke Automation
ScriptsScripts
Self Documenting File DrivenSelf Documenting File Driven
Test AutomationTest Automation
18. ““Improbably excellent teamsImprobably excellent teams
become so through a sustainedbecome so through a sustained
series of entirely probable smallseries of entirely probable small
improvements ”improvements ”
Jason GormanJason Gorman
Bletchley Park :Bletchley Park :
www.bletchleypark.org/content/contact/donation/support.rhtmwww.bletchleypark.org/content/contact/donation/support.rhtm