Agile testing focuses on delivering valuable working software through collaboration, feedback, and automation. It involves the whole team taking responsibility for quality. Agile testers provide continuous feedback, prioritize value, and think critically to challenge assumptions and find problems. They collaborate with developers to shift testing left in the SDLC through approaches like specification by example and behavior driven development which define examples of desired behavior to build shared understanding.
2. What is Testing?
●Testing seeks to discover threats to the value of
software
●Questioning a product in order to evaluate it
o James Bach
●Testing is done to find information
o With that information critical project decisions
can be made
3. What is Agile Testing?
Agile testing applies the agile mindset
● It suggests attitudes and activities that help to
deliver valuable software
Applies to both
●Testing on an agile team
●Using agile testing on a traditional team
4. The Agile Testing Mindset
● The whole team is responsible for quality
o ‘QA will find it’ isn’t accepted
● Testing is not a phase, it’s an ongoing activity
5. The Agile Testing Mindset
● Short feedback loops
● Continuous improvement
● Collaboration
o How can I help?
6.
7. Why Bother?
● To fix defects as close to when they were
introduced as possible
● To keep the project moving forward
o By continually providing test feedback as the
product is developed
14. Agile Testers
Principles for agile testers
● Provide continuous feedback
● Focus on delivering value
● Prioritize high bandwidth communication
15. Agile Testers
Need to be critical thinkers
● Challenge assumptions
Need to be problem solvers
● Fix problems, not symptoms
Need to be good communicators
● They are required to give feedback
16. Agile Testers
Need to learn the business domain
● Take the customer’s point of view
Need to think differently
● Ask the what if questions
17. Agile Testers and Feedback
Give feedback
● With empathy for the customer
● With empathy for the developers
18. Giving Feedback
● Have a positive intent
o You are trying to help people
● Take responsibility for the feedback
o Provide and gather context
● You must be clear
o Otherwise your feedback is ineffective
19. Receiving Feedback
● Assume a positive intent
o People want to do their best work
● Give people the benefit of the doubt
o People have bad days
● Thank people for constructive feedback
o You want to improve to do your best work
20. Feedback Skills for Agile Testers
● Active listening
o Listen to learn
o Don’t jump to conclusions
Feedback Steps from Jurgen Appelo
● Provide context
● List observations
● Express emotions
● Sort by value
● Offer suggestions
21. Agile Testers
Trust is like money
● It can take years to earn it
and it only takes minutes
to lose it
22. Agile Testers and Programming
It is important to be technically aware
● Take a holistic view of your technical environment
Everyone cannot be good at everything
● A technical context enhances collaboration
23. Agile Testers and Programming
So does this mean testers need to program?
● Definitely maybe
Test automation enables agile testing
● Agile testing is not sustainable without automation
Instead of the individual look at the team
● The team needs to be able to program
24. T Shaped Testers
The T represents depth of knowledge and breadth of
knowledge
T
Depth
Breadth
25. T Shaped Testers
Depth is the tester’s specialized knowledge
● Integration testing
● Exploratory testing
● Etc
Depth
Breadth
T
26. T Shaped Testers
Breadth is the tester’s generalized knowledge
● Programming
● Configuration management
● Databases
● Etc
Depth
Breadth
T
29. Test Automation
You can apply agile skills to any test project
● But automation enables agile testing to succeed
30. Test Automation
Why?
● Manual testing takes a long time
● Manual testing is expensive
● You want to free your testers to do more valuable
work
31. Test Automation
More reasons
● It provides earlier feedback more often
● It is a form of documentation
● It enables Continuous Integration and Continuous
Delivery
32. Testing vs. Checking
Checking
● Things that machine could do
o Expected results are well defined
Testing
● Things that a human must do
o Observe
o Explore
o Infer
42. Exploratory Testing
How do you do it if you don’t have an expected
result?
● Use test heuristics
● Use experience and domain knowledge
What is it not?
●Undisciplined
●Undocumented
43. Test heuristics?
● A ‘rule of thumb’ for testing software
o Using experience to find defects
Heuristics
44. Variable Analysis
● Identify anything whose value can change.
● Goldilocks
o Too Big, Too Small, Just Right
● Starvation
o CPU, Memory, Network, or Disk at maximum
capacity
Elisabeth Hendrickson - Test Heuristics Cheat Sheet
Heuristics Examples
45. I'm not schooled in the science of human factors,
but I suspect surprise is not an element of a robust
user interface
Chip Rosenthal
46. Exploratory Testing
Design
● Look for interesting test variations
Executing
● When you think of the test
Learning
● Discover behaviour, look for clues about defects
Steering
● Take those clues and exercise the code more
49. Shift Left in the SDLC
We want to test early and often
● How do we start to prevent defects even before
code is written?
50. Shift Left in the SDLC
Involve test throughout the Software Development
Lifecycle
Prevent defects
● By collaborating early and often
o Business, dev, and test
● By challenging requirements early
51. Build the Right Thing, Build it Right
Testers can help:
● Build the right thing
o That meets the customers needs
o Might not be what they asked for
● Build it right
o Well designed, coded, tested
53. SBE and BDD
Specification by Example (SBE)
Behaviour Driven Development (BDD)
Both methods involve testers earlier in the SDLC
● To discover the right thing to build
Both methods use business language
● This allows Business, Dev, and Test to collaborate
54. SBE and BDD
Specification by Example (SBE) uses process
patterns to collaborate
●Defines scope from business goals
●Gathers key examples
●Refines examples
●Automates testing of the examples
55. SBE and BDD
Behaviour Driven Development (BDD)
● Defines tests first
● Captures examples of desired system behaviour
● Automates testing of those examples
56. SBE and BDD
People get distracted by automation tools
It isn’t about the tools
● It’s about the conversations
● It’s about gaining a shared understanding
Paraphrasing Liz Keogh
58. Stories and SBE, BDD
Stories are a lightweight method to frame desired
system behaviour
As a student
I want to register for a course
So that I can graduate
Who
What
Why
60. Specification Workshops
You need to get the details, SBE and BDD suggest a
conversation
Discuss the story in a specification workshop to
● Understand how it should behave
● Discover enough detail to develop and verify
● Create shared understanding
o Business
o Dev
o Test
61. Who is in that workshop? Why?
Business
QADev
The solution we want
62. Examples
Refine the examples from the Specification
Workshop using the Given When Then format to
● Define a story’s scope
● Make the story testable
● Give you the story’s key details
Don’t create an example for everything
● You’ll get lost in the details
63. Example
GIVEN Java 101 has >= 1
places available
WHEN Sarah registers for Java
101
THEN Sarah is registered in
Java 101
Postcondition
Event
Precondition
Details documented
collaboratively
66. Key Success Factors
●Use the whole team approach
●Adopt the agile testing mindset
●Automate regression tests
●Provide and get feedback
●Build the team’s core agile practices
●Collaborate with the business, and development
●Look at the big picture
72. Dag Rowe
● @dagrowe
● ca.linkedin.com/in/dagrowe
Thanks to Pythian for sponsoring the pizza, and
venue!
www.pythian.com
73. Sources
●Agile Testing: A Practical Guide for Testers and
Agile Teams
●More Agile Testing: Learning Journeys for the
Whole Team
●User Story Mapping: Discover the Whole Story,
Build the Right Product
●Specification by Example: How Successful Teams
Deliver the Right Software
●http://dannorth.net/introducing-bdd/
Software Value is contextual, it varies by project, organization and team
business value of the software is most important thing
So we test
Why because Tested code is better code
better code is more reliable
more reliable code makes customers happier
happier customers make sure the team still gets paid
Can I ship it? When will we be done? What risks do we see?
Based on the basics of testing, it applies the mindset
Of course this is simple to say but hard to do
You can’t test quality in, you have to build it in - throwing code over the wall doesn’t work
Dev, test, product management, ops, and support all test in different ways
We test assumptions as well
This does not mean people are interchangable
Continuous testing ensures continuous progress
Per build, per story, per iteration, per release
Learning
skills
agile practices
what to do better next time
Instead of the test function acting as gate keepers or the quality police - we ask how can we help?
Focus on delivering value to the customer - if they aren’t using it it isn’t valuable
Challenge requirements to get to the true needs
Before a separate test phase
If it is a regression, with automated testing - weekly, daily, hourly, per commit
If it is complex - found by testers inside the sprint it the code is being built
Not a month later when devs have switched tasks and will have to spelunk the code to figure out what they were doing
To generate a safety net that enable developers to change the code with confidence
To create real business impacts
Not this
Things don’t work as well if everyone has separate priorities
Handoffs are in place team to team
which leads to ...
Everyone tests - not just people with test or quality in their title
Functional lines should blur
Developers are ‘Test Infected’
All roles actively collaborate
The whole team is responsible for delivering a product with quality built in
As with many things agile - it is about the philosophy and mindset
Applied in the context of your projects
There are no magic bullets
Except at the Texas School Book Depository
Feedback
The fundamental job of a tester is to provide valuable information
Value
Results oriented, customer focused
Comm
F2F, Skype, Phone, IM
Don’t be document driven
Keep a customer focus - how can I help? all users of the feedback
Dev Proj. mgmt Ops Support Documentation etc
Use a investigative journalist metaphor
Report observations, and their impact
Add context
Examine multiple points of view
Learn intent of features
Learn who uses the features
Need to know the organizations priorities
Devs are awesome at building things, they need focus - test gets to take the holistic view and provide valuable contrast in their thinking
Systems Thinking
De Bono’s Six Thinking Hats
Beginner’s Mind
Customer
Has real needs, make their life better
Dev
Coding isn’t easy
With empathy for the project
We have timelines and costs to maintain
Remember they are invested in their work as well
Provide and gather context
Why is this important?
What are your assumptions?
What are the other persons assumptions?
Best Work
They care about that
Some people are not effective communicators
Benefit of the Doubt
If the feedback is starting to cause conflict
Are they ill?
Overtired?
Questioning
what problem is being solved?
how will I know it works?
who uses this?
what’s the worst thing that could happen?
Reflection
So that means?
Clarification
What do you mean by ..
Summarisation
To rephrase ..
I understood ..
Feedback wrap
Why am I focussing so much on feedback?
because
You need to maintain a good relationship even if your particular objective gets blocked
You need to work with these people tomorrow as well
What does good design look like?
Are there code maintainability challenges?
Database design and DB query skills
Business analysis skills
The skills required will depend on your context
You can they talk about data persistence and disaster recovery scenarios
You can talk about the DB triggers that create behaviour in the system
You can understand the challenges of cache invalidation
There is value in being able to read code, and SQL, to find the intent of a developer’s change
The time required to run manual tests will only increase as more are written
This has produced a great deal of debate on the interwebs
Ultimately I think more knowledge is a good thing
For a tester it allows you to find new classes of risk to your product
Also called Generalizing Specialists
So lets look at building a team with cross functional skills
UX testing
Test feedback
Test data management
Test design
Etc
System architecture
Technical writing
BDD
Story mapping
Impact mapping
Etc
So you want a team who can can code with quality, can test effectively, and can automate tests
Think of it as a matrix of competencies
So some testers can program and some devs can test - the important thing is that the team has the skills to deliver
Here we move on to test automation
To deal with change effectively you need a test automation plan
Note, you can start without automation, but you should plan your strategy for building it out
It is expensive, but should have ROI
You still need manual testers, because you really want to free them from regression testing to be more valuable to your team
We’re talking about automation but you’ll still need manual testers
To illustrate I’ll add the concept of testing vs. checking
Sapient testing
We should look at what can be automated - James Bach and Michael Bolton propose Testing vs Checking as a useful model
You still need manual testers to actually test, vs performing checks that can be automated
Sapient testing
Another way of looking at it - automation can tackle the repetitive know knows
I know when I log in with a valid password I should get to the index page
For the rest of it humans are required
In dev, test, and the rest of the project
The example James and Michael use
You don’t call compiling a program ‘automated programming’
programming is something humans do
Similarly, they call what computers can do ‘automated checking’
Testing is something humans do
Back to automation - while I can’t cover specifics here I can present a model
As you go up the pyramid, tests are more expensive
Time to run
More maintenance
You want a larger amount of tests at the lower levels - but it is important to test many different ways
Having a broader approach is often more effective than focusing on one method as thoroughly as possible
Contrast this to a common anti pattern
The SW test ice cream cone
Here we optimize for manual testing
Here is the now classic agile testing quadrants
These show options inside a quadrant
You will need to tailor testing to your team and product
However, I will point out that it is useful to touch a system multiple ways
vs putting all your concentration on 1 or 2 types of testing
you can then catch multiple classes of bugs
the 80/20 rule applies
This is Gojko Adzic’s take on the testing quadrants.
I like this one as it takes into account the concept of Checking and Testing.
Similarly to automation I can’t go into depth here
You need both
It complements automated testing to find different classes of bugs
That is Ad hoc testing - ‘let’s give it a go’
You can use your past experience
E.G. this is on google slides
I still expect it to do similar things to power point
Next is a heuristic you know intuitively
Tired Ted, Impatient Ian ..
Landmark tour
Look at Session Based Test Management from James Bach
Not a product recommendation
Borrowing a phrase from Paul Gerrard
It may not be called testing
Get testers involved earlier in the software development process
What if we had a conversation before things were even coded?
F16 fighting falcon story from SBE
mach 3
why
Gojko Adzic’s software quality model
We can test and gather or give feedback at all these levels - either manually or via automation
Looking at the pyramid we immediately realize that as a software delivery team
we can only assure the 3 bottom levels of the pyramid
and to assure our product is Useful and Successful we need feedback from the final customer
How do we shift left?
Defines scope from business goals
Are we done yet?
Gathers key examples
What does this feature look like?
Refines examples
Concrete examples of system behaviour
Automates testing of the examples
Leading to ‘Living Documentation’
like Test Driven Development (TDD)
Captures examples of desired system behaviour
via conversations
Automates testing of those examples
Uses a Given, When, Then format
As they both involve executable specifications, these two practices are very complementary
It isn’t about the tools
It’s about the conversations
To explore examples and behaviour
To gain a shared understanding
Focus on delivering value instead of documents or automation
The Goal is To Effectively Implement features,
Stories are required for the workshop
basis of the discussion
Defines the intent of the feature
the workshop elaborates the story
You find the examples discussed in the workshop - examples of concrete behaviour
On to the examples
This
Defines a story’s scope
Makes it testable
Give you the key details
Business logic
Boundary conditions
Important - Don’t create an example for everything
You’ll get lost in the details
E.G. Great idea to kill Amazon - increase sales by offering free book shipping
book plus washing machine? - no
book to Papua New Guinea? - no
Book to US - yes
Book to Canada - yes
Here we have specific testable behaviour
Doesn’t this look a lot like a test?
If you have a set of these examples it defines the feature
Most importantly Biz, Dev, Test had a conversation - they have a shared understanding of the feature and its goals
A shared understanding saves a lot of documentation - lets take a vacation photo as an example
A photo
You may say ‘That’s cute’
I remember a river behind the house in Nova Scotia where I had dinner with family friends
Our host runs a Christmas tree farm, my daughter found the water and we had to play at the bank for at least a half hour
You wouldn’t know that because you were not there
Similarly in a workshop you will have picked up all sorts of details that are not documented - but you will understand the feature
And testers will know from the beginning, instead of during the transition from dev to test
The conversation is the most important thing.
By collaborating instead of handing off documents - AKA shelfware - we have increased contextual knowledge to deliver the right thing
- Requirements are not frozen
- change is accepted
- Priorities change
- This is usual - Try to make change as efficient as possible