Testing what we know, or have a clear understanding of, is relatively straight forward, as is making decisions based on the expected result. But today’s world is presenting us with the Unknown and the Ambiguous, which can only be approached by hypothesizing and experimenting - a lot! This requires intentional thinking, and a different strategy to observe in context. This session will uncover how testers are helping their teams and product owners, by basing their testing on the science behind creating hypotheses and running experiments. A testing mindset and probing the context around use cases are some of the most valuable competencies testers bring to the team in order to enable decisions based on data.

1. Agile and Testing the Unknown: The Art and Science
with guest speaker Ardita Karaj
Agenda:
Presentation 6:15 p.m. to 7:15 p.m.
Q&A 7:15 p.m. to 7:30 p.m.
Event Survey
October 27, 2020

2. Welcome!
All people will start the meeting in Mute mode.
We ask you to remain in Mute for the duration of the meeting
to avoid background noise.
Questions can be put into the chat window.
Someone will monitor them and consolidate similar questions
into one.
The meeting will be recorded.

3. Welcome from the TASSQ Board of Directors
Desislava Nikolova
President
GuillaumeVulliez
Vice President
Amar Gadgil
Board Member
Neil Price-Jones
Treasurer
Venugopal Pula
Board Member
Vida Shafaghat
Secretary
RobVirdee
Board Member
Bob Thompson
Board Member
Tatiana Andronache
Board Member

4. Agile Practices and Testing:
The Art & Science of Testing the Unknown
Ardita Karaj with TASSQ
October 27, 2020

5. About me
Enterprise coach, trainer and change agent, experienced in
Product Management, Business improvement and Lean
Startup.
Started the career as a software engineer, worked with
teams that pushed edges of technology and won awards.
Taking the mentor and leader stance, I found that success
came when business and technology worked closer together.
20 years of experience working with many organization of
different types (from startup to government), different
markets (insurance, banking, retail, entertainment, etc) and
different sizes. Worked with many leaders to find innovative
and collaborative ways for their organizations to increase
customer satisfaction and build successful businesses.
With a vision for using technology as a competitive
advantage, and understanding the need for organizations to
become leaders by disrupting markets, founded Industrial
Logic Canada offering professional services with focus on
Continuous Delivery.
I organize and speak in several conferences around the
World every year.

6. Experimentation
6

7. Future of Testing is Postmodern
https://medium.com/@jarbon/
Testing is the art and science of taking a piece of software and trying all sorts
of ways to break it….
Jason Arbon
…Our community has spent too much time finding which school is correct:
“Analytical”, “Risk-Based”, “Quality Assurance”, “Modern”, “Context-Driven”,
etc.

8. How testers evolved testing, deliberately
Evolution on development came from Testing mindset
• Chaos Monkey/Gorilla/Kong
• Property based testing
• Exploratory testing
• Use of AI for dynamic features
• Testers as Machine Learning engineers
• …

9. • Defects
• Performance
• Correct data
• Security
• Accessibility
• Ease of Use
• Richness of features
• No unused features
• Flow of actions
• User friendly
• Desired by client
• Refactored
• Tested
• Clean
• Integrated
• Easy to read
• Easy to change
Product Quality Code Quality

10. Where do we need testing to start?
Expand testing mindset to Business
• Shift left, early in the process
• Not just test performance and matching requirement
• Start by testing the need for those requirement
• Start by testing the business idea behind those requirements

11. STRATEGY
An approach to achieve an end goal with the
means we have
Strategy provides a framework on how to make
decisions based on a set of values and a period
of time ahead

12. Cynefin
a sense-making
framework
https://www.cognitive-edge.com/cynefin-st-davids-day-2020-1-of-n/
Aporetic/Confused

13. How much structure do we need?
Clear/Obvious – Just build it. If it’s too big, slice it to stories/scenarios, and do the most valuable ones
first.
Complicated/Needs some planning – Do Story Mapping, write all the stories, and prioritize the most
valuable and/or most risky ones first
Complex/A lot of unknowns – Don’t try to find all the stories. Identify one or two experiments that will
provide some value and teach you something about the problem and its solution. Use what you learn to
evolve on the rest.
Chaotic – Put out the fire first; writing and splitting stories probably isn’t important right now.
Confused/Disordered – Figure out in which situation you’re in before acting so you don’t take the wrong
approach.
Is the request:
13

14. Forming scientific hypothesis
• For a hypothesis to be scientific, the scientific
method requires that one can test it.
• Hypothesis is Testable, Precise and Discrete
(see David Bland https://precoil.com)
• Hypothesis is a proposed explanation of a
situation/context
• To validate/invalidate an hypothesis, we need
to understand the variables that affect the
results
• Changing more than 1 variable for each
experiment will result in hard to define affects
Variables:
- Quality of seeds
- Water
- Temperature
- Fertilizer

15. Forming and implementing an Hypothesis
5 steps:
1.Ask a question
2.Initial research
3.Formulate hypothesis
4.Run experiment
5.Answer question or
Refine Hypothesis
1. How can I be healthier?
2. Food, Exercise, Mental health, Sleep
3. I believe that by moving to a vegetarian diet, I
will have less muscle inflammatory pain. I will
know I am successful when I feel 70% less
muscle pain every day.
4. No meat for 2 weeks. Measure pain
5. I did have less pain but was 40%. I also
noticed better sleep.
• New: I believe that by moving to a plant
based diet, I will have less muscle
inflammatory pain. I will know I am
successful when I feel 70% less muscle
pain every day.

17. 17

18. Where is your Test column on the board?
At the end:
Testing is simply a verification, an after
thought, a late insight for improvement
of the solution
At the front:
Testing leads the development and
brings better ways to architect the
solution and how to offer it to clients

19. Learning Release(s) —> Earning Release
https://medium.com/@Ardita_K/the-learning-release-70374d2450b3
What is expected from a Release?

20. 20
Product Teams use testing strategies to
Instrument the product. Products need
to be instrumented to gather info/metrics
How do we know we achieved the ROI?

21. Instrumentation Framework
● No instrumentation has ambiguity
● Bad instrumentation is costly
21

22. Example on setting up an experiment
22

23. Example on setting up an experiment
23

24. Hypothesis
We believe that by doing ____
we will see ____ as result
We know we are successful when we observe _____

25. Fast, Frugal Learning with a Feature Fake
25
https://www.industriallogic.com/blog/fast-frugal-learning-with-a-feature-fake/
The business goal:
Increase engagement of the community
Feature: Group chat

26. Fast, Frugal Learning with a Feature Fake
26
https://www.industriallogic.com/blog/fast-frugal-learning-with-a-feature-fake/
Experiment lasted 3 days
Did not provide compelling data to make a real investment on GroupChat feature

27. Lost my name
The business goal:
Increase user experience on
checkout
Feature: New wrapping paper
27

28. Takeaways…
• Testing mindset is needed early in Agile
Product teams. Do we know the problem?
• Test not only the functionality, Test the
Business Idea
• Embed Learning in the Product
(Instrumentation)
• Experiment to Learn before deciding how
much to earn
• Build Hypothesis and run Experiments in
5 steps
• Define the parameters and experiment by
changing one at a time
• Testers can help their agile teams early in
the process; bring focus

29. Thank you!
QUESTIONS AND DISCUSSIONS
Hypothesis, Experiment,Testing mindset, Shift Left …
Ardita Karaj, ardita@IndustrialLogic.ca