I believe that our existing models of testing are not fit for purpose – they are inconsistent, controversial, partial, proprietary and stuck in the past. They are not going to support us in the rapidly emerging technologies and approaches. The certification schemes that should represent the interests and integrity of our profession don’t, and we are left with schemes that are popular, but have low value, lower esteem and attract harsh criticism. My goal in proposing the New Model is to stimulate new thinking in this area.
eurostarconferences.com
testhuddle.com
Exploring the Future Potential of AI-Enabled Smartphone Processors
New model
1. A New Model for Testing
@paul_gerrard
Paul Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com
Programme Chair 2014
2. Helping clients transform their testing through
INNOVATION, COACHING and LEADERSHIP
Our CLIENTS
– Want to be agile rather than follow Agile dogma
– Have a pragmatic approach and are focused on delivery
– Want a solution that fits, not a badly fitting suit.
3. Typical tester job spec? (May 2014)
• Skills & Requirements
– Thorough C# knowledge including Linq
– Good .Net experience including charting and winforms
applications.
– Experience building and maintaining automated test systems
– Ability to interact with the developers to anticipate future
needs
– Ability to influence to ensure good test procedures are
developed and implemented
• Desirable Skills
– Some knowledge of Java for mobile testing
– An interest in photography.
4. Agenda
• Pressure to Change
• Three Innovations
• Where is this Leading?
• A New Model forTesting
• Consequences
• Shift-Left is More Important than Agile
5. What's happening?
• Stampede to mobile computing
• Big Data
• Internet of (Every)Thing, Pervasive Computing
• Continuous Delivery, DevOps
• (Test) Analytics, Data-Driven Development
• Shift-left, embedded testers, no test team
• Did you really think Agile was the last word?
7. #1
Agile doesn’t work
but being agile might
Agile is no longer innovative.
Challenging Agile and being agile
might be
Is continuous delivery for you?
8. Agile doesn’t work?
• It works for some
• But many organisations have difficulties
• Mostly a cultural problem
– Requirements take months to elicit and agree
– Complex stakeholder relationships
– Stakeholders with ‘all the knowledge’ can’t be
spared to be the on-site customer/product owner
– Lack of confidence => big system/acceptance tests
• We are Agile but ...
9. Pragmatism rules (again)
• Pragmatic companies use the things that work
and discard those that don’t
• Let's be honest: Most are doing some form of
hybrid approach:
– Chaotic interfaces between Agile teams and
structured businesses
– TDD, BDD,ATDD, SBE today…
– Continuous Delivery, DevOps… tomorrow?
11. TestingBDD/Test-Driven)
Trusted
Requirements +
Stories
Shift left (redistributed testing)
Functional
Requirements
Build (+little testing) but
lots of rework
Testing (lots of
checking + bugs)
Trusted
Requirements +
Stories
Testing
SAVED
TIME
BDD/Test-Driven
Who is going to do all this?
BAs, Developers or Testers?
15. Analysis Insight Decision
Development
and Testing
Production
Testing
IntroducingTest Analytics
“IntroducingTest Analytics”, Paul Gerrard:
http://gerrardconsulting.com/index.php?q=node/630
Julian Harty, "Symbiotic Relationships betweenTesting and SMAC":
http://blog.bettersoftwaretesting.com/2013/07/symbiotic-relationships-between-
testing-and-social-mobile-analytics-cloud-computing/
“The capture, integration and analysis of test and
production monitoring data to inform business and
software development decision-making”.
16. Data silos for test analytics
(an illustration)
Stakeholder Requirements Assurance (Production)
Application
Monitoring
Production
Environment
Monitoring
Stakeholders
Business Goals
and measures
Stakeholder
Involvement/
Engagement
Risk
Requirements
Stories/Feature
Descriptions
Glossary of terms
andTerm Usage
Processes
Process Paths
(Workflows)
ManualTests
GeneratedTest
Code (unit,
integration,
system level)
Application
Instrumentation
AutomatedTest
Execution History
Test Status
Feature Code
Change History
Application
Process Flows
User Accesses/
Activities
Feature Calls,
Response times
Feature use cases
(data usage)
Application
Alerts/Failures
Database Accesses
Database Content
Production
failures
System assets
Resource Usage
Log
Performance data
System Events
System Alerts/
Failures/Incidents
Outages
17. Where is this leading?
If I knew that, I wouldn't be here...
20. From high process to DIY process
• In the past, high process was seen to be the
technical solution to the software problem
• But technical solutions cannot solve human
problems
• Agile promised flexibility, but a lot of Agile is
really 'high process-in-the-small'
• True agility means a team can adapt or select
its process on-the-fly.
21. The old ways won't
work in the future
We need a New Model ofTesting
(free from logistics)
22. A New Model for
Testing
Don't throw stones.Yet.
I'm working on a refined version (with others).
Want to help/review?
23. Forget Logistics
(for the time being)
Document or not?
Agile v waterfall?
Planned v exploratory?
This technology v that technology?
28. Judgement, exploring and testing
Testing
(the system)
Our model(s) are adequate
Our model(s) are not adequate
Exploring
(sources)
Judgement
Creates test
models
Uses test
models
We explore sources of knowledge to build test models that inform our testing
BTW – Do Developers explore the same way? I think so.
33. Do developers and
testers think the same
way?
Similar processes, not thinking the same
Easier to embed testers in development
Developer testing will improve
40. • Analysis, enquiry and elicitation
• Modelling
• Creation of custom models, using
heuristics, guesses, brainstorming,
ideation, creative thinking
• Custom test design techniques
• Comparison of models, value, advantages,
disadvantages, compromises
• Identification, validation and use of
oracles
• Predicate logic and proof
• Hypothesis and inference
• Socratic method
• Rapid Review and Inspection techniques
• Test case design
• Test models and the meaning of coverage
• Testing as controlled experiment
• Observation, Note taking, recording
A very different skillset
• Basic data analysis and statistics
• Decision-making with incomplete data
• Computer forensics
• Fault tree analysis
• Failure diagnosis
• Bug advocacy, triage processes and
negotiation
• Meaningful software and test metrics
• Visual presentation of data
• Reporting and presentation skills
• Understanding stakeholders
• Test analytics
• Risk management, risk-based testing and
decision-making
• Critical Thinking
• Interpersonal skills
• Dealing with uncertainty/fallibility
44. Testing doesn’t just
apply to existing
software, at the end
We test systems, not software.
We test early, we test often but it
might not be called testing
45. Testers Need to Learn
How to Code
Some, perhaps many will have to
46. Testing is about
measuring achievement,
not quality
Align systems delivery with project
goals and risks and you’ve got a seat
on the board
48. New Model Testing
changes what testers &
developers need to
learn
Roles of developers/testers become blurred
Perhaps they are not different disciplines after all?
Why have a tester certification?
49. The phase after
development is
REWORK, not Testing
Testing must align with development,
not compete with it or rescue it
50. A New Model for Testing
http://dev.sp.qa/download/newModel
@paul_gerrard
Paul Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com
Programme Chair 2014