Mais conteúdo relacionado Semelhante a HBT - A Revolutionary Approach to Testing Software (20) Mais de STAG Software Private Limited (20) HBT - A Revolutionary Approach to Testing Software1. Hypothesis Based Testing (HBT)
A revolutionary approach to testing software
T Ashok
ash@stagsoftware.com
in.linkedin.com/in/AshokSTAG
ash_thiru
Webinar: 22 June 2011, 1430-1530 IST
2. How can we seek inspirations from nature
to deliver “Clean Software”?
In the next hour, let us look at the act of
testing differently and see how we can
think better and test scientifically?
© 2011. STAG Software Private Limited. All rights reserved.
3. Quality is...
“... at least meeting expectations”
Meeting expectations implies
“satisfying properties”
i.e. Expect certain properties to be satisfied.
Properties are impeded by “potential defects”
© 2011. STAG Software Private Limited. All rights reserved.
4. Properties of matter
Physical & Chemical properties of
matter allow us to:
... classify
... understand behaviors, interactions
... enable checking purity
Think!
How can we use a similar train of
thought to identify “properties of
cleanliness” and then “types of
defects”?
http://www.science.wellsresearch.org/wp-content/uploads/2010/02/CBScienceHSPhysicalScienceSt2.jpg
© 2011. STAG Software Private Limited. All rights reserved.
5. “Properties of the system”
Cleanliness criteria End user expectations
“affected by”
Issues in specifications,
Potential Defect Types (PDT) structure, environment
and behavior
© 2011. STAG Software Private Limited. All rights reserved.
6. Expectations
“Properties of the system”
Needs Cleanliness criteria
Features
Environment
“impedes”
Behavior
Structure
Potential Defect Types (PDT)
Material
Expectations delivered
by Needs (Requirements)
via Features
that display Behavior
constructed from Materials
in accordance to a Structure
in a given Environment
© 2011. STAG Software Private Limited. All rights reserved.
7. Fractional Distillation
A technique to separate mixtures
that have components of different
boiling points
Think!
There are a variety of defect
types that may be present in the
system.
To optimally uncover the
From : http://withfriendship.com defects, can we separate these
types of defects on the basis of
certain properties and
optimally uncover the defects?
© 2011. STAG Software Private Limited. All rights reserved.
8. Fetal growth - Delivering systems (Babies too!)
Visualizing health :
- Size, Shape, Parts
- Types of issues not present
Think!
How can we measure the
‘intrinsic quality’ at a stage?
Quality Growth
At each stage certain types of
Source :http://www.environment.ucla.edu/media/images/Fetal_dev5.jpg
defects detected & removed.
Quality grows.
Can we chart this as “cleanliness
index”?
© 2011. STAG Software Private Limited. All rights reserved.
9. Sherlock Holmes
Focus on intellectual power to solve mysteries.
Strong application of deductive logic.
Think!
Can we hypothesize as to what can go
wrong and prove/disprove?
Hypothesize potential defect types and
prove presence of instances. If they are
not present, can we reason their
absence?
© 2011. STAG Software Private Limited. All rights reserved.
10. Test methodologies focus on activities that are driven by a process which are
powered by tools, yet successful outcomes still depend a lot on experience.
Typically methodologies are at organizational level.
On the other hand HBT
... is a personal scientific methodology enabled by STEMTM ,
a defect detection technology to deliver “Clean Software”
© 2011. STAG Software Private Limited. All rights reserved.
11. Scientific approach to uncovering defects
Cleanliness criteria What is the end user expectation of “Good Quality”?
Potential Defect Types What types of issues can result in poor quality?
Evaluation Stage When should I uncover them?
Test Types How do I uncover them?
Test Techniques What techniques to generate test cases?
Scenarios/Cases What are the test cases? Are they enough?
Scripts How do I execute them?
Metrics & Management How good is it? How am I doing?
© 2011. STAG Software Private Limited. All rights reserved.
12. HBT: Quality Levels, Aspects & Issues
Aspect Issues
Usage
L9 End user value User flows, experience
Environment, Data, Logic
L8 Clean Deployment Compatibility, migration
Structure, Logic, Usage
L7 Attributes met Performance, security, volume, load...
Environment
L6 Environment cleanliness Resource leaks, Compatibility...
Logic, Environment
L5 Robustness Error handling
Data, Logic
L4 Behavior correctness Functionality
L3 Structural integrity Structure
Internal structural issues
Structure, Usage
L2 Input interface cleanliness UI issues
Data
L1 Input cleanliness
Input data handling
© 2011. STAG Software Private Limited. All rights reserved.
13. HBT - A Quick Introduction
SIX stages of powered EIGHT disciplines
DOING by
of THINKING
D8 D1
S6 S1
Analysis & Business value
Assess & Understand
management understanding
ANALYSE EXPECTATIONS
D7 D2
Execution & Defect
D8 D1
reporting STEM Core hypothesis
Tooling D7 D2
S5 STEM Understand 32 core
SUPPORT S2
D6 D3 CONTEXT concepts
D5 D4 Strategy &
Visibility
planning D3
D6
Devise Formulate
HYPOTHESIS Tooling Test design
PROOF
S4 S3 D5 D4
HBT powered STEM
Personal test methodology by Defect detection technology
© 2011. STAG Software Private Limited. All rights reserved.
14. D1 Business value understanding D2 Defect hypothesis D3 Test strategy & planning
Landscaping
EFF model Orthogonality principle
Viewpoints
(Error-Fault-Failure) Tooling needs assessment
Reductionist principle
Defect centricity principle Defect centered AB
Interaction matrix
Negative thinking Quality growth principle
Operational profiling
Orthogonality principle Techniques landscape
Attribute analysis
Defect typing Process landscape
GQM
D4 Test design D5 Tooling
Reductionist principle Automation complexity
Input granularity principle 32 core assessment
Box model Minimal babysitting
Behavior-Stimuli approach concepts principle
Techniques landscape Separation of concerns
Complexity assessment Tooling needs analysis
Operational profiling
D6 Visibility D7 Execution & Reporting D8 Analysis & Management
GQM
Contextual awareness Gating principle
Quality quantification
Defect rating principle Cycle scoping
model
© 2011. STAG Software Private Limited. All rights reserved.
15. How is HBT different?
from other methodologies...
Goal
al
p ic
Powered by experience
Ty
drives
T
Activities
HB
......................................
detection technology
Powered by defect
......................................
...................................... Activities
...................................... ......................................
(STEM)
...................................... ......................................
...................................... ......................................
......................................
......................................
......................................
hopefully
results in
Goal
© 2011. STAG Software Private Limited. All rights reserved.
16. Hypothesis Based Testing (HBT)
S1
Cleanliness criteria Potential defect types S3
S2
Staged & purposeful
Expectations detection
S4
Complete test cases
S6 Goal directed measures Sensible automation S5
© 2011. STAG Software Private Limited. All rights reserved.
17. Cleanliness criteria Potential defect types
Clear Baseline
S1, S2
Staged & purposeful
detection
Expectations
Set a clear goal for quality
Complete test cases
Example: Clean Water implies
Goal directed
measures
Sensible automation 1.Colorless
2.No suspended particles
3.No bacteria
4.Odorless
What information(properties) can be used
to identify this?
...Marketplace,Customers, End users
...Requirement(flows), Usage, Deployment
... Features, Attributes
...Stage of development, Interactions
... Environment, Architecture
... Behavior, Structure
© 2011. STAG Software Private Limited. All rights reserved.
18. Cleanliness criteria Potential defect types
A goal focused approach
S3 to cleanliness
Staged & purposeful
detection Identify potential defect types that
Expectations can impede cleanliness
Complete test cases
Example:
Data validation
Timeouts
Goal directed Sensible automation Resource leakage
measures
Calculation
Storage
Presentation
Transactional ...
Scientific approach to hypothesizing defects is about looking at
FIVE Aspects - Data, Logic, Structure, Environment & Usage
from
THREE Views - Error injection, Fault proneness & Failure
Use STEM core concepts
> Negative thinking (Aspect)
> EFF Model (View)
“A Holmes-ian way of looking
at properties of elements”
© 2011. STAG Software Private Limited. All rights reserved.
19. Cleanliness criteria Potential defect types Levels, Types & Techniques -
STRATEGY
Staged & purposeful
detection
S4 NINE levels to Cleanliness
Expectations
L9 End user value
Complete test cases
L8 Clean Deployment
Goal directed L7 Attributes met
Sensible automation
measures
L6 Environment cleanliness
Quality Levels L3 L5 Robustness
PDT7
PDT6
L4 Behaviour correctness
L2 PDT5
Test Techniques (T1-T4)
PDT4
L1 PDT3 TT5 T4 L3 Structural integrity
PDT2
PDT:
PDT1 Potential Defect Types TT4 L2 Input interface cleanliness
T3
TT3 L1 Input cleanliness
Test Types PDT7
TT5
PDT6 TT2
T2
TT4 PDT5
PDT4 TT1 T1
TT3
PDT3
TT2 PDT2 TT:
TT1 PDT1 Test Types
“Fractional distillation of bug mixture”
© 2011. STAG Software Private Limited. All rights reserved.
20. Cleanliness criteria Potential defect types
Countable test cases &
Fault coverage
Staged & purposeful
detection
Expectations Use STEM Core concepts
> Box model
Complete test cases
> Behaviour Stimuli approach
S4
> Techniques landscape
Goal directed > Coverage evaluation
Sensible automation
measures
to
- Model behaviour
- Create behaviour scenarios
- Create stimuli (test cases)
Test Scenarios/Cases
R1 PDT1 Irrespective of who designs, #scenarios/
TS1 TC1,2,3
R2 PDT2 cases shall be same - COUNTABLE
TT
R3 TS2 TC4,5,6,7 PDT3
Requirements & Fault traceability
That test cases for a given
requirement shall have the ability
to detect specific types of defects Guarantee test adequacy.
FAULT COVERAGE Guarantee implies
that the means to the end is rational & provable
© 2011. STAG Software Private Limited. All rights reserved.
21. HBT Test Case Architecture
Organized by Quality levels
sub-ordered by items (features/modules..),
Level segregated by type,
ranked by importance/priority,
Item sub-divided into conformance(+) and robustness(-),
classified by early (smoke)/late-stage evaluation,
Type tagged by evaluation frequency,
linked by optimal execution order,
classified by execution mode (manual/automated)
Priority
Focus
Stage
A well architected set of test cases is like a
effective bait that can ‘attract‘ defects in the
Frequency
system.
Order
It is equally important to ensure that they
Mode
are well organized to enable execution
optimisation and have the right set of
information to ensure easy automation.
© 2011. STAG Software Private Limited. All rights reserved.
22. Cleanliness criteria Potential defect types Focused scenarios + Good
Automation Architecture
Staged & purposeful
detection
Expectations Level based testcriteria
Cleanliness scenarios yield
Complete test cases shorter scripts that are
more flexible for change and
S5 easily maintainable.
Goal directed Sensible automation
measures
L9 End Expectations
user value
L8 Clean Deployment
L7 Attributes met
L6 Environment cleanliness
L5 Robustness
L4 Behavior correctness
L3 Structural integrity
L2 Input interface cleanliness
L1 Input cleanliness
© 2011. STAG Software Private Limited. All rights reserved.
23. Cleanliness criteria Potential defect types
“Cleanliness Index” -
Improved visibility
Staged & purposeful
detection L4
Expectations PDT10 TT8
Complete test cases PDT9 TT7
S6 L3 PDT9 TT6
Cleanliness
PDT8 TT5
Goal directed Sensible automation
measures PDT7
TT4
L2 PDT6
PDT5
TT3
L1 PDT4
PDT3
TT2
PDT2
PDT1 TT1
Stage
Quality report
CC1 CC2 CC3 CC4
R1 Met
R2
Not met
R3
Partially met
R4
R5
“Growth of a baby”
© 2011. STAG Software Private Limited. All rights reserved.
24. HBT - A Case Study
Two Teams, one using HBT & The other conventional approach
Test case details Test case details (HBT)
Module HBT Normal Increase Module Total Positive Negative
M1 100 28 257% M1 100 59 41
M2 85 52 63% M2 85 68 17
M3 95 66 44% M3 95 67 28
M4 132 72 83% M4 132 112 20
M5 127 28 354%
M5 127 85 42
M6 855 116 637%
M6 855 749 106
TOTAL 1394 362 285%
TOTAL 1394 1140 254
Nearly 3x increase in #test cases increasing
probability of higher defect yield 2x improvement in negative cases
increasing probability of better defect yield
Defect details Effort details (person-hours)
HBT Normal Increase Stage HBT Normal
#Defects 32 16 100% Test analysis &
30 20
Design
20 (Major), 12(Minor)
Out of these 32 defects, few were Front loading of effort resulted
residual defects, one being critical to in lowering support cost
corrupt the entire data.
© 2011. STAG Software Private Limited. All rights reserved.
25. HBT Results
50%-1000% reduction in post-release defects
Re-architecting test assets increases test coverage by 250%
30% defect leakage reduction from early stage
Smart automation - 3x reduction in time
Deskilling - Less experienced staff do better, faster ramp up, lower cost
‘Holes’ found & fixed at requirement stage
Test assessment accelerates integration, de-risks deployment
© 2011. STAG Software Private Limited. All rights reserved.
26. Summarizing...
S2
Cleanliness criteria Potential defect types S3
S1
Staged & purposeful
detection
Expectations S4
Complete test cases
Goal directed
S6 Sensible automation S6
measures
Clear baseline
A goal focused approach to cleanliness
Levels, Types & Techniques - STRATEGY
Countable test cases & Fault coverage
Focused scenarios +
Good automation architecture
“Cleanliness Index” - Improved visibility
© 2011. STAG Software Private Limited. All rights reserved.
27. Thank you!
“...the value addition for us came from a blend of
HBT/STEM and the dedication/hard work from senior
management to excel in their services....”
Praveen Kanipakam, President
SHARP Software Development India
“...the methodology and team was the potent
combination that jumpstarted our QA."
Tapesh Kumar Agarwal, CTO & Co-Founder
Meddiff Technologies
Connect with us...
@stagsoft
“..the focus on scientific way of doing testing with blog.stagsoftware.com
their methodology HBT was reflected in the
effectiveness of test artifacts created by their team.”
Raj Khare, Chairman & Founder
SureWaves
“..the quality of test design and completeness of
testing as a result of their methodology is
evident in every release.”
Iype Isaac, CTO
Bankbazaar.com
STEMTM is the trademark of STAG Software Private Limited
© 2011. STAG Software Private Limited. All rights reserved. www.stagsoftware.com