3. 3 Copyright 2014. Confidential – Distribution prohibited without permission
About Me
▪ Sales Engineer, Consultant and generally a nosy person
▪ Traversed through all phases of the software development lifecycle
▪ Most time spent between Development and Operations (usually
both)
▪ Really enjoying the energy in testing right now
− Test specialists
− Developers who test
− Testers who code
5. 6 Copyright 2014. Confidential – Distribution prohibited without permission
The Need For Speed
▪ Every business is an IT business
− Known as “Software-defined Enterprise”, even traditionally brick and mortar business is becoming
software based
▪ Customers demand that you deliver new features faster whilst maintaining high
levels of quality
▪ If you don’t, your competitor probably will
10. 12 Copyright 2014. Confidential – Distribution prohibited without permission
SPECIFY INTEGRATE RELEASEREGRESSION
VALUE CHAINCONCEPT CASH
Test Effort
USER ACCEPTANCE
USER ACCEPTANCE
Acceptance
Driven Testing
“I add value by sharpening the
acceptance criteria of requested
features”
Automate ALL
“Test automation serves as the safety net
for my new functionality: I focus on running
the appropriate tests continuously during
the iterations”
T
E
S
T
D
E
S
I
G
N
B
U
I
L
D
D B
T
T D
Development = Test
Test = Development
“Testing is transforming to a
automation mindset and skill
instead of a separate activity”
Testing is Changing
11. 13 Copyright 2014. Confidential – Distribution prohibited without permission
Testing is Changing: Challenges
▪ Many test tools for each of the test levels, but no single place to answer “Good
enough to go live?”
▪ Requirements coverage is not available
− “Did we test enough?”
▪ Minimize the mean time to repair
− Support for failure analysis
JUnit, FitNesse, JMeter, YSlow,
Vanity Check, WireShark, SOAP-UI,
Jasmine, Karma, Speedtrace,
Selenium, WebScarab, TTA,
DynaTrace, HP Diagnostics, ALM
stack AppDynamics, Code Tester for
Oracle, Arachnid, Fortify, Sonar, …
12. 14 Copyright 2014. Confidential – Distribution prohibited without permission
Testing is Changing: Challenges
▪ Thousands of tests makes test sets hard to manage:
− “Where is my subset?”
− “What tests add most value, what tests are superfluous?”
− “When to run what tests?”
▪ Running all tests all the time takes too long, feedback is too late
▪ Quality control of the tests themselves and maintenance of testware
▪ Tooling overstretch
14. 16 Copyright 2014. Confidential – Distribution prohibited without permission
Testing is Changing: Best Practices
▪ Focus on functional coverage, not technical coverage
▪ Say 40 user stories, 400 tests
− Do I have relatively more tests for the more important user stories?
− How do I link tests to user stories/features/fixes?
▪ Metrics
− Number of tests
− Number of tests that have not passed in <time>
− Flaky tests
− Duration
16. 18 Copyright 2014. Confidential – Distribution prohibited without permission
MakingSense ofTest Results
▪ Real go/no go decisions are non-trivial
− No failing tests
− 5 % of failing tests
− No regression (tests that currently fail but passed previously)
− List of tests-that-should-not-fail
▪ Need historical context
▪ One integrated view
▪ Data to guide improvement
19. 21 Copyright 2014. Confidential – Distribution prohibited without permission
MakingSense ofTest Results
Executing tests from Jenkins is great, but…
▪ Different testing jobs use different plugins or scripts, each with different
visualization styles
▪ No consolidated historic view available across jobs
▪ Pass/Unstable/Fail is too coarse
− How to do “Passed, but with known failures”?
23. 29 Copyright 2014. Confidential – Distribution prohibited without permission
A Central Hub for ApplicationQuality
What is needed:
1. A single, integrated overview of all the test (= quality, risk) information related
to your current release
2. …irrespective of where or by whom the information was produced
3. The ability to analyze and “slice and dice” the test results for different
audiences and use cases
4. The ability to access historical context and other test attributes to make real-
world “go/no-go” decisions
25. 31 Copyright 2014. Confidential – Distribution prohibited without permission
Beyond Test Automation
It’s a bit of an open question:
▪ Google: it’s too expensive and time-consuming to run all the tests all the time
- automated selection of a subset of tests to run
▪ Dave Farley: if you can’t run all the tests all the time, you need to optimize
your tests or you have the wrong tests in the first place
26. 32 Copyright 2014. Confidential – Distribution prohibited without permission
Beyond Test Automation
Middle ground:
▪ Label your tests along all relevant dimensions to ensure that you can easily
select a relevant subset of your tests if needed
▪ Consider automatically annotating tests related to features (e.g.
added/modified in the same commit), or introducing that as a practice
▪ Use data from your test aggregation tool to ignore flaky/”known failure” tests
(and then fix those flaky tests, of course ;-))
34. 40 Copyright 2014. Confidential – Distribution prohibited without permission
Next steps
▪ Next-Generation Testing: The Key to Continuous Delivery
https://xebialabs.com/resources/whitepapers/next-generation-testing-the-
key-to-continuous-delivery/
▪ An Introduction to XL TestView
https://www.youtube.com/watch?v=_17xKtB3iWU
▪ Download XL TestView
https://xebialabs.com/products/xl-testview/community