Creating a successful continuous testing environment by Eran Kinsbruner

1. Maturing Your Path Toward DevOps
with Continuous Testing

2. AGENDA
• State of Continuous Testing? What are the Benefits of CT?
• Fast Feedback and Continuous Visibility
• Testing that matches your skillset & What to Automate?
• Test Authoring and Coding Best Practices
• KPI for Continuous Testing Measurements
• Bonus: ML and Future of CT
Eran Kinsbruner
Dir., Lead SW Evangelist
andAuthor, Perfecto
Twitter: @ek121268 (https://twitter.com/ek121268)
Blog: http://continuoustesting.blog
LinkedIn: https://www.linkedin.com/in/eran-kinsbruner-4b47a81/

3. Continuous Testing for Digital Has Never Been More Complex
3© 2018, Perfecto Mobile Ltd. All Rights Reserved.
Augmented Reality
Virtual Reality
IOT
Artificial Intelligence
Mobile
Progressive Web
Continuous
Testing for
Digital

4. WHAT IS CONTINUOUS TESTING?
5/24/2018 4© 2018, Perfecto Mobile Ltd. All Rights Reserved.
Continuous Testing is the process of executing
automated tests as part of the software
delivery pipeline in order to obtain feedback on
the business risks associated with a software
release candidate as rapidly as possible.

5. 5 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
What the DevOps Process Probably Looks Like
Process
Impact
Organizational
Impact
2-3 WEEKS 1-3 WEEKS
❌ Unstable
❌ Labor-intensive
❌ Cluttered
❌ Slow
• Manual testing.
• Unreliable and flaky
executions.
• Long time to analyze results.
• Increases time to release.
• Increases risk and reduces
flexibility during the cycle.
• Reduces innovation time
versus bug fixes time.
• QA isn’t part of the daily cycle
since they have limited
feedback to share.
• Testing holds back
innovation.
”End of
Cycle”
Testing

6. 6 | Automated Testing in DevOps perfecto.ioI. The State of Digital Transformation
Mobile Testing Challenges
Complex test
creation leads to
high percentage of
manual testing
Untrusted test
automation leads
to high percentage
of manual testing
Hard and costly
to manage in-
house lab
Outdated mobile
test Lab
Reporting
noise (too
many false
negatives)
resulting in
irrelevant test
feedback
Unable to execute
parallel tests and
scale with the
right amount of
coverage

7. Key Benefits Of a Mature DevOps
State of DevOps Report, 2018 - DORA

8. Value of CT and DevOps
State of DevOps Report, 2018 - DORA
State of Software Testing - Panaya

9. Licenses In-use Disconnected
Orchestration issues
Network Availability Data
Backend issues
Objects Codding Time Other
Scripts & FW issues
• 10% of devices, causing 80% of lab issues
Networking Stability Config Lock
Lab issues
Reasons for “Distrustful automation” (Functional UI Testing, mobile)
52% success rate
Backend
30%
Lab
15%
Orchestration
15%
Scripts & FW
40%
80% of issues have a pattern

10. Licenses In-use Disconnected
Orchestration issues
Network Availability Data
Backend issues
Objects Codding Time Other
Scripts & FW issues
• 10% of devices, causing 80% of lab issues
Networking Stability Config Lock
Lab issues
Smart insights are key for Test Automation success
Backen
d
30%
Lab
15%
Orchestration
15%
Scripts & FW
40%
What’s
wrong
with my
Scripts
What’s wrong
with my
Executions
What’s wrong
with my Lab
What’s wrong
with my
Backend

11. 11 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Let’s Focus on Testing
Testing Is Led By 3 Main Personas
B U S I N E S S T E S T E R
S D E T ( S O F T W A R E D E V E L O P E R I N T E S T I N G )
S O F T W A R E D E V E L O P E R
Performs exploratory and structured manual testing
Creates functional and nonfunctional code-based test
automation scripts (via Selenium, Appium, etc.)
Creates unit and build acceptance tests

12. How should testing types be dispersed among the three personas
and throughout the DevOps pipeline?
SDETDeveloper
Business Tester:
Manual?
BDD?
Codeless?

13. 13 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
2/28/2019 13© 2015, Perfecto Mobile Ltd. All Rights Reserved.
BDD/Codeless
Scripting
Flaky and
common test
failures and
complex manual
test cases
Interactive Tests
UI/UX manual tests
High Skilled
(Dev/SDET)
code-based
scripting
Developers and
SDETs
Business Testers
Ownership
Business Testers
Ownership
Balancing Test Creation for the Three Different
Personas with the Right Tools

14. 14 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
2/28/2019 14© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Interactive Tests
UI/UX manual tests
Developers and
SDETs
Business Testers
Ownership
Business Testers
Ownership
Balancing Test Creation for the Three Different
Personas with the Right Tools

15. Test AUTOMATION– What to Automate?
1. What’s the test engineer’s gut feeling 😊😊
2. Risk calculated as probability to occur and
impact to customers
3. Value – does the test provide new
information and, if failed, how much time to
fix?
4. Cost efficiency to develop – how long does
it take to develop and how easy is it to
script?
5. History of test – volume of historical
failures in related areas and frequency of
breaks
Source: Angie Jones

16. Some Recommended CT Measurements (KPIs)
• How fast are testing activities moving, and what is slowing down these activities?
• Test flakiness
• Test duration
• % of automated vs. manual tests
• Application quality measurements
• # of escaped defects and in which areas
• MTTD – mean time to detection of defect
• Build quality
• Pipeline efficiency measurements
• # of user stories implemented per iteration
• Test automation as part of DoD across iterations
• Broken builds with categories
• CI length trending
• Lab availability and utilization
• Quality costs measurements
• Operational costs, lab availability issues
• Cost of hardware/software
• Costs of defects by severity and stage

17. Key ML Use Cases In Test Automation
• Recognize objects
• Transcribe speech – Accessibility
• Make quality related decisions based on data
• Identify Trends and/or Patterns
• Security use cases – Identify signatures e.g.
2/28/2019 17© 2015, Perfecto Mobile Ltd. All Rights Reserved.

18. 18 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Test Authoring Workflow and
Skillset Required
 Define manual test scenarios, BDD & user
stories style scenarios
 Create scripts in Java/JS etc. from within IDEs
(IntelliJ, Eclipse)
 Define Objects using Object Spy tools, DOM
viewers etc.
 Page based test steps creation
 Insert Visual validations and Assertions
 Typically takes longer to develop, more complex
(~6 Hours per test)
 Medium-High code development skills required
Test Maintenance
 Test changes are required proactively, tests are more
error-prone to changes in production, objects
 Tests are managed and maintained in an SCM tool
(GIT, Perforce)
 Tests are recorded with no coding in most
cases
 Codeless tools UI used for creation
 Objects are “learned” and generated on-the-fly
 Test scenarios are often less structured and
more exploratory based (flow based)
 Time to author is shorter, can be ~1 hour per
test
 Test reusability is easy
 Business tester, lower to no-coding skills is
sufficient
 Tools support self-healing with auto test correction
 Local or cloud test versioning is used, no GIT
integration
SDET/Developers Business Testers (&Developers?)
Mindset and Workflow Changes in Test Automation

19. 19 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Tools Maturity
 Highly mature, includes samples, best
practices, documentation
 Integrations exists to ALM tools, Defect
management etc.
Testing Types and App Types
Supported
 Functional, API, Load, others
 Mobile Native (Appium e.g.) and Desktop
Web (Selenium e.g.)
 Emerging technology, less mature, no well-
defined guidelines and practices (e.g. guide
to shift from standard to ML?)
 Web is more mature than mobile codeless,
basic integrations only
 Mostly functional (E2E) and basic API
 Most support web, mobile is lagging behind
SDET/Developers Business Testers (&Developers?)
Test Execution
 Configured environment using tools like
TestNG data provider
 Execution done locally, via CI, cloud-based
 Execution management is built-in the
codeless tools
 Execution done locally, via CI, cloud-based
Mindset and Workflow Changes in Test Automation

20. 20 | Solving the 3 Biggest Questions in Continuous Testing perfecto.io
Perfecto’s Smart Automation for Continuous Testing
SMART Test Execution
Fast and parallel test execution with
multi-team orchestration abilities and
management.
SMART ML-Driven Test
Reporting and Analysis
"Single pane of glass" provides
visibility and scales to support
millions of test results.
SMART Test Creation
Automated creation that matches your
team’s skillset (Appium, Espresso,
XCUITest, Quantum BDD).
SMART CLOUD Lab
Always on and stable. Always up
to date. Supports all mobile OS
and platforms.
The Perfecto
human factor
increases your
chances to succeed.
Black Belt
Testing Experts
Training
Dedicated
Success Manager
24/7 VIP
Support

21. Q & A

22. Thank you