How to Optimise Continuous Testing

Sauce Labs
Sauce LabsSr. Manager, Demand Generation & Operations
Achieving Continuous Testing through Test
Optimization and Automation
June 28th, 2018
Gerie Owen
VP, Knowledge and Innovation US
Giovanni Rago
Solutions Engineer, EMEA
Webinar Presenters
Agenda
Continuous Testing Test Optimization
Introduction
SummaryTest Automation Q&A
Agenda
Continuous Testing Test Optimization
Introduction
SummaryTest Automation Q&A
About
QualiTest
QA and testing is
all that we do!
World’s largest
pure play
testing
company
Over 3,000 testing
professionals
worldwide
Business
Assurance
QA &
Testing
Focus
Outcome
Based
Testing
(OBT)
Contractual guarantees for
quality improvement &
cost reduction
Managed
Testing
Solutions
(MTS)
Long term, SLA-based testing
outsourcing
Industry & Technology
Expert
Global
Service
Delivery
Model
On-Shore, Near-Shore,
Off-Shore and Crowd
Testing
Sauce Labs
Analytics
Comprehensive
Coverage
Secure
Most
Reliable
Experience
DebuggingHighly Scalable
Agenda
Continuous Testing Test Optimization
Introduction
SummaryTest Automation Q & A
ContinuousTesting
| ContinuousTesting is the basis for Continuous Delivery
| Continuous Delivery requires not only increased velocity, but more importantly
increased quality.
| Testing Cycles must be shortened yet quality must be increased.
| Increases in microservices architectures require high-quality functional testing including
automated API testing.
| Builds, integrations and deployments are automated; testing must be integrated into the
pipeline.
ContinuousTesting
| A core capability in Agile and DevOps where all testing activities run continuously in
an integrated fashion with development and delivery. Immediate bug fixing is
enabled, test environments are provisioned instantly, and unit, functional, and
nonfunctional tests are run in an automated way, orchestrated by Continuous
Integration and Continuous Delivery tools. The ForresterWave™: ModernApplication
FunctionalTest AutomationTools, Q4 2016
ContinuousTesting
| ContinuousTesting requires risk analysis and process improvement as well as
implementation of automation throughout the entire software development process.
| “Shift Left” i.e., moving automated testing to the lowest levels of the testing pyramid
is an important part of the approach.
ContinuousTesting vsTest Automation
| ContinuousTesting requires automation, however, it encompasses much more.
| ContinuousTesting is an approach to managing risk by focusing not only on improving
testing efficiency but more importantly, increasing the effectiveness of our test
processes.
| Assesses business risk coverage as its primary goal
| Establishes a safety net that helps the team protect the user experience
| Requires a stable test environment to be available on demand
| Integrates seamlessly into the software delivery pipeline and DevOps toolchain
| Delivers actionable feedback appropriate for each stage of the delivery pipeline
Attributes of ContinuousTesting
ContinuousTesting
| ContinuousTesting Minimizes Business RiskThrough:
| Test Optimization
| Automation throughout the Continuous Delivery Pipeline
Agenda
Continuous Testing Test Optimization
Introduction
SummaryTest Automation Q & A
Test Optimization
| Test Optimization encompasses risk assessment of the entire test scope:
| Analyze test case for quality, effectiveness, automation potential
| Evaluating the test suites to determine if they are providing the maximum amount of coverage
with the fewest number of test cases.
OptimizeTest Scope
| Select end-to-end tests that focus on the critical and high-risk features of the
application
| Understand all integrations involved in critical user workflows including the
technologies employed in those applications (web, mobile, message/API-layer etc.)
| Implement service virtualization for integrations and components that aren’t readily
available in order to run end-to-end test continuously
| Use the most realistic data possible for each test, include synthetic data generation
where needed
| Include exploratory testing to find user-experience and other bugs not easily found by
automated tests
OptimizingTest Cases
| Tests should be focused on the functionality they are validating.
| Tests should be as simple as possible; the more complex the test, the more validation
is required for the test code itself.
| Even though service virtualization can be used to fill in for missing components in
integration testing, tests should as independent as possible.
OptimizingTest Suites
| Include test cases that validate high risk areas of code
| Include test cases that validate customer-centric features
| Include test cases that validate critical business workflows
| Use FunctionalAutomation tools that create multi-purpose scripts
| Cross-PlatformTesting
• Scripts should run on Desktop,Web, Mobile & Server
| Cross-BrowserTesting
• Scripts should run on multiple browsers including Chrome, Edge, Safari and any others required
| PerformanceTesting
• Scripts should collect performance statistics including page loads etc.
Agenda
Continuous Testing Test Optimization
Summary
Introduction
Q & ATest Automation
Test Automation
| Test automation is critical for Continuous Integration and Continuous Delivery
| Fast feedback is not just a nice-to-have
| Automated tests require a strategy
| Which cases to automate?
| How to automate them?
| When to run automated tests? (At which point in the delivery pipeline?)
| An automation framework can be a software project of its own
| Specific technical know-how is needed
| Maintenance needs to be accounted for
| Good software engineering practices apply
Dependable, Consistent, ReproducibleTests
| Test environment should mimic production to the highest degree possible
| Having a system free of artifacts or stale data is paramount for reproducible results
| Every session should be “fresh”: clean system cache, browser cache, cookies, etc.
| Make sure your execution workstation and system under test have enough bandwidth
and memory to support the number of simultaneous testing threads
Beware the Maintenance Overhead
| …big deal in UI testing!
| Can explode and needs to be managed carefully
| Can be kept under control by following best practices
| PageObject model
| In general: loose coupling, good software development practices.
Enabling BetterTesting
| Strive to build up speed
| Monitor test performance over time
| Leverage parallelism whenever possible
| Fast test suites mean more testing, better QA
| Trust in your framework is precious
| Flaky tests produce noise, noisy builds reduce user compliance
| An automation friendly SUT makes a big difference
| Get your developers to help out!
Enabling BetterTesting
| Use the right tool for the job
| Keep functional tests separate from performance tests
| If reliable testing can be achieved faster with a unit test, use a unit test
| Place functional tests in the appropriate place in the build
| Create shortest and most succinct tests possible by creating a state closest to the
single feature being tested
| API Calls to insert data
| Deep linking
| Setup andTeardown Methods
| …enable atomic and autonomous tests!
KeepingTests Atomic
| Atomic
| Test a single feature
| Short and succinct
KeepingTests Atomic
| Example:
KeepingTests Autonomous
| Autonomous
| Tests should run independently from other tests
| One test’s actions should not drive another test’s acceptance criteria
KeepingTests Autonomous
| (Bad) Example:
KeepingTests Autonomous
| (Good) Example:
Benefits of Atomic and AutonomousTests
| Easier to debug and maintain
| Atomic -> Concise tests tend to be less complex
| Autonomous -> Low coupling ensures tests are “sandboxed”
| Easier to parallelize
| Atomic + Autonomous -> Independent tests testing independent features
Why Run in Parallel
| High test coverage across multiple key platforms is highly beneficial
| …but the resulting number of tests can be high, and execution time can explode
| Running tests sequentially is not an option anymore: parallelization is needed
Agenda
Continuous Testing Test Optimization
Summary
Introduction
Test Automation Q & A
Summary
|ContinuousTesting Focuses on Risk
|Optimize theTest Process
|AutomatedCheckpoints
|Automated feedback loops
|Integrate testing through the delivery pipeline including deployments at
each stage
|MatureTest Automation Enables ContinuousTesting
|Continued maintenance must be accounted for
|Sticking to best practices pays big dividends
|Parallelism enables high speed at high coverage
Agenda
Continuous Testing Test Optimization
Introduction
SummaryTest Automation Q & A
How to Optimise Continuous Testing
Thank you
1 de 36

Recomendados

Automation testing strategy, approach & planning por
Automation testing  strategy, approach & planningAutomation testing  strategy, approach & planning
Automation testing strategy, approach & planningSivaprasanthRentala1975
82.4K visualizações21 slides
Integration Monday - Logic App Patterns por
Integration Monday - Logic App PatternsIntegration Monday - Logic App Patterns
Integration Monday - Logic App PatternsBizTalk360
1.7K visualizações26 slides
Test Automation Framework Designs por
Test Automation Framework DesignsTest Automation Framework Designs
Test Automation Framework DesignsSauce Labs
167.1K visualizações56 slides
How to Design a Successful Test Automation Strategy por
How to Design a Successful Test Automation Strategy How to Design a Successful Test Automation Strategy
How to Design a Successful Test Automation Strategy Impetus Technologies
12.3K visualizações18 slides
Building a Test Automation Strategy for Success por
Building a Test Automation Strategy for SuccessBuilding a Test Automation Strategy for Success
Building a Test Automation Strategy for SuccessLee Barnes
7.4K visualizações32 slides
Test Automation in Agile por
Test Automation in AgileTest Automation in Agile
Test Automation in AgileAgile Testing Alliance
6.2K visualizações19 slides

Mais conteúdo relacionado

Mais procurados

Automation testing por
Automation testingAutomation testing
Automation testingBiswajit Pratihari
12.3K visualizações161 slides
DevOps & SRE at Google Scale por
DevOps & SRE at Google ScaleDevOps & SRE at Google Scale
DevOps & SRE at Google ScaleKaushik Bhattacharya
1.9K visualizações48 slides
Inverting The Testing Pyramid por
Inverting The Testing PyramidInverting The Testing Pyramid
Inverting The Testing PyramidNaresh Jain
53.2K visualizações46 slides
Cloud Native: what is it? Why? por
Cloud Native: what is it? Why?Cloud Native: what is it? Why?
Cloud Native: what is it? Why?Juan Pablo Genovese
1.4K visualizações21 slides
SonarQube - The leading platform for Continuous Code Quality por
SonarQube - The leading platform for Continuous Code QualitySonarQube - The leading platform for Continuous Code Quality
SonarQube - The leading platform for Continuous Code QualityLarry Nung
2.5K visualizações66 slides
Using Azure DevOps to continuously build, test, and deploy containerized appl... por
Using Azure DevOps to continuously build, test, and deploy containerized appl...Using Azure DevOps to continuously build, test, and deploy containerized appl...
Using Azure DevOps to continuously build, test, and deploy containerized appl...Adrian Todorov
674 visualizações61 slides

Mais procurados(20)

Automation testing por Biswajit Pratihari
Automation testingAutomation testing
Automation testing
Biswajit Pratihari12.3K visualizações
DevOps & SRE at Google Scale por Kaushik Bhattacharya
DevOps & SRE at Google ScaleDevOps & SRE at Google Scale
DevOps & SRE at Google Scale
Kaushik Bhattacharya1.9K visualizações
Inverting The Testing Pyramid por Naresh Jain
Inverting The Testing PyramidInverting The Testing Pyramid
Inverting The Testing Pyramid
Naresh Jain53.2K visualizações
Cloud Native: what is it? Why? por Juan Pablo Genovese
Cloud Native: what is it? Why?Cloud Native: what is it? Why?
Cloud Native: what is it? Why?
Juan Pablo Genovese1.4K visualizações
SonarQube - The leading platform for Continuous Code Quality por Larry Nung
SonarQube - The leading platform for Continuous Code QualitySonarQube - The leading platform for Continuous Code Quality
SonarQube - The leading platform for Continuous Code Quality
Larry Nung2.5K visualizações
Using Azure DevOps to continuously build, test, and deploy containerized appl... por Adrian Todorov
Using Azure DevOps to continuously build, test, and deploy containerized appl...Using Azure DevOps to continuously build, test, and deploy containerized appl...
Using Azure DevOps to continuously build, test, and deploy containerized appl...
Adrian Todorov674 visualizações
SRE-iously! Defining the Principles, Habits, and Practices of Site Reliabilit... por Tori Wieldt
SRE-iously! Defining the Principles, Habits, and Practices of Site Reliabilit...SRE-iously! Defining the Principles, Habits, and Practices of Site Reliabilit...
SRE-iously! Defining the Principles, Habits, and Practices of Site Reliabilit...
Tori Wieldt1.5K visualizações
Test Strategy-The real silver bullet in testing by Matthew Eakin por QA or the Highway
Test Strategy-The real silver bullet in testing by Matthew EakinTest Strategy-The real silver bullet in testing by Matthew Eakin
Test Strategy-The real silver bullet in testing by Matthew Eakin
QA or the Highway439 visualizações
Test Automation Best Practices (with SOA test approach) por Leonard Fingerman
Test Automation Best Practices (with SOA test approach)Test Automation Best Practices (with SOA test approach)
Test Automation Best Practices (with SOA test approach)
Leonard Fingerman21.1K visualizações
Cigniti Independent Software Testing Services por Cigniti Technologies Ltd
Cigniti Independent Software Testing ServicesCigniti Independent Software Testing Services
Cigniti Independent Software Testing Services
Cigniti Technologies Ltd2.4K visualizações
SRE 101 por Diego Pacheco
SRE 101SRE 101
SRE 101
Diego Pacheco1.1K visualizações
Test Automation Framework Design | www.idexcel.com por Idexcel Technologies
Test Automation Framework Design | www.idexcel.comTest Automation Framework Design | www.idexcel.com
Test Automation Framework Design | www.idexcel.com
Idexcel Technologies11.8K visualizações
Azure API Management por Daniel Toomey
Azure API ManagementAzure API Management
Azure API Management
Daniel Toomey5.2K visualizações
Developing a Testing Strategy for DevOps Success por DevOps.com
Developing a Testing Strategy for DevOps SuccessDeveloping a Testing Strategy for DevOps Success
Developing a Testing Strategy for DevOps Success
DevOps.com1.1K visualizações
Test Automation Strategies For Agile por Naresh Jain
Test Automation Strategies For AgileTest Automation Strategies For Agile
Test Automation Strategies For Agile
Naresh Jain54K visualizações
Leveraging Azure DevOps across the Enterprise por Andrew Kelleher
Leveraging Azure DevOps across the EnterpriseLeveraging Azure DevOps across the Enterprise
Leveraging Azure DevOps across the Enterprise
Andrew Kelleher529 visualizações
Performance Engineering Basics por Impetus Technologies
Performance Engineering BasicsPerformance Engineering Basics
Performance Engineering Basics
Impetus Technologies9K visualizações
Agile Testing Framework - The Art of Automated Testing por Dimitri Ponomareff
Agile Testing Framework - The Art of Automated TestingAgile Testing Framework - The Art of Automated Testing
Agile Testing Framework - The Art of Automated Testing
Dimitri Ponomareff29.8K visualizações
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |... por HostedbyConfluent
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...
HostedbyConfluent2.5K visualizações

Similar a How to Optimise Continuous Testing

Test automation and beyond developing an effective continuous test strategy d... por
Test automation and beyond developing an effective continuous test strategy d...Test automation and beyond developing an effective continuous test strategy d...
Test automation and beyond developing an effective continuous test strategy d...GerieOwen
111 visualizações29 slides
Automation Concepts por
Automation ConceptsAutomation Concepts
Automation ConceptsNishant Worah
10.9K visualizações29 slides
SAP Testing Services por
SAP Testing ServicesSAP Testing Services
SAP Testing Servicesr_shanki
4.2K visualizações16 slides
Does dev ops need continuous testing devops days des moines 2018 v1 por
Does dev ops need continuous testing devops days des moines 2018 v1Does dev ops need continuous testing devops days des moines 2018 v1
Does dev ops need continuous testing devops days des moines 2018 v1GerieOwen
104 visualizações23 slides
Continuous testing maximising velocity, quality and customer happiness por
Continuous testing maximising velocity, quality and customer happinessContinuous testing maximising velocity, quality and customer happiness
Continuous testing maximising velocity, quality and customer happinessPractiTest
755 visualizações35 slides
Atagg 2015 Test automation and effective continuous integration por
Atagg 2015 Test automation and effective continuous integrationAtagg 2015 Test automation and effective continuous integration
Atagg 2015 Test automation and effective continuous integrationAgile Testing Alliance
406 visualizações8 slides

Similar a How to Optimise Continuous Testing(20)

Test automation and beyond developing an effective continuous test strategy d... por GerieOwen
Test automation and beyond developing an effective continuous test strategy d...Test automation and beyond developing an effective continuous test strategy d...
Test automation and beyond developing an effective continuous test strategy d...
GerieOwen111 visualizações
Automation Concepts por Nishant Worah
Automation ConceptsAutomation Concepts
Automation Concepts
Nishant Worah10.9K visualizações
SAP Testing Services por r_shanki
SAP Testing ServicesSAP Testing Services
SAP Testing Services
r_shanki4.2K visualizações
Does dev ops need continuous testing devops days des moines 2018 v1 por GerieOwen
Does dev ops need continuous testing devops days des moines 2018 v1Does dev ops need continuous testing devops days des moines 2018 v1
Does dev ops need continuous testing devops days des moines 2018 v1
GerieOwen104 visualizações
Continuous testing maximising velocity, quality and customer happiness por PractiTest
Continuous testing maximising velocity, quality and customer happinessContinuous testing maximising velocity, quality and customer happiness
Continuous testing maximising velocity, quality and customer happiness
PractiTest755 visualizações
Atagg 2015 Test automation and effective continuous integration por Agile Testing Alliance
Atagg 2015 Test automation and effective continuous integrationAtagg 2015 Test automation and effective continuous integration
Atagg 2015 Test automation and effective continuous integration
Agile Testing Alliance406 visualizações
manual & automation testing por Bharat Dawar
manual & automation testingmanual & automation testing
manual & automation testing
Bharat Dawar117 visualizações
FUNTASY - Functional testing automated system por Qualitest
FUNTASY - Functional testing automated systemFUNTASY - Functional testing automated system
FUNTASY - Functional testing automated system
Qualitest656 visualizações
Test Automation por Kiran Rayachoti
Test AutomationTest Automation
Test Automation
Kiran Rayachoti528 visualizações
Test Automation for QTP por Kiran Rayachoti
Test Automation for QTPTest Automation for QTP
Test Automation for QTP
Kiran Rayachoti915 visualizações
Aspires Testing Services Presentation por Vasanthmusic
Aspires Testing Services PresentationAspires Testing Services Presentation
Aspires Testing Services Presentation
Vasanthmusic1K visualizações
Upstream testing. por Mindtree Ltd.
Upstream testing.Upstream testing.
Upstream testing.
Mindtree Ltd. 730 visualizações
Qspectrum - Cloud based Test Automation Solutions por QSpectrum
Qspectrum - Cloud based Test Automation SolutionsQspectrum - Cloud based Test Automation Solutions
Qspectrum - Cloud based Test Automation Solutions
QSpectrum671 visualizações
Automation Best Practices.pptx por pavelpopov43
Automation Best Practices.pptxAutomation Best Practices.pptx
Automation Best Practices.pptx
pavelpopov4318 visualizações
Software Testing por Kiran Kumar
Software TestingSoftware Testing
Software Testing
Kiran Kumar1.3K visualizações
Guideto Successful Application Test Automation por aimshigh7
Guideto Successful Application Test AutomationGuideto Successful Application Test Automation
Guideto Successful Application Test Automation
aimshigh76.5K visualizações
Software Test Automation por Josh Case
Software Test AutomationSoftware Test Automation
Software Test Automation
Josh Case564 visualizações
Test Orchestration in DevOps por Knoldus Inc.
Test Orchestration in DevOps Test Orchestration in DevOps
Test Orchestration in DevOps
Knoldus Inc.90 visualizações
Firmware Application Test Automation por Indium Software
Firmware Application Test AutomationFirmware Application Test Automation
Firmware Application Test Automation
Indium Software377 visualizações
Introduction to Automation Testing por Archana Krushnan
Introduction to Automation TestingIntroduction to Automation Testing
Introduction to Automation Testing
Archana Krushnan8.7K visualizações

Mais de Sauce Labs

Simplify Salesforce Testing with AI-Driven Codeless Tools por
Simplify Salesforce Testing with AI-Driven Codeless ToolsSimplify Salesforce Testing with AI-Driven Codeless Tools
Simplify Salesforce Testing with AI-Driven Codeless ToolsSauce Labs
649 visualizações17 slides
Testing on Mobile Devices with Location Services por
Testing on Mobile Devices with Location ServicesTesting on Mobile Devices with Location Services
Testing on Mobile Devices with Location ServicesSauce Labs
767 visualizações57 slides
Your Framework for Success: introduction to JavaScript Testing at Scale por
Your Framework for Success: introduction to JavaScript Testing at ScaleYour Framework for Success: introduction to JavaScript Testing at Scale
Your Framework for Success: introduction to JavaScript Testing at ScaleSauce Labs
270 visualizações20 slides
Automating Hybrid Applications with Appium por
Automating Hybrid Applications with AppiumAutomating Hybrid Applications with Appium
Automating Hybrid Applications with AppiumSauce Labs
351 visualizações103 slides
Quality at Speed: More API Testing, Less UI Testing por
Quality at Speed: More API Testing, Less UI TestingQuality at Speed: More API Testing, Less UI Testing
Quality at Speed: More API Testing, Less UI TestingSauce Labs
546 visualizações30 slides
Creating Digital Confidence with Test Automation por
Creating Digital Confidence with Test AutomationCreating Digital Confidence with Test Automation
Creating Digital Confidence with Test AutomationSauce Labs
356 visualizações16 slides

Mais de Sauce Labs(20)

Simplify Salesforce Testing with AI-Driven Codeless Tools por Sauce Labs
Simplify Salesforce Testing with AI-Driven Codeless ToolsSimplify Salesforce Testing with AI-Driven Codeless Tools
Simplify Salesforce Testing with AI-Driven Codeless Tools
Sauce Labs649 visualizações
Testing on Mobile Devices with Location Services por Sauce Labs
Testing on Mobile Devices with Location ServicesTesting on Mobile Devices with Location Services
Testing on Mobile Devices with Location Services
Sauce Labs767 visualizações
Your Framework for Success: introduction to JavaScript Testing at Scale por Sauce Labs
Your Framework for Success: introduction to JavaScript Testing at ScaleYour Framework for Success: introduction to JavaScript Testing at Scale
Your Framework for Success: introduction to JavaScript Testing at Scale
Sauce Labs270 visualizações
Automating Hybrid Applications with Appium por Sauce Labs
Automating Hybrid Applications with AppiumAutomating Hybrid Applications with Appium
Automating Hybrid Applications with Appium
Sauce Labs351 visualizações
Quality at Speed: More API Testing, Less UI Testing por Sauce Labs
Quality at Speed: More API Testing, Less UI TestingQuality at Speed: More API Testing, Less UI Testing
Quality at Speed: More API Testing, Less UI Testing
Sauce Labs546 visualizações
Creating Digital Confidence with Test Automation por Sauce Labs
Creating Digital Confidence with Test AutomationCreating Digital Confidence with Test Automation
Creating Digital Confidence with Test Automation
Sauce Labs356 visualizações
Just Enough (Automated) Testing por Sauce Labs
Just Enough (Automated) TestingJust Enough (Automated) Testing
Just Enough (Automated) Testing
Sauce Labs254 visualizações
Using Axe to Add Accessibility Checks to Your Existing Selenium Tests por Sauce Labs
Using Axe to Add Accessibility Checks to Your Existing Selenium TestsUsing Axe to Add Accessibility Checks to Your Existing Selenium Tests
Using Axe to Add Accessibility Checks to Your Existing Selenium Tests
Sauce Labs506 visualizações
How Open Source Helps to Bring Back Product Obsession por Sauce Labs
How Open Source Helps to Bring Back Product ObsessionHow Open Source Helps to Bring Back Product Obsession
How Open Source Helps to Bring Back Product Obsession
Sauce Labs292 visualizações
Webinar: A Sneak Peek at Selenium 4 with Simon Stewart por Sauce Labs
Webinar: A Sneak Peek at Selenium 4 with Simon StewartWebinar: A Sneak Peek at Selenium 4 with Simon Stewart
Webinar: A Sneak Peek at Selenium 4 with Simon Stewart
Sauce Labs637 visualizações
[Deu] Test Automatisierung Mit Web Driver.io por Sauce Labs
[Deu] Test Automatisierung Mit Web Driver.io[Deu] Test Automatisierung Mit Web Driver.io
[Deu] Test Automatisierung Mit Web Driver.io
Sauce Labs941 visualizações
Accelerating Innovation: Leveraging Open Source to Optimize Your Shift-Left I... por Sauce Labs
Accelerating Innovation: Leveraging Open Source to Optimize Your Shift-Left I...Accelerating Innovation: Leveraging Open Source to Optimize Your Shift-Left I...
Accelerating Innovation: Leveraging Open Source to Optimize Your Shift-Left I...
Sauce Labs363 visualizações
Accelerating Your Digital Agenda with Continuous Testing ft. Forrester por Sauce Labs
Accelerating Your Digital Agenda with Continuous Testing ft. ForresterAccelerating Your Digital Agenda with Continuous Testing ft. Forrester
Accelerating Your Digital Agenda with Continuous Testing ft. Forrester
Sauce Labs557 visualizações
How to Measure Success in Continuous Testing por Sauce Labs
How to Measure Success in Continuous TestingHow to Measure Success in Continuous Testing
How to Measure Success in Continuous Testing
Sauce Labs483 visualizações
From Zero to 2.7 Million - How Verizon Media Embraced Open Source to Accelera... por Sauce Labs
From Zero to 2.7 Million - How Verizon Media Embraced Open Source to Accelera...From Zero to 2.7 Million - How Verizon Media Embraced Open Source to Accelera...
From Zero to 2.7 Million - How Verizon Media Embraced Open Source to Accelera...
Sauce Labs403 visualizações
5 Steps to Jump Start Your Test Automation por Sauce Labs
5 Steps to Jump Start Your Test Automation5 Steps to Jump Start Your Test Automation
5 Steps to Jump Start Your Test Automation
Sauce Labs716 visualizações
Sauce Labs Webinar: Rising Importance of Software Testing por Sauce Labs
Sauce Labs Webinar: Rising Importance of Software TestingSauce Labs Webinar: Rising Importance of Software Testing
Sauce Labs Webinar: Rising Importance of Software Testing
Sauce Labs353 visualizações
BDD With Selenide by Hima Bindu Peteti por Sauce Labs
BDD With Selenide by Hima Bindu PetetiBDD With Selenide by Hima Bindu Peteti
BDD With Selenide by Hima Bindu Peteti
Sauce Labs555 visualizações
Closer To the Metal - Why and How We Use XCTest and Espresso by Mario Negro P... por Sauce Labs
Closer To the Metal - Why and How We Use XCTest and Espresso by Mario Negro P...Closer To the Metal - Why and How We Use XCTest and Espresso by Mario Negro P...
Closer To the Metal - Why and How We Use XCTest and Espresso by Mario Negro P...
Sauce Labs321 visualizações
Continuous Delivery for "Mature" Codebases by Melisa Benua por Sauce Labs
Continuous Delivery for "Mature" Codebases by Melisa BenuaContinuous Delivery for "Mature" Codebases by Melisa Benua
Continuous Delivery for "Mature" Codebases by Melisa Benua
Sauce Labs113 visualizações

Último

Cencora Executive Symposium por
Cencora Executive SymposiumCencora Executive Symposium
Cencora Executive Symposiummarketingcommunicati21
159 visualizações14 slides
The Power of Generative AI in Accelerating No Code Adoption.pdf por
The Power of Generative AI in Accelerating No Code Adoption.pdfThe Power of Generative AI in Accelerating No Code Adoption.pdf
The Power of Generative AI in Accelerating No Code Adoption.pdfSaeed Al Dhaheri
32 visualizações18 slides
2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue por
2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue
2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlueShapeBlue
147 visualizações23 slides
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f... por
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...TrustArc
170 visualizações29 slides
Why and How CloudStack at weSystems - Stephan Bienek - weSystems por
Why and How CloudStack at weSystems - Stephan Bienek - weSystemsWhy and How CloudStack at weSystems - Stephan Bienek - weSystems
Why and How CloudStack at weSystems - Stephan Bienek - weSystemsShapeBlue
238 visualizações13 slides
Future of AR - Facebook Presentation por
Future of AR - Facebook PresentationFuture of AR - Facebook Presentation
Future of AR - Facebook PresentationRob McCarty
64 visualizações27 slides

Último(20)

The Power of Generative AI in Accelerating No Code Adoption.pdf por Saeed Al Dhaheri
The Power of Generative AI in Accelerating No Code Adoption.pdfThe Power of Generative AI in Accelerating No Code Adoption.pdf
The Power of Generative AI in Accelerating No Code Adoption.pdf
Saeed Al Dhaheri32 visualizações
2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue por ShapeBlue
2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue
2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue
ShapeBlue147 visualizações
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f... por TrustArc
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc170 visualizações
Why and How CloudStack at weSystems - Stephan Bienek - weSystems por ShapeBlue
Why and How CloudStack at weSystems - Stephan Bienek - weSystemsWhy and How CloudStack at weSystems - Stephan Bienek - weSystems
Why and How CloudStack at weSystems - Stephan Bienek - weSystems
ShapeBlue238 visualizações
Future of AR - Facebook Presentation por Rob McCarty
Future of AR - Facebook PresentationFuture of AR - Facebook Presentation
Future of AR - Facebook Presentation
Rob McCarty64 visualizações
Business Analyst Series 2023 - Week 4 Session 8 por DianaGray10
Business Analyst Series 2023 -  Week 4 Session 8Business Analyst Series 2023 -  Week 4 Session 8
Business Analyst Series 2023 - Week 4 Session 8
DianaGray10123 visualizações
Optimizing Communication to Optimize Human Behavior - LCBM por Yaman Kumar
Optimizing Communication to Optimize Human Behavior - LCBMOptimizing Communication to Optimize Human Behavior - LCBM
Optimizing Communication to Optimize Human Behavior - LCBM
Yaman Kumar38 visualizações
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue por ShapeBlue
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlueCloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue
ShapeBlue138 visualizações
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti... por ShapeBlue
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
DRaaS using Snapshot copy and destination selection (DRaaS) - Alexandre Matti...
ShapeBlue139 visualizações
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ... por ShapeBlue
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...
ShapeBlue126 visualizações
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or... por ShapeBlue
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...
ShapeBlue198 visualizações
Initiating and Advancing Your Strategic GIS Governance Strategy por Safe Software
Initiating and Advancing Your Strategic GIS Governance StrategyInitiating and Advancing Your Strategic GIS Governance Strategy
Initiating and Advancing Your Strategic GIS Governance Strategy
Safe Software176 visualizações
LLMs in Production: Tooling, Process, and Team Structure por Aggregage
LLMs in Production: Tooling, Process, and Team StructureLLMs in Production: Tooling, Process, and Team Structure
LLMs in Production: Tooling, Process, and Team Structure
Aggregage42 visualizações
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlue por ShapeBlue
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlueWhat’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlue
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlue
ShapeBlue263 visualizações
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue por ShapeBlue
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlueVNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue
ShapeBlue203 visualizações
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R... por ShapeBlue
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...
ShapeBlue173 visualizações
"Surviving highload with Node.js", Andrii Shumada por Fwdays
"Surviving highload with Node.js", Andrii Shumada "Surviving highload with Node.js", Andrii Shumada
"Surviving highload with Node.js", Andrii Shumada
Fwdays56 visualizações
Redefining the book supply chain: A glimpse into the future - Tech Forum 2023 por BookNet Canada
Redefining the book supply chain: A glimpse into the future - Tech Forum 2023Redefining the book supply chain: A glimpse into the future - Tech Forum 2023
Redefining the book supply chain: A glimpse into the future - Tech Forum 2023
BookNet Canada44 visualizações
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda... por ShapeBlue
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...
ShapeBlue161 visualizações

How to Optimise Continuous Testing

  • 1. Achieving Continuous Testing through Test Optimization and Automation June 28th, 2018
  • 2. Gerie Owen VP, Knowledge and Innovation US Giovanni Rago Solutions Engineer, EMEA Webinar Presenters
  • 3. Agenda Continuous Testing Test Optimization Introduction SummaryTest Automation Q&A
  • 4. Agenda Continuous Testing Test Optimization Introduction SummaryTest Automation Q&A
  • 5. About QualiTest QA and testing is all that we do! World’s largest pure play testing company Over 3,000 testing professionals worldwide Business Assurance QA & Testing Focus Outcome Based Testing (OBT) Contractual guarantees for quality improvement & cost reduction Managed Testing Solutions (MTS) Long term, SLA-based testing outsourcing Industry & Technology Expert Global Service Delivery Model On-Shore, Near-Shore, Off-Shore and Crowd Testing
  • 7. Agenda Continuous Testing Test Optimization Introduction SummaryTest Automation Q & A
  • 8. ContinuousTesting | ContinuousTesting is the basis for Continuous Delivery | Continuous Delivery requires not only increased velocity, but more importantly increased quality. | Testing Cycles must be shortened yet quality must be increased. | Increases in microservices architectures require high-quality functional testing including automated API testing. | Builds, integrations and deployments are automated; testing must be integrated into the pipeline.
  • 9. ContinuousTesting | A core capability in Agile and DevOps where all testing activities run continuously in an integrated fashion with development and delivery. Immediate bug fixing is enabled, test environments are provisioned instantly, and unit, functional, and nonfunctional tests are run in an automated way, orchestrated by Continuous Integration and Continuous Delivery tools. The ForresterWave™: ModernApplication FunctionalTest AutomationTools, Q4 2016
  • 10. ContinuousTesting | ContinuousTesting requires risk analysis and process improvement as well as implementation of automation throughout the entire software development process. | “Shift Left” i.e., moving automated testing to the lowest levels of the testing pyramid is an important part of the approach.
  • 11. ContinuousTesting vsTest Automation | ContinuousTesting requires automation, however, it encompasses much more. | ContinuousTesting is an approach to managing risk by focusing not only on improving testing efficiency but more importantly, increasing the effectiveness of our test processes.
  • 12. | Assesses business risk coverage as its primary goal | Establishes a safety net that helps the team protect the user experience | Requires a stable test environment to be available on demand | Integrates seamlessly into the software delivery pipeline and DevOps toolchain | Delivers actionable feedback appropriate for each stage of the delivery pipeline Attributes of ContinuousTesting
  • 13. ContinuousTesting | ContinuousTesting Minimizes Business RiskThrough: | Test Optimization | Automation throughout the Continuous Delivery Pipeline
  • 14. Agenda Continuous Testing Test Optimization Introduction SummaryTest Automation Q & A
  • 15. Test Optimization | Test Optimization encompasses risk assessment of the entire test scope: | Analyze test case for quality, effectiveness, automation potential | Evaluating the test suites to determine if they are providing the maximum amount of coverage with the fewest number of test cases.
  • 16. OptimizeTest Scope | Select end-to-end tests that focus on the critical and high-risk features of the application | Understand all integrations involved in critical user workflows including the technologies employed in those applications (web, mobile, message/API-layer etc.) | Implement service virtualization for integrations and components that aren’t readily available in order to run end-to-end test continuously | Use the most realistic data possible for each test, include synthetic data generation where needed | Include exploratory testing to find user-experience and other bugs not easily found by automated tests
  • 17. OptimizingTest Cases | Tests should be focused on the functionality they are validating. | Tests should be as simple as possible; the more complex the test, the more validation is required for the test code itself. | Even though service virtualization can be used to fill in for missing components in integration testing, tests should as independent as possible.
  • 18. OptimizingTest Suites | Include test cases that validate high risk areas of code | Include test cases that validate customer-centric features | Include test cases that validate critical business workflows | Use FunctionalAutomation tools that create multi-purpose scripts | Cross-PlatformTesting • Scripts should run on Desktop,Web, Mobile & Server | Cross-BrowserTesting • Scripts should run on multiple browsers including Chrome, Edge, Safari and any others required | PerformanceTesting • Scripts should collect performance statistics including page loads etc.
  • 19. Agenda Continuous Testing Test Optimization Summary Introduction Q & ATest Automation
  • 20. Test Automation | Test automation is critical for Continuous Integration and Continuous Delivery | Fast feedback is not just a nice-to-have | Automated tests require a strategy | Which cases to automate? | How to automate them? | When to run automated tests? (At which point in the delivery pipeline?) | An automation framework can be a software project of its own | Specific technical know-how is needed | Maintenance needs to be accounted for | Good software engineering practices apply
  • 21. Dependable, Consistent, ReproducibleTests | Test environment should mimic production to the highest degree possible | Having a system free of artifacts or stale data is paramount for reproducible results | Every session should be “fresh”: clean system cache, browser cache, cookies, etc. | Make sure your execution workstation and system under test have enough bandwidth and memory to support the number of simultaneous testing threads
  • 22. Beware the Maintenance Overhead | …big deal in UI testing! | Can explode and needs to be managed carefully | Can be kept under control by following best practices | PageObject model | In general: loose coupling, good software development practices.
  • 23. Enabling BetterTesting | Strive to build up speed | Monitor test performance over time | Leverage parallelism whenever possible | Fast test suites mean more testing, better QA | Trust in your framework is precious | Flaky tests produce noise, noisy builds reduce user compliance | An automation friendly SUT makes a big difference | Get your developers to help out!
  • 24. Enabling BetterTesting | Use the right tool for the job | Keep functional tests separate from performance tests | If reliable testing can be achieved faster with a unit test, use a unit test | Place functional tests in the appropriate place in the build | Create shortest and most succinct tests possible by creating a state closest to the single feature being tested | API Calls to insert data | Deep linking | Setup andTeardown Methods | …enable atomic and autonomous tests!
  • 25. KeepingTests Atomic | Atomic | Test a single feature | Short and succinct
  • 27. KeepingTests Autonomous | Autonomous | Tests should run independently from other tests | One test’s actions should not drive another test’s acceptance criteria
  • 30. Benefits of Atomic and AutonomousTests | Easier to debug and maintain | Atomic -> Concise tests tend to be less complex | Autonomous -> Low coupling ensures tests are “sandboxed” | Easier to parallelize | Atomic + Autonomous -> Independent tests testing independent features
  • 31. Why Run in Parallel | High test coverage across multiple key platforms is highly beneficial | …but the resulting number of tests can be high, and execution time can explode | Running tests sequentially is not an option anymore: parallelization is needed
  • 32. Agenda Continuous Testing Test Optimization Summary Introduction Test Automation Q & A
  • 33. Summary |ContinuousTesting Focuses on Risk |Optimize theTest Process |AutomatedCheckpoints |Automated feedback loops |Integrate testing through the delivery pipeline including deployments at each stage |MatureTest Automation Enables ContinuousTesting |Continued maintenance must be accounted for |Sticking to best practices pays big dividends |Parallelism enables high speed at high coverage
  • 34. Agenda Continuous Testing Test Optimization Introduction SummaryTest Automation Q & A

Notas do Editor

  1. Read the Agenda During the webinar, I will focus more on medical device testing while David will elaborate on medical device localisation
  2. Read the Agenda During the webinar, I will focus more on medical device testing while David will elaborate on medical device localisation
  3. Read the Agenda During the webinar, I will focus more on medical device testing while David will elaborate on medical device localisation
  4. Read the Agenda During the webinar, I will focus more on medical device testing while David will elaborate on medical device localisation
  5. Regression Test Selection (RTS) is one of the most popular methods for test case suite optimization. This method divides the test suite into reusable test cases, retestable test cases and obsolete test cases. Apart from all these, it also creates new test cases that test the program for areas not covered in current test cases. Another great way to maintain the effectiveness of the regression test suite is to have a good tracking mechanism between the features under development . This should be a consistent activity in order to maintain the test suites effectively as it would help test manager to verify the feature checklist and validate the test coverage for a feature that is being developed in the release. The effectiveness of the regression test suite can be easily maintained by monitoring the changes to the test suite. A clearly outlined process will ensure that only tests that are useful to the entire testing strategy get added to the test suite, which ensures the efficiency and usability of the test harness at a high level. Considering periodic cleanup of old tests is another great approach to maintain effectiveness of a feature rich regression test suite. In this scenario, all the existing tests in the test suite need to be analyzed for their effectiveness in a specific scenario. Also, there will be scenarios where certain features will not be supported because of the different product direction. In such cases, the relevant regression test suites should also be eased out. It will ensure robustness of the regression test suite for a long period of time. You can also measure the effectiveness of regression test suites on a release-to-release basis. It will allow you to know the root cause for reduction in the effectiveness of the test harness if any, and enable you to take appropriate action on the same. Collection of some metrics and their analysis could also be useful when it comes to the effectiveness of the regression test suite. It will help you get good visibility on the effectiveness of the regression test suite. You can consider different metrics such as percentage of defects found by the regression tests suite, their importance, etc.
  6. Read the Agenda During the webinar, I will focus more on medical device testing while David will elaborate on medical device localisation
  7. Read the Agenda During the webinar, I will focus more on medical device testing while David will elaborate on medical device localisation
  8. Read the Agenda During the webinar, I will focus more on medical device testing while David will elaborate on medical device localisation