O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

State of DevOps 2018: Continuous Testing is Required for DevOps Success

583 visualizações

Publicada em

Led by Nicole Forsgren, Gene Kim, and Jez Humble, DORA (DevOps Research and Assessment) is recognized as the leading authority on the factors driving DevOps success. Their upcoming “2018 Accelerate State of DevOps” report is the result of five years of research, with over 30,000 data points from thousands of companies, to understand precisely what practices enable teams to deliver better software faster.

Join Nicole Forsgren, PhD, CEO and Chief Scientist at DORA and Wolfgang Platz, Founder and Chief Strategy Officer at Tricentis, to explore what this year’s report reveals about the role of Continuous Testing in DevOps success. The focus will be on helping IT leaders understand:

Why and how high-performing teams make Continuous Testing a linchpin of their DevOps process
What specific strategies and practices enable high-performing teams to deliver faster and dedicate more resources to innovation
Which quality-related technology and process investments have the greatest impact on DevOps success

Publicada em: Tecnologia
  • Seja o primeiro a comentar

State of DevOps 2018: Continuous Testing is Required for DevOps Success

  1. 1. © 2018 by © 2018 by . Wolfgang Platz, Founder & Chief Strategy Officer The journey to Continuous Testing
  2. 2. © 2018 by employees 600+ locations Austria Germany Switzerland United Kingdom The Netherlands Poland USA Australia Singapore India founded 2008 customers Forbes Top 10 Global 2000 Companies 6 4 5 4 4 4 investment computer services major banks Australian banks life insurance diversified insurance support awards Leader Gartner MQ Leader Forrester Wave ASP Award 2016 & 2017
  3. 3. © 2018 by
  4. 4. © 2018 by PLAN Agile Dev Operationsmeets Dev Ops DevOps is all about Continuous Testing! http://devops.com/2015/02/02/devops-continuous-testing/ 4
  5. 5. © 2018 by Continuous testing is change It’s way more than just UI automation
  6. 6. © 2018 by Focus Area Exploratory Testing ! Explore ! Risk Based Prioritization Test Case Design CT Integration (CI, CD) UI Automation: Script-based UI Automation: Model-based API Testing Active Test Data Management Test Driven Service Virtualization ! Integrate ! Optimize ! Automate Test Management Manual Testing M1 M2 M3 M4 M5CTMM Continuous Testing Maturity Model
  7. 7. © 2018 by 7 DORA CTMM + Provides comprehensive DevOps assessment based on a thorough data analysis, best practices and benchmarks Covers entire DevOps cycle Is primarily a diagnosis instrument Provides comprehensive Continuous Testing assessment based on a best practices and benchmarks from Tricentis’ customer base Specific focus on Continuous Testing Is both a diagnosis and a therapy instrument
  8. 8. © 2018 by Customer Example: Core Banking Solution 8
  9. 9. © 2018 by > 80% Manual Testing < 20% Automated Testing UI API Today The reliance on manual testing is the #1 technical challenge in app development1. 1 … Source: World Quality Report 2015 - 2016 35% IT Budget Spend on QA & Testing1 $35B Annual Spend on Testing Services Legacy platforms require weeks to execute testing. 9
  10. 10. © 2018 by Focus Area Exploratory Testing ! Explore ! Risk Based Prioritization Test Case Design CT Integration (CI, CD) UI Automation: Script-based UI Automation: Model-based API Testing Active Test Data Management Test Driven Service Virtualization ! Integrate ! Optimize ! Automate Test Management Manual Testing M1 M2 M3 M4 M5CTMM Continuous Testing Maturity Model
  11. 11. © 2018 by Focus Area Exploratory Testing ! Explore ! Risk Based Prioritization Test Case Design CT Integration (CI, CD) UI Automation: Script-based UI Automation: Model-based API Testing Active Test Data Management Test Driven Service Virtualization ! Integrate ! Optimize ! Automate Test Management Manual Testing M1 M2 M3 M4 M5 Risk weighted functional structure Projected most relevant 600 test cases onto it UI test automation most relevant 600 test cases
  12. 12. © 2018 by Today’s reporting is based on numbers of test cases. What does it tell us? Version 7.5 42,278 10,086 910 Is 53,274 the right number of test cases? Can I go productive? 12
  13. 13. © 2018 by 13 Front wingGear box Tires Telemetry Software Steering Wheel Cassis & MonocoqueEngineRear wing BrakesExhaust
  14. 14. © 2018 by 14
  15. 15. © 2018 by BusinessRiskcoverage[%] Test Cases (sorted by contribution) t, $ 0% 20% 40% 60% 80% 100% 20% 40% 60% 80% 100%
  16. 16. © 2018 by We need to introduce a new currency in testing. 66% 9% 15% 10%Core Bank Risk Coverage [%] Risk Coverage 16
  17. 17. © 2018 by Continuous Testing requires Automation 17 80% Manual Testing 20% Automated Testing UI API Today Future Exploratory Tests Automated UI Tests API Tests Orchestrated Service Virtualization +85% - key enabler for high automation rates -
  18. 18. © 2018 by BusinessRiskcoverage[%] Test Cases (sorted by contribution) t, $ 0% 20% 40% 60% 80% 100% 20% 40% 60% 80% 100% Quickly obtain > 40% business risk coverage through automation (aka Heartbeat 1) - Rapid Automation Package M2 M2
  19. 19. © 2018 by If you always do what you’ve always done, you will always get what you’ve always got. 19 Albert Einstein, Relativity Theory Henry Ford, Ford T Automobile Mark Twain, Author Anthony Robbins, Author & NLP Trainer
  20. 20. © 2018 by Model-Based Browser Assert.AreEqual(“331,00", driver.FindElement(By.XPath("//table[@id='PriceList']/t body/tr[10]/td[2]/b")) .GetAttribute("value")); Browser Script-Based Code Synchronization Behavior Code Recovery Handling Code Data Driven Framework Code Keyword Driven Framework 1 2 3 4 20
  21. 21. © 2018 by 100 pax 3 weeks ManualTesters Test automation learning curves 21 100 pax 3 months “We will train them 3 months on Selenium based on our framework!” Head of QA Services “Go train them on Tricentis Tosca – you got 3 weeks!” 15% 85% With Tricentis Tosca, 85% of manual testers become productive automation specialists in 3 weeks. ManualTesters
  22. 22. © 2018 by Focus Area Exploratory Testing ! Explore ! Risk Based Prioritization Test Case Design CT Integration (CI, CD) UI Automation: Script-based UI Automation: Model-based API Testing Active Test Data Management Test Driven Service Virtualization ! Integrate ! Optimize ! Automate Test Management Manual Testing M1 M3 M4 M5M2 Introduced test case design … according to use-case relevance
  23. 23. © 2018 by Test Cases (sorted by contribution) 20% 40% 80% 100%60% t, $ BusinessRiskcoverage[%] 0% 20% 40% 60% 80% 100% M2 Risk coverage depends on application of test case design + efficiency of test automation access (UI or API) Push business risk coverage + maximize automation Optimize TestingM3 M5…
  24. 24. © 2018 by Focus Area Exploratory Testing ! Explore ! Risk Based Prioritization Test Case Design CT Integration (CI, CD) UI Automation: Script-based UI Automation: Model-based API Testing Active Test Data Management Test Driven Service Virtualization ! Integrate ! Optimize ! Automate Test Management Manual Testing M1 M3 M4 M5M2 Introduced test case design … according to use-case relevance Expand UI test automation (+ 500) + 900 test cases Extend coverage through API testing
  25. 25. © 2018 by Integration Tests Unit Tests UAT System Integration Tests … Automated Test The Agile test pyramid ←
  26. 26. © 2018 by Set-up Maintenance Future Test Automation will focus on API testing Task Runtime Timing 4x 6x 100+ x Factor API testing 25% 16% <1% progressive UI test automation 100% 100% 100% regressive
  27. 27. © 2018 by Integration Tests Unit Tests UAT System Integration Tests … API test automation The Agile test pyramid ← … UI Test automation
  28. 28. © 2018 by Focus Area Exploratory Testing ! Explore ! Risk Based Prioritization Test Case Design CT Integration (CI, CD) UI Automation: Script-based UI Automation: Model-based API Testing Active Test Data Management Test Driven Service Virtualization ! Integrate ! Optimize ! Automate Test Management Manual Testing M1 M3 M4 M5M2 Introduced test case design … according to use-case relevance Expand UI test automation (+ 500) + 900 test cases Extend coverage through API testing Implement synthetic test data gen
  29. 29. © 2018 by The challenge… § Complex (test) processes require test data in certain, well defined states. § Production data may not contain data for new functionality yet. § Production data may not be used due to privacy issues. § Test data may expire (e.g.: persons/contracts grow old). § Test data is altered through the test itself (consumption of test data).
  30. 30. © 2018 by Requirements Epics/ UserStories Test Data Design Test Case Data Test Data Basis Manual Test Case Automated Test Case Test Data Management
  31. 31. © 2018 by Focus Area Exploratory Testing ! Explore ! Risk Based Prioritization Test Case Design CT Integration (CI, CD) UI Automation: Script-based UI Automation: Model-based API Testing Active Test Data Management Test Driven Service Virtualization ! Integrate ! Optimize ! Automate Test Management Manual Testing M1 M3 M4 M5M2 Introduced test case design … according to use-case relevance Drive test execution through CI Expand UI test automation (+ 500) + 900 test cases Extend coverage through API testing Implement synthetic test data gen
  32. 32. © 2018 by DevOps Tool-Chain Epics Requirements Code/ Config Versioning Analyze & Build Test (Explore, Automate) Optimize, Automate, (incl. Virtualize) Staging & Deployment Monitoring XCode Container VM-config PLAN CODE BUILD CONTINUOUS TESTING DEPLOY MONITOR
  33. 33. © 2018 by Focus Area Exploratory Testing ! Explore ! Risk Based Prioritization Test Case Design CT Integration (CI, CD) UI Automation: Script-based UI Automation: Model-based API Testing Active Test Data Management Test Driven Service Virtualization ! Integrate ! Optimize ! Automate Test Management Manual Testing M1 M4 M5M3M2 Further expansion of test automation (+ 6,000); mainly API Implement Service Virtualization
  34. 34. © 2018 by App to be tested Tester performs user-input Tester verifies results Request Response System B System C Request Response Step 1 Step 2 Step 3 Step 4 Step 5 Example: A large portion of the app’s business logic can only be tested with systems B, C available. X X X
  35. 35. © 2018 by … requires discipline … requires Test Driven Service Virtualization Integration Tests Unit Tests UAT System Integration Tests ←The Agile test pyramid
  36. 36. © 2018 by Focus Area Exploratory Testing ! Explore ! Risk Based Prioritization Test Case Design CT Integration (CI, CD) UI Automation: Script-based UI Automation: Model-based API Testing Active Test Data Management Test Driven Service Virtualization ! Integrate ! Optimize ! Automate Test Management Manual Testing M1 M5M3 M4M2 DevOps Contract: Each use-case can be administered via API Introduced exploratory testing
  37. 37. © 2018 by 1 2 3 4 5 … … … … … … … Risks
  38. 38. © 2018 by Risks Exploratory Branching New testing ideas continually occur during exploratory testing. New Testing Idea Exploratory Testing Perfect counterpart to specification-based automated & manual testing. Test Cases vs. Scenarios Pre-specified inputs vs. hypothetical situation
  39. 39. © 2018 by Focus Area Exploratory Testing ! Explore ! Risk Based Prioritization Test Case Design CT Integration (CI, CD) UI Automation: Script-based UI Automation: Model-based API Testing Active Test Data Management Test Driven Service Virtualization ! Integrate ! Optimize ! Automate Test Management Manual Testing M1 M3 M4 M5M2
  40. 40. © 2018 by Testing Lifecycle Continuous Integration Shift Left Testing Continuous Delivery Continuous Testing Maturity Model M1 - M3 M3 - M4 M4 - M5 Roles & Responsibilities In-Team Collaboration Across-Team Collaboration Across-Stage Collaboration 2x 5x3x Initiation Collaboration Optimization Prevent testing to become the bottleneck for Digital Transformation Focus on product, people and process Scaling can increase your time-to-market up to 5x The first phase is achievable within 3 - 6 months Scaling Continuous Testing 3 to 6 months up to 12 months up to 18 months
  41. 41. © 2018 by Contact us Wolfgang Platz w.platz@tricentis.com
  42. 42. © 2018 by Questions Dr. Nicole Forsegren, Wolfgang Platz ?

×