Accelerating Your Test Execution Pipeline
•Transferir como PPTX, PDF•
2 gostaram•448 visualizações
Our very own Bria Grangard will take you through the ways in which you can speed up your testing process. Check it out to learn about test frameworks, automation, parallel testing and more.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Accelerating Your Test Execution Pipeline
Semelhante a Accelerating Your Test Execution Pipeline (20)
Mais de SmartBear
Mais de SmartBear (20)
Último
Último (20)
Accelerating Your Test Execution Pipeline
- 1. Accelerating Your Test Execution Pipeline Thursday, May17th PorteShed,Galway
- 2. @Bria_Grangard Who Am I? • SmartBear Software • Automated UI functional testing tools & test management solutions • Stay in Touch! • @Bria_Grangard • Education • Went to Dartmouth: AB in Engineering, BE in Biomedical Engineering, MEM with a healthcare focus • What do I love to do? • Run, dance, play board games (Settlers of Catan anyone?)
- 4. We provide tools for development, testing, and operations teams to create great software, faster than ever. AccelerateSDLCWorkflows | ImproveQualityatEveryStage | RealizeRapidTime-to-Value • HQ in Boston, MA, USA, with 7 offices globally • Founded in 2009 • Open Source Innovator (Swagger & SoapUI) 6.5M+ Users 194 Countries 22K+ Companies TestComplete SoapUI Pro SwaggerHub CrossBrowserTesting QAComplete AlertSite
- 5. UI API Create Great Software, Without Tradeoffs Perform Code & Doc Review Collaborator Design, Develop, & Document APIs SwaggerHub DEV TEST OPS Code Create Automated UI Functional Tests (Web, Desktop, Mobile) TestComplete Script Run Tests On Real Devices in the Cloud CrossBrowserTesting Create Web Load Tests LoadComplete Spec Create Automated API Functional Tests (REST, SOAP, and more) SoapUI Pro Script Virtualize API & Database Services ServiceV Pro Create API Load Tests LoadUI Pro Monitor Web & API Performance, Availability, & Functional Correctness AlertSite Manage Manual & Automated Tests QAComplete Integrations …100 +
- 6. | SB Test Page What’s Going on in the Testing World? @Bria_Grangard BDD AI Machine Learning DevOps Shift Left Agile Automation
- 7. Page There are bottlenecks in today’s development processes. • Irontriangletrade-off • Teams today are constantly feeling pressure to deliver software faster, without compromising quality • Automationmightrampup • There is only a certain point as to how scalable automation can be • Testenvironmentsareoftentheroot-causeofthebottlenecks • They are very time consuming and costly @Bria_Grangard
- 8. Page @Bria_Grangard The promise of the new software delivery cycle WaterfallAgile Design Build Test Implement DevOps Week1 Week2 Week3 Week4
- 9. Page @Bria_Grangard Time Consuming Nature Of Web Testing #ofTests 0 200 400 600 800 1000 1200 MVP Feature Set 1 Feature Set 2 Feature Set 3 V2 More Features = More Testing Age Of Product #ofBrowsers 0 2 4 6 8 10 12 MVP Feature Set 1 Feature Set 2 Feature Set 3 V2 Popularity of Product
- 10. Page @Bria_Grangard 1 2 3 4 Test Less Hire More Testers Diversify Your Testing Run Tests In Parallel More Bugs Increased Cost
- 11. | SB Test Page Now… How Can We Go Faster? @Bria_Grangard 1. Test Frameworks 2. Automation 3. Parallel Testing
- 13. Page The Basics of a Test Framework @Bria_Grangard Requirements Tests Defects What do we make and how should it behave Make sure it works as stated in the Requirement Definition Sets Environments Actual Results do not equal Expected Results
- 14. Page @Bria_Grangard What is a Test Framework? A Test Framework: Links tests to other SDLC items Is NOT a Test Automation Framework but often contains one Allows for rapid creation of tests from reusable components Separates data from logic (REUSABILITY) Provides a standardized test “language” and reporting structure for an application under test
- 15. Page @Bria_Grangard Elements of a Test Framework • Library: A repository of all your decomposed scripts, separated into their components • Test Data Sources: A repository of all data sources • Helper Functions: A repository of all decomposed test scripts, automated or manual, that are inputs or checks • Test Environments: A list of all covered testing environments, broken out by type (OS, browsers…) • Modules: The combination of library items with any helper functions and test data sources–plus environments • Structure / Hierarchies: The “folder” structure of modules
- 16. Level 2: Figure Out What Tests Should be Automated @Bria_Grangard
- 17. Page There are many types of testing that need to be done… @Bria_Grangard Browser: Chrome HTML5, Angular JS Network Service/API/Database End-to-End Testing
- 18. Page @Bria_Grangard Automated Testing Pyramid Approach UI Service Unit Manual
- 19. Page @Bria_Grangard A Little Manual v Automated Math Product v2 # of Test Cases # of Browsers Supported Total Test Cases Avg Test Run Time Product v2 1,000 10 10,000 .5 Min Total Test Time 83 Hrs With 2 QA engineers, That is 1 week of testing # of Test Cases # of Browsers Supported Total Test Cases Avg Test Run Time 1,000 10 10,000 4 Min Total Test Time 666 Hrs With 5 manual testers, that is 3.5 weeks of testing
- 20. Page Decide on What to Automate • Environment Setup/Teardown • DataEntry • FormFilling • Varyingdatainputs inarepetitive process • Exposingbackend data(APIs,DBtable,etc.) • Repetitive/boring tasksthat arepronetoinattention errors • Taskswithhighreusevalue acrossmanyworkflows • Testswithtimingorscreenresponsivenessasacriteriaforsuccess • Manynon-functionaltesttypes,suchasperformancetesting • Capturing Results @Bria_Grangard
- 21. Page 21 @Bria_Grangard Speeding Up Your Pipeline Longer Time To Test Behind Releases Shorter Manual Testing Record & Replay Unit Testing POM Atomic Testing ContinuousTesting
- 22. PagePage But my Dev team says I have days to test, not weeks… @Bria_Grangard
- 23. Level 3: Go Faster. @Bria_Grangard
- 24. Page Let’s Go Faster! We can,withparalleltesting @Bria_Grangard Test 2Test 2Test 2 Test 3 Test 3 Test 3 Running tests sequentially, we were able to run our tests in 1 week Test 1Test 1 Test 1 Test 1 Test 1 Test 1 Test 2 Test 2 Test 2 Test 3 Test 3 Test 3 With 20 Parallel executions, we can run our entire test suite in only 4 hours
- 25. Page Types of test to run in parallel Testingacrossdifferentbrowsersanddevicesisoneofthemosttime consuming aspectsoftestingthefrontendofyourwebsiteorweb application. Runmoretests,against morebrowserconfigurations by running theminparallel. According tothetestingpyramid,Unittestsshould beyourmost abundant testtypeinyourentiretestingsuite.Because ofthis,running 14,000 unittestsinunderanhourisreallyonlypossible withamassive parallel testinginfrastructure investment. Needtogetyourminimumtestingdoneinthenext20minuteswhileyou pushahotfix?Onlywaytodothatistoruntheminparallel, allowing you togetthemosttestingdoneintheshortestamountoftime. Because deployments arehappening atsucharapidpace,regression testingisonofthebestwaystohaveatypeof“testingversioncontrol” makingsurethefunctionality ofthenewbuild, matchesthatofthelast stablebuild. Running thesetestsinparallel allows moretobetested. Cross Browser Testing Unit Testing Regression Testing Smoke Testing Most effectivetests tosee ROI fromfor paralleltesting @Bria_Grangard
- 26. Page Benefits of Parallel Testing 1. Quick Deployments 2. Faster Feedback 3. Cross Browser Testing 4. Better Test coverage 5. Saves Valuable Time @Bria_Grangard
- 27. | SB Test Page Key Takeaways 1. Testframeworksareimportantandakeytosuccess. 2. Whenautomating—choose whichteststoautomatewisely 3. Paralleltestinglet’syougofaster. @Bria_Grangard
Notas do Editor
- With the advent of the Agile Manifesto and the “maturation” of DevOps over the past few years, it seems that the promise of continuous deployments at a speed faster than light is almost fulfilled. Everywhere we look, development and ops team brag about how many times they ship a week, a day, or even an hour. There isn’t less testing happening in agile or devops, the phase has simply been broke down, sped up, and spread out.
- To add to the innate problem of speed in the software delivery cycle, there are a few speed bumps unique to testing web applications. As our product matures, we’ll have an increasing number of test cases to run in regression before each deployment. This can add exponential amounts of testing – and test execution time – in just a few months. Precious time most products do not have. Also, as our product gains in popularity, we’ll have to support an increasing amount of browsers and devices. Customers now want to use their personal iPads to access your web portal that just 5 years ago was only accessed by the latest version of IE on…yep, you guessed it. Windows XP Service Pack 3. This again ads massive time to the time needed to sufficiently test across needed browsers. LEAD: There are only a few ways to radically change your testing time to keep up with modern development.
- There are only a few ways to radically change your testing time to keep up with modern development. You can just test less. This results in more bugs, and we’d not recommend this strategy. You can also increase the amount of testers on your team to keep up with demand. OR You can diversify your testing suite with a mix of unit, API, and UI tests. And run tests in parallel, executing tests in continuous batches of 10, 20, 50 tests at one time.
- Let’s star high level with the software development lifecycle. While you all are probably very familiar with this let’s talk about some points. Requirements. Where it all begins. What do we want to make, what will we make, and how should whatever we make really behave. Need to have a solid foundation for success… you need to know what your requirements are and what you want them to accomplish. No confusion. It’s kind of like taking that moment to prepare for your essay before you write it. For Tests you can have multiple types of tests. Test Definition Test set What environments you want to run the test on-operating systems, browsers, resolutions, etc. Defects—it’s inevitable. Your tests are going to find some bugs and there will be some problem. But if you have a defect you want to be able to log it properly and tie it to the correct test and requirement. Releases—all of these things are encapsulated in a release. What are you pushing out for the next major or minor release/build.
- There is some feedback you get only from a UI layer test. So while you should minimize the efforts of UI testing, you still need to have these tests to complete end-to-end testing workflows. In this case, the test starts at the browser level: Mozilla, chrome, firefox whatever you’re using. It then touchses on HTML5 or Angular JS depending on what you're using. It then touches on network level testing and services/API/Database test.
- Mike Cohn’s Test Automation Pyramid is usually the go-to guide for deciding on how many tests should be automated. (After discussing this in much detail as well as the recent debate) The best way to incorporate a myriad of testing practices into your development cycle is to find and implement the right tools. Utilizing source control management tools (SCMS) such as Git or Subversion, CI tools like Jenkins, or defect management tools such as JIRA or Bugzilla, can speed up your processes as they integrate with a wide variety of automated test tools and provide more poignant feedback on which part of a test is failing.
- In this image we are showing the evolution of the entire development pipeline.—transforming from manual testing to continuous testing. Even with the benefits a diversified testing process can provide, test automation and easier to debug pieces, there is still a cap on time to test and time to release. To reach the final two phases on the right, atomic testing and continuous testing, teams will have to run their tests in parallel. In most cases, this will require teams to move their testing to the cloud for cost and speed reasons.
- Let’s take an example… Let’s imagine a real world example: in release 3 of your product, you have 8 hours of sequential regression testing to perform before the team feels confident to deploy. By release 5, this may be twice the amount of hours you’ll need to run your tests and as a bonus, your product is getting popular and is being used by more users on an increasing number of different devices. Before you were testing for only Chrome and FireFox, but now you see that you need Android and iOS devices, Safari and multiple versions of Internet Explorer. So you have 16 hours of tests and 10 different devices or browser to cover. This would take us 160 hours for complete test coverage before our deployment. With parallel testing environments, we can run our 16 hours of tests on 10 different devices at the same time, saving us 146 hours of testing time.
- Time is finite. So we need to maximize the time we have. Parallel testing allows you to test faster with a quicker turn around in deployments. No developer wants to spend more time testing their product than they did developing it. How can parallel testing be done? By setting up multiple VMs and other infrastructure devices or by using a cloud test service.
- Cross Browser Tests Regression tests--regression testing is essential between deployments. The process is vital to ensure that a piece of software is still functional after a new update has been made prior to the next shipment. A challenge with running regression tests is the limited amount of time often provided and the growing nature of your test suite as your application matures. Developers looking to push their product out faster may give very little time between when they’ve finished their build process and the production date to conduct any testing. Running through the regression suite quickly with concurrent tests will get build engineers any necessary feedback faster and will keep your DevOps teams happy. Unit Testing: unit tests are a great choice to run in parallel because they are naturally small since they test very particular or niche functions of an application, but can number in the thousands to tens of thousands. Smoke tests or build verification testing: is the final testing practice teams should consider implementing in parallel. Since smoke testing focuses on ensuring that only the most important functions of an application work as expected, it is typically used to decide whether or not to proceed with further testing. A smoke test that passes is an indicator to go ahead with more testing. If it fails, teams will stop testing and ask for a new build with the initial required fixes. Smoke testing is usually done to ensure that the minimal viable amount of testing is done with any quick fixes going into production immediately. Smoke tests can be conducted manually or through automation, but parallel testing will enable the process to go faster. Testing the few business critical processes, such as log-ins and checking out,. To get the product out the door in under five minutes can bring in revenue and make customers happy. This ensures enough time to test as thoroughly as they would like in between deployments—parallel testing accelerates this.
- Quick deployments—run as many tests as you want concurrently whether it’s 2, 10, 50, or 100. Faster feedback—the faster you run your tests, the faster the feedback. In older development models and testing practices it could take weeks to receive feedback once a developer built a feature and sent it across to QA. By that time, developers have moved on and may not remember what the function that failed was. Without writing a comment for every piece of code, this can be a real challenge to work with. “shifting left” “continuous testing” Cross Browser Testing—this is drastically more achievable with parallel testing. Typically, expanding coverage to incorporate all necessary environments can become very challenging and time consuming. With parallel testing you can test against multiple browsers and browser types all in a shorter timeframe than if you were running tests sequentially. Better Test Coverage—enabling teams to test more in less time inherently means better test coverage. Environments and platforms Saves valuable time