Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (18)
Semelhante a Continuous Testing 2016
Semelhante a Continuous Testing 2016 (20)
Último
Último (20)
Continuous Testing 2016
- 1. Continuous Testing 2016 Karim Fanadka, DevTest Manager
- 2. About me: Karim Fanadka Job: DevTest Manager at HPE Software, 7+ Product: StormRunner Load Favorite activates: Traveling, Ski, Blogging Ski – France 2013 Road Trip – Iceland 2015
- 3. We are going to talk about: Agile Modern QA SaaS Automation Environments Hotfixes Continuous Deployment
- 4. SaaS – Software as a Service
- 5. Before SaaS – On Premise Software Users
- 6. SaaS – Software as a Service Data Base Servers File System Firewall Users SaaS
- 7. Which product my team is testing : StormRunner Load - SaaS product Performance testing solution that makes it easy to plan, run, and scale web and mobile testing over the cloud.
- 9. 9 1 Create and run a load test using your favorite scripting tool 2 Load Generators spin up in AWS EC2 or MS Azure 3 Results and app insights are available in real time The product my team is testing: StormRunner Load New Soon Performance testing solution that makes it easy to plan, run, and scale web and mobile testing over the cloud.
- 11. StormRunner Load as a Service Data Base Servers File System Firewall Users SaaS Cloud
- 12. Agile – Do It Fast
- 13. Before Agile - Delivery Timeline 1 or 2 Major releases to production in 1 year
- 14. Agile StormRunner Load Delivery Timeline March April AugustJune Nov Dec Whats the CodeName of the next release? If you know the answer, I will invite you to a AlleyCat BubbleBobble Elevator Action Frogger Golden Axe Hang-On FebruaryDec CommanderKeen Digger 6 Major releases to production in 1 year Year: 2015
- 15. Agile – The Challenges • Test new features • Test Regression • Automation • Environments • Continues Integration / Continues Delivery • Hotfixes Do it in 1.5 months • Data Migration (When update SaaS with the newer version) • Performance Testing • Security Testing • Globalization • Documentation • And more …
- 16. Modern QA Team (DevTest)
- 17. What is DevTest? Using top end QA/DEV practices combined with DevOps techniques QA & DEV Working very close, on the same IDE, code base, and in united process Automation is a MUST Manual is for new features / exception flows that are difficult to automate NO MORE “Deliveries for QA” YES for “Push to Production”
- 18. Training “we are testing, and breaking code constantly but we help rebuild it better…….”
- 19. Continues Testing Push new code Release branch (1.5 months) Test new code Short Regression Push to production Automation Developer Engineer Test Engineer DevTest Team Builds in 1 Release (1.5 month): 1,400 • Full: 236 • Commit: 1,164
- 21. Automation
- 22. Automation 0 30 80 100 120 150 170 200 250 300 0 0 75 120 147 220 390 450 600 650 0 0 0 0 100 200 270 270 270 270 0 100 200 300 400 500 600 700 Automation Active Tests - Trend Client Backend Cloud 85% Coverage 100% Coverage 65% Coverage X 2 Browsers Builds Number (JumpMan 1.95): 1,617 • Number of Full : 427 • Number of Commits: 1,190
- 23. 23 Labels automation coverage SPEC – 66 pages
- 24. 24 Product Tree – Regression plan
- 27. Frameworks: Frameworks Tests Client Testing Framework API Testing Framework Cloud Framework Results
- 28. Bot 28
- 29. 29
- 30. Testing based analytics from production
- 32. Cost cloud tool
- 33. Cloud Testing
- 34. Example for Automation challenge 17 Regions in the Cloud 15 different metrics 255 Tests
- 35. Automation – Cloud Framework - Report
- 37. Test Environments Farm QA Security Performance IntegrationLocal (Linux) Staging
- 38. Hotfixes in 1 day
- 39. Hotfixes Lifecycle 07:00 Production bug 08:00 Research by QA and Dev 10:00 Fixing bug 16:00 Testing (Staging) 14:00 Testing (Local) 18:00 Push to production
- 40. Articles • 10 best practices for QA teams to deliver quality software, fast • Blurring the lines: How the development tester role changes on agile teams • 3 steps to building test databases for the real world • Plan your tests and release criteria the right way
- 41. Tip “Changing our mindset takes precedence over technology. When we work together, we will succeed.” Thank you
- 42. @kim0z85 Question? https://www.facebook.com/karim.f.as https://il.linkedin.com/in/kareemfa kareem9k@gmail.com Karim.Fanadka@HPE.com Karim Fanadka DevTest Manager HPE Software
- 43. 46
- 44. 47 12/27/2015 3/6/2016 3/6/2016 Go live 3/2/2016 Staging 2/28/2016 CF 2/14/2016 FF 1/3/2016 DevTest Start Ice Climber V1.9 (Demo Timeline) • Production • Staging • CF • FF • Dev
- 45. 48 CI Production FF CF Staging Security + SAAS Feature discovery Defects -> ER’s
- 46. In Progress 49 12/27/2015 3/6/2016 3/6/2016 Go live 3/2/2016 Staging 2/28/2016 CF 2/14/2016 FF 1/3/2016 DevTest Start Ice Climber V1.9 (Demo Timeline) Preparations: Feature discovery (meetings) Defects review (re prioritize, clean) Writing automation spec - example Automation design for exist and new frameworks Testing: Test new features (Top priority, to make sure no delay in the timeline) Write automation tests to increase coverage (in the first 2 weeks, content is very low)
- 47. Feature Freeze 50 12/27/2015 3/6/2016 3/6/2016 Go live 3/2/2016 Staging 2/28/2016 CF 2/14/2016 FF 1/3/2016 DevTest Start Ice Climber V1.9 (Demo Timeline) 10 days for stabilization : Test un finished features (2 days in the FF) Prepare [Cloud deployment, move to Release branch, prepare regression plan, update AGM] Regression ~7 days – Product matrix Migration – Copy DB from production + File system In addition: Prepare Security Env Demo all the features to Security / SaaS Discover next release features
- 48. Code Freeze 51 12/27/2015 3/6/2016 3/6/2016 Go live 3/2/2016 Staging 2/28/2016 CF 2/14/2016 FF 1/3/2016 DevTest Start Ice Climber V1.9 (Demo Timeline) 2 days for final validations : Verify defects that was fixed in the end of FF Validate Security + Performance + Globalization have no holders Staging ~1 day sanity Production ~1 day sanity
- 49. CI/CD 52 Follow all Jenkins failures Test issue: Fix test Code issue: Contact Dev and fix directly / Open defect CI/CD Env issue: Contact DevOps Deployment SaaS QA1 / SaaS Q2 / Migration / Security Cloud images On Premise Builds Number (JumpMan 1.95): 1,617 • Number of Full : 427 • Number of Commits: 1,190
- 50. Defects
- 51. Coverage : 65% - example Tests: 700 active tests UI product tree covergae Automation Frameworks Client Backend Coverage: 85% - example API discover Cloud Coverage: All regions – 20 Regions Protocols – Metrics NV Save months of WD Cover new LG’s very fast Duration: 1.5 hour 54 ~13 min (all tests) Commit Full ~2.5 min (main flows)
- 52. Automation 0 30 80 100 120 150 170 200 250 300 0 0 75 120 147 220 390 450 600 650 0 0 0 0 100 200 270 270 270 270 0 100 200 300 400 500 600 700 Automation Active Tests - Trend Client Backend Cloud 85% Coverage 100% Coverage 65% Coverage X 2 Browsers Builds Number (JumpMan 1.95): 1,617 • Number of Full : 427 • Number of Commits: 1,190
- 53. Next slides are reference – DON’T PRESENT 56
- 54. 57 Labels automation coverage SPEC – 66 pages
- 55. 58 Product Tree – Regression plan
- 56. 59 API Coverage
- 57. 60 API Coverage
Notas do Editor
- In this section I will talk about SaaS – Software As A Service, about the transition from “Install Software” or having CD’s to install software to the new way of software, by Web Browser, example : Microsoft Outlook is a On Premise software it means you need to install, but if we think about Gmail then we can see that no need to install anything, at the same time we have the same service, even could be better using Gmail because it’s available from everywhere, Mobile, Computer, Mac, Etc..
- On Premise Software: Is installed and runs on computers Not reachable from any device Not updated on daily / monthly bases Need space on the Device / Memory
- SaaS – Software as a service: No need to install, run from browser Reachable from any device that support browser Updated on daily / monthly bases (think about how Facebook, Gmail add new features when they want) Not require space on the Device / Memory
- To let you understand what is the challenge I and my team face I want to share 1 slide about the product we test StormRunner Load.
- StormRunner Load started as a small idea at HPE, here we can see 1 of the first functional design diagrams on the glass. StormRunner Load started as a startup inside a huge enterprise company.
- Screenshot from StormRunner Load, example how Storm detect Anomalies in the throughput of the application. Throughput : Throughput is a measure of how many units of information a system can process in a given amount of time. It is applied broadly to systems ranging from various aspects of computer and network systems to organizations.
- StormRunner Load is a SaaS product, but it’s not just a web application it’s also considered as a Tool, that use cloud to generate the users to simulate the load test (example run load test using 1,000 users, 500 users running from London Cloud, another 500 users running from China). It means I and my team face another layer for testing and it’s the Cloud, currently we hold 17 regions, each region we have 7 snapshots (cloud machine image, Windows, Linux).
- How Agile affect us, why it’s needed?
- Before 2 years I worked in LoadRunner product, for performance testing, this product is on premise. The delivery was every year. In the best case we delivered 2 releases. (I call these days, the easy days )
- Today, the delivery pipeline changed, because customers need more updates, on monthly bases, and to have a hotfixes when they want and not to wait for the next release – as we worked 2 years ago. 6 major releases in 1 year, in addition to the hotfixes and special requirements from customers.
- No need to read all the tasks, but the important thing is to notice for how many tasks we face in the daily work, actually we did the same tasks before in 1 years, today we should do the same tasks even more, and to do it in 1.5 month.
- In this section I will talk about the team I built, A modern QA team that I called DevTest, the name coming from Developer Tester, because the members of the team have the ability to developer modern tool in addition to write automation, and do test.
- Please translate the text from the slide
- Example for the training we give for each new member in the team, the list is longer than what we see in the slide, it just a pick to the content we ask the engineers to learn.
- Continues Testing: we don’t stop getting content in hour bases, we are not a bottleneck for Developers, we test fast and give earlier feedback to developers. In this slide I’m using animation to show how we get the content from developers.
- The modern QA do more than just testing, as you can see we touch almost all the aspects of the project.
- The Cool part
- Update to 2016 May PMR (JumpMan)
- When a test fail in CI/CD (Jenkins) the branch of Release will be blocked, in this slide we see that it is closed because tests failed, in the next picture we will see when it is opened after the bug was fixed and automation passed successfully.
- Real example from my team, Olga is a DevTest in my team, she send a notification mail to Developers saying that branch was closed, and a defect was opened because a automatic test failed (automation find a real defect) I am sure that a developer pushed code and damaged some other parts in the product. Then we see a mail from Developer his name is Ilan.
- Journal is my private notes, this slide will be present again while the presentation when different notes: *
- Journal is my private notes, this slide will be present again while the presentation when different notes: *
- Journal is my private notes, this slide will be present again while the presentation when different notes: *
- Journal is my private notes, this slide will be present again while the presentation when different notes: *
- We built this tool that discover all the API from the product we test
- We developed a tool to calculate cloud costs. StormRunner Load use cloud services, therefore when we test the product we consume cloud services.
- In this section I will talk about SaaS – Software As A Service, about the transition from “Install Software” or having CD’s to install software to the new way of software, by Web Browser, example : Microsoft Outlook is a On Premise software it means you need to install, but if we think about Gmail then we can see that no need to install anything, at the same time we have the same service, even could be better using Gmail because it’s available from everywhere, Mobile, Computer, Mac, Etc..
- Example for a automation challenge my team face: As I mentioned in previous slides that we use the cloud to run load test around the world, and for that use we support 17 regions, in each region we use a 6 different images (Linux, Windows), from each region we expect to have 15 different data type that come back to the from of the product (I mean to the charts the users can see in the web page).
- Example for a automation challenge my team face: As I mentioned in previous slides that we use the cloud to run load test around the world, and for that use we support 17 regions, in each region we use a 6 different images (Linux, Windows), from each region we expect to have 15 different data type that come back to the from of the product (I mean to the charts the users can see in the web page).
- The solution was to build a new framework based on JavaScript, that cover all the challenge that I mentioned in the previous slide, here is the generated report.
- The test environments changed when we work with SaaS products, we have a dedicated farms for different use.
- When we deploy a new version of the product for testing?
- Mention a real story from StormRunner
- Изменение нашего мышления берет верх над технологией. Только работая вместе, мы добьемся успеха. Изменение нашего мышления важнее технологии. Работая вместе мы добьемся успеха!
- Update to 2016 May PMR (JumpMan)