Continues Testing: We hear this quote frequently, in small and large companies around the world. My team and I have been a part of this evolution. I was asked to join an ambitious project at work. My task, was to build a team to be the first DevTest group in my organization.
The journey I will share with you is how to integrate the agile mindset into your non-agile team, focusing on methodologies, techniques, and technologies.
6. SaaS – Software as a Service
Data Base
Servers
File System
Firewall
Users SaaS
7. Which product my team is testing :
StormRunner Load - SaaS product
Performance testing solution that makes it easy to plan, run, and
scale web and mobile testing over the cloud.
8.
9. 9
1 Create and run
a load test
using your
favorite
scripting tool
2 Load Generators
spin up in AWS
EC2 or MS Azure
3 Results and app
insights are
available in real
time
The product my team is testing: StormRunner Load
New
Soon
Performance testing solution that makes it easy to plan, run, and
scale web and mobile testing over the cloud.
10.
11. StormRunner Load as a Service
Data Base
Servers
File System
Firewall
Users SaaS Cloud
13. Before Agile - Delivery Timeline
1 or 2 Major releases to production in 1 year
14. Agile StormRunner Load Delivery Timeline
March April AugustJune Nov Dec
Whats the CodeName of the next release?
If you know the answer, I will invite you to a
AlleyCat BubbleBobble Elevator Action Frogger Golden Axe Hang-On
FebruaryDec
CommanderKeen Digger
6 Major releases to production in 1 year
Year: 2015
15. Agile – The Challenges
• Test new features
• Test Regression
• Automation
• Environments
• Continues Integration
/ Continues Delivery
• Hotfixes
Do it in 1.5
months
• Data Migration (When update SaaS with
the newer version)
• Performance Testing
• Security Testing
• Globalization
• Documentation
• And more …
17. What is DevTest?
Using top end QA/DEV practices combined with DevOps techniques
QA & DEV Working very close, on the same IDE, code
base, and in united process
Automation is a MUST
Manual is for new features / exception flows that are
difficult to automate
NO MORE “Deliveries for QA”
YES for “Push to Production”
19. Continues Testing
Push new code
Release branch (1.5 months)
Test new code
Short Regression
Push to production
Automation
Developer Engineer
Test Engineer DevTest Team
Builds in 1 Release (1.5 month): 1,400
• Full: 236
• Commit: 1,164
40. Articles • 10 best practices for QA teams to deliver quality software, fast
• Blurring the lines: How the development tester role changes on agile teams
• 3 steps to building test databases for the real world
• Plan your tests and release criteria the right way
41. Tip
“Changing our mindset takes
precedence over technology.
When we work together, we will
succeed.”
Thank you
46. In Progress
49
12/27/2015 3/6/2016
3/6/2016
Go live
3/2/2016
Staging
2/28/2016
CF
2/14/2016
FF
1/3/2016
DevTest Start
Ice Climber V1.9 (Demo Timeline)
Preparations:
Feature discovery (meetings)
Defects review (re prioritize, clean)
Writing automation spec - example
Automation design for exist and new frameworks
Testing:
Test new features (Top priority, to make sure no delay in the timeline)
Write automation tests to increase coverage (in the first 2 weeks, content is very low)
47. Feature Freeze
50
12/27/2015 3/6/2016
3/6/2016
Go live
3/2/2016
Staging
2/28/2016
CF
2/14/2016
FF
1/3/2016
DevTest Start
Ice Climber V1.9 (Demo Timeline)
10 days for stabilization :
Test un finished features (2 days in the FF)
Prepare [Cloud deployment, move to Release branch, prepare regression plan, update AGM]
Regression ~7 days – Product matrix
Migration – Copy DB from production + File system
In addition:
Prepare Security Env
Demo all the features to Security / SaaS
Discover next release features
48. Code Freeze
51
12/27/2015 3/6/2016
3/6/2016
Go live
3/2/2016
Staging
2/28/2016
CF
2/14/2016
FF
1/3/2016
DevTest Start
Ice Climber V1.9 (Demo Timeline)
2 days for final validations :
Verify defects that was fixed in the end of FF
Validate Security + Performance + Globalization have no holders
Staging
~1 day sanity
Production
~1 day sanity
49. CI/CD
52
Follow all Jenkins failures
Test issue: Fix test
Code issue: Contact Dev and fix directly / Open defect
CI/CD Env issue: Contact DevOps
Deployment
SaaS QA1 / SaaS Q2 / Migration / Security
Cloud images
On Premise
Builds Number (JumpMan 1.95):
1,617
• Number of Full : 427
• Number of Commits: 1,190
51. Coverage : 65% - example
Tests: 700 active tests
UI product tree covergae
Automation
Frameworks
Client Backend
Coverage: 85% - example
API discover
Cloud
Coverage:
All regions – 20 Regions
Protocols –
Metrics
NV
Save months of WD
Cover new LG’s very fast
Duration: 1.5 hour
54
~13 min (all tests)
Commit
Full
~2.5 min (main flows)
In this section I will talk about SaaS – Software As A Service, about the transition from “Install Software” or having CD’s to install software to the new way of software, by Web Browser, example : Microsoft Outlook is a On Premise software it means you need to install, but if we think about Gmail then we can see that no need to install anything, at the same time we have the same service, even could be better using Gmail because it’s available from everywhere, Mobile, Computer, Mac, Etc..
On Premise Software:
Is installed and runs on computers
Not reachable from any device
Not updated on daily / monthly bases
Need space on the Device / Memory
SaaS – Software as a service:
No need to install, run from browser
Reachable from any device that support browser
Updated on daily / monthly bases (think about how Facebook, Gmail add new features when they want)
Not require space on the Device / Memory
To let you understand what is the challenge I and my team face I want to share 1 slide about the product we test StormRunner Load.
StormRunner Load started as a small idea at HPE, here we can see 1 of the first functional design diagrams on the glass.
StormRunner Load started as a startup inside a huge enterprise company.
Screenshot from StormRunner Load, example how Storm detect Anomalies in the throughput of the application.
Throughput : Throughput is a measure of how many units of information a system can process in a given amount of time. It is applied broadly to systems ranging from various aspects of computer and network systems to organizations.
StormRunner Load is a SaaS product, but it’s not just a web application it’s also considered as a Tool, that use cloud to generate the users to simulate the load test (example run load test using 1,000 users, 500 users running from London Cloud, another 500 users running from China).
It means I and my team face another layer for testing and it’s the Cloud, currently we hold 17 regions, each region we have 7 snapshots (cloud machine image, Windows, Linux).
How Agile affect us, why it’s needed?
Before 2 years I worked in LoadRunner product, for performance testing, this product is on premise.
The delivery was every year. In the best case we delivered 2 releases. (I call these days, the easy days )
Today, the delivery pipeline changed, because customers need more updates, on monthly bases, and to have a hotfixes when they want and not to wait for the next release – as we worked 2 years ago.
6 major releases in 1 year, in addition to the hotfixes and special requirements from customers.
No need to read all the tasks, but the important thing is to notice for how many tasks we face in the daily work, actually we did the same tasks before in 1 years, today we should do the same tasks even more, and to do it in 1.5 month.
In this section I will talk about the team I built, A modern QA team that I called DevTest, the name coming from Developer Tester, because the members of the team have the ability to developer modern tool in addition to write automation, and do test.
Please translate the text from the slide
Example for the training we give for each new member in the team, the list is longer than what we see in the slide, it just a pick to the content we ask the engineers to learn.
Continues Testing: we don’t stop getting content in hour bases, we are not a bottleneck for Developers, we test fast and give earlier feedback to developers.
In this slide I’m using animation to show how we get the content from developers.
The modern QA do more than just testing, as you can see we touch almost all the aspects of the project.
The Cool part
Update to 2016 May PMR (JumpMan)
When a test fail in CI/CD (Jenkins) the branch of Release will be blocked, in this slide we see that it is closed because tests failed, in the next picture we will see when it is opened after the bug was fixed and automation passed successfully.
Real example from my team, Olga is a DevTest in my team, she send a notification mail to Developers saying that branch was closed, and a defect was opened because a automatic test failed (automation find a real defect)
I am sure that a developer pushed code and damaged some other parts in the product.
Then we see a mail from Developer his name is Ilan.
Journal is my private notes, this slide will be present again while the presentation when different notes:
*
Journal is my private notes, this slide will be present again while the presentation when different notes:
*
Journal is my private notes, this slide will be present again while the presentation when different notes:
*
Journal is my private notes, this slide will be present again while the presentation when different notes:
*
We built this tool that discover all the API from the product we test
We developed a tool to calculate cloud costs.
StormRunner Load use cloud services, therefore when we test the product we consume cloud services.
In this section I will talk about SaaS – Software As A Service, about the transition from “Install Software” or having CD’s to install software to the new way of software, by Web Browser, example : Microsoft Outlook is a On Premise software it means you need to install, but if we think about Gmail then we can see that no need to install anything, at the same time we have the same service, even could be better using Gmail because it’s available from everywhere, Mobile, Computer, Mac, Etc..
Example for a automation challenge my team face:
As I mentioned in previous slides that we use the cloud to run load test around the world, and for that use we support 17 regions, in each region we use a 6 different images (Linux, Windows), from each region we expect to have 15 different data type that come back to the from of the product (I mean to the charts the users can see in the web page).
Example for a automation challenge my team face:
As I mentioned in previous slides that we use the cloud to run load test around the world, and for that use we support 17 regions, in each region we use a 6 different images (Linux, Windows), from each region we expect to have 15 different data type that come back to the from of the product (I mean to the charts the users can see in the web page).
The solution was to build a new framework based on JavaScript, that cover all the challenge that I mentioned in the previous slide, here is the generated report.
The test environments changed when we work with SaaS products, we have a dedicated farms for different use.
When we deploy a new version of the product for testing?
Mention a real story from StormRunner
Изменение нашего мышления берет верх над технологией. Только работая вместе, мы добьемся успеха.
Изменение нашего мышления важнее технологии. Работая вместе мы добьемся успеха!
