Why do we need to have software testing happen in a continuous manner? This deck explains the importance of Continuous Integration and a case study of 24x7 Testing.

1. Continuous Testing
Dr Ganesh Neelakanta Iyer
QA Architect
GANESHNIYER
gaiyer@progress.com
http://ganeshniyer.com

2. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.2
About Me
 B.Tech. in CSE from Mahatma Gandhi University, Kerala in 2004 with University First Rank
 Seven years of Industry work experience
 Masters and Ph.D. from National University of Singapore
 Several talks/workshops in India, USA, Europe
 Research interests: Cloud computing, Game theory,
Software Quality Analysis, IoT solutions
 Personal Interests: Kathakali, Teaching, Traveling,
Photography, Cooking
 Website: http://ganeshniyer.com
 Currently
• QA Architect at Progress Software, India
• Visiting Faculty at IIIT-H
• Adjunct Professor at BVRIT
GANESHNIYER

3. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.3
Quick Poll

4. Are you using Version Control?
Safe?

5. Can you release new version of
your software in one day?
Quick

6. Can you release new, well-
tested version of your software
in one day?
Quick & Safe?

7. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.7
Agenda
 Software Development today
 Continuous Integration and Continuous Testing
 24x7 Testing

8. What are top changes in
Software development
that were disruptive for
existing test
methodologies?

9. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.9

10. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.10
Agile
Credit: Blazemeter; Continuous Testing in Practice

11. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.11

12. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.12
Test Challenges with Agile Methodology
Frequently changing requirements
•Thanks to Agile  Changes can happen at any point of time during release cycle
• Requirements and Test Cases developed in parallel or test first strategy
Frequent releases
•Progress DataDirect Cloud released new features every 9 days in 2015*
• Testing needs to happen Early and Often, Everyone needs to Test
Understanding and testing real world behavior
• How customers see the product behavior after a release?
• No SRS-level waterfall documents to drive testing plan
Multiple environments & differences for test and production systems
• 3rd party systems integrated to the product
• Raises a requirement for GoLive Testing  But in minimal time

13. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.13
Software Test Challenges With Cloud
 Web-driven SaaS applications
 Everything in a few mouse clicks
 Frequent releases / Frequent changes
 No SRS documents
 Publically shared resources
 Multi-tenant penetration testing
Paradigm Shift
 High Availability and Resiliency are key
 Seamless upgrades
Everything is Agile
Availability Everything is shared

14. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.14
Continuous Integration & Automation are Key in Agile
Automation
• Nightly / Weekly
Build
Integration
• Combination of
unit tests and
regression tests
Dev-
environment
• Unit-tested

15. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.15
Continuous Integration and
Continuous Testing

16. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.16
Dev/Test  Expectations continue to increase
Better – Cheaper - Faster
Past
• Balance between speed, cost
and risk
Today
• Demand for releasing high
quality software faster
Credit: IBM article on Continuous Testing

17. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.17
Applications are becoming more complex to test
 Setting up complex environments
 Delays caused by lack of system
availability
 Third party services can be costly
 Testers spend more time setting up
tests than testing
Testing teams
spend on setting
up test
environments,
instead of testing
30-50%
average
time
Credit: IBM article on Continuous Testing

18. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.18
Continuous Integration and continuous testing
Accelerate software
delivery – for faster
time to value
Balance speed,
cost, quality and risk
– for increased
capacity to innovate
Reduce time to
customer feedback
– for improved
customer
experience
PC Credit: https://https://goo.gl/A2AaQw

19. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.19
Shift left, test early, test often

20. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.20
The Testing
Process – From
Development to
Operations
Credit: Blazemeter; Continuous Testing in Practice

21. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.21
Continuous Testing in Practice – Requirements Checklist
Tools, Scripting and
Overall Test Creation
• The provision of libraries and APIs to create “homegrown” tests
• The ability to create browser-based tests
• Ability for load and scalability testing
Provisioning and
Resources
• The option to run the same sets of tests with different provisioning
• The ability to easily adopt a new provisioning scheme when a test
configuration is executed in a different environment (e.g. Dev, CI,
Pre-Prod, Post-Prod)
• The ability to start testing as part of the CI process
Automation
• The ability to run any test configuration, a test, or a set of tests
using simple API calls
• The option to run as many tests as required in parallel
On-Demand Testing
and Re-Running
Failed Use Cases
• The ability to run any type of test on-demand. This is critical for
test development, debugging, troubleshooting and identifying the
root cause of a failure
• The option to automate test executions and run the same test on-
demand
Credit: Blazemeter; Continuous Testing in Practice

22. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.22
Continuous Testing in Practice – Requirements Checklist
Modules, Builds,
Release Candidates,
Releases and
Production
• The ability to combine different configuration fragments into one test
configuration in order to comprehensively test build, release, and
production snapshots.
Version Control Friendly
Incremental Testing
• The ability to support version control, incremental testing and
associate test configurations, sets of tests, and tests with versions (e.g
code, build, RC, releases).
Automatic ‘Failure’
Alerts
• Automatic indications of failures, with an alerting scheme per
developer, module and project
• The ability to gather all test artifacts and immediately send to relevant
people
• The option to run the failed tests again to identify the root cause
Reporting
• A seamless integration with existing reporting solutions (e.g. Jenkins
Performance Trend)
• The option to group tests by builds
• The ability to give jobs a ‘pass’ or ‘fail’ status and group accordingly
• Pass/fail trend reports
Credit: Blazemeter; Continuous Testing in Practice

23. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.23
24x7 Testing; A Case study of
what we do at Progress

24. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.24
24 x 7 Testing
Understanding and testing real world behavior
What is 24x7?
• Running a set of test
cases continuously 24 x
7
• Simulates real time
customer behavior
How 24x7?
• A few parallel threads
accessing the product
• Tests consisting of
regression (or it’s subset)
tests
Why 24 x 7?
• Many things can’t be
caught by regular
functional testing
• Error handling and retries
• Product can be made
more resilient and robust

25. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.25
24 x 7 Testing
VMs / Build Integration Machines
ANT ScriptTests
Reports using
Kendo UI controls
Decides
parallelism
HTTP POST reqBatch
Script
Decides
frequency
Logs
SUT
• Progress
products such
as Rollbase
• Uses any test
environments
• Existing test
cases can be
leveraged
Batch Script
• Decides the
frequency
• E.g. run on an
hourly basis
• Run in an
infinite loop
24x7
ANT Script
• We configure
#Threads,
#Tests to run
in parallel
• Tests can run
on VMs / Build
integration
machines
Reports
• Shown using
Keno UI
controls /
widgets
• Driver scripts
sends results
to the portal
• Machine
stores the logs

26. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.26
Case Study - Example
 Cloud platform that let you build SaaS-
based business apps in days/hours
 Responsive UI with built-in themes
 Typical apps customers build
• CRM
• Employee Management
• Bug Tracking System
What could be some typical tests for this platform?

27. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.27
Case study - Identifying tests; Example
 100 parallel treads
• 50 of them access “Bug Tracking System” application
• 50 of them access “CRM Application”
 Use cases
1. Create a bug and assign it to a user which will run an email trigger and related record trigger
– 10 users
2. Update the workflow status of some existing bug which will run a trigger – 20 users
3. Filter operation on bug object – 20 users
4. Create a lead from CRM portal and attach the lead to an account – 20 users
5. Look for all leads whose workflow status is “Open - Not Contacted” – 10 users
6. Create a new opportunity, assign it to an owner and attach it to an account – 10 users
7. Change the workflow status of an existing opportunity to “Closed-won” – 10 users
Tests should not
be self-cleaning
Why?

28. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.28
Layout of the infrastructure
BATCH script
• Checkout latest test scripts
• Compile and run them
ANT script
• Properties file containing the
details of test agents
• Invoke scripts in the agent machine
Central
Machine
Test
Agent 1
Test
Agent 2
…
Test
Agent n

29. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.29
Central Server

30. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.30
Central Server – Build.properties

31. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.31
Central Server – <Product name>.properties
testtype=Telerik
testlist="Psexec.exe 172.29.36.205 -u Administrator -p Progress!2009 -i 2 -w c:Telerik
c:Teleriktelerik.bat";"Psexec.exe 172.29.36.207 -u Administrator -p Progress!2009 -i 2
-w c:Telerik c:Teleriktelerik.bat";"Psexec.exe 172.29.36.208 -u Administrator -p
Progress!2009 -i 2 -w c:Telerik c:Teleriktelerik.bat";"Psexec.exe 172.29.36.209 -u
Administrator -p Progress!2009 -i 2 -w c:Telerik c:Teleriktelerik.bat";"Psexec.exe
172.29.36.215 -u Administrator -p Progress!2009 -i 2 -w c:Telerik
c:Teleriktelerik.bat";"Psexec.exe 172.29.36.90 -u Administrator -p Progress!2009 -i 2 -
w c:Telerik c:Teleriktelerik.bat";"Psexec.exe 172.29.36.227 -u Administrator -p
Progress!2009 -i 2 -w c:Telerik c:Teleriktelerik.bat";"Psexec.exe 172.29.36.231 -u
Administrator -p Progress!2009 -i 2 -w c:Telerik c:Teleriktelerik.bat";"Psexec.exe
172.29.36.229 -u Administrator -p Progress!2009 -i 2 -w c:Telerik
c:Teleriktelerik.bat";"Psexec.exe 172.29.36.233 -u Administrator -p Progress!2009 -i 2
-w c:Telerik c:Teleriktelerik.bat"

32. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.32
24 x 7 Testing – Sample report

33. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.33
Key Advantages of 24x7 testing
 Catch issues which are difficult to catch using normal functional
testing
 Make products resilient and robust
 Catch unexpected integration issues
 Test real time customer behavior

34. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.34
Catch issues which are difficult to catch using normal functional testing
https://cdn.tutsplus.com/wp/uploads/2013/12/session-diagram.jpg
http://biobeasts.artix.com/content/images/2015/07/memory-leak.jpg

35. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.35
Make products resilient and robust

36. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.36
Catch unexpected integration issues

37. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.37
Test real time customer behavior
http://blog.perfectomobile.com/wp-content/uploads/2015/10/WINDTUNNELGRAPHIC.jpg

38. © 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.38
Advantages of continuous Testing
Helping to make software development faster
• Continuous testing can make it possible to making software available sooner without
compromising quality
Reduced Risk
• Every time a change is made to code or new code is written, there is the potential for error
to occur. Identifying the problem is much easier when continuous testing is performed and
the issue is caught immediately so problems can be corrected.
Making consistency simpler
• Automation of testing makes it easier to achieve consistency by providing a carefully
configured environment and maintaining the same correct configuration for all relevant
tests
Faster releases
• The process of regularly using continuous testing means that cleaner code can be released
to end users in a timely manner
https://www.etesting.com/2014/02/25/benefits-of-continuous-testing/

39. Are you ready to implement
Continuous Testing in your
organization?

40. gaiyer@progress.com
http://ganeshniyer.com
GANESHNIYER