An example of architectural solution based on consolidate approach is considered. Proposed solution allows to reduce tests time running especially while working on testing of application API with some time-consuming process.

1. CONSOLIDATED
TEST CASES
EXECUTION
Dmitry Hes
SEPTEMBER, 2016

2. 2CONFIDENTIAL
CONSOLIDATED TEST CASE EXECUTION
INTRODUCTION

3. 3CONFIDENTIAL
INTRODUTION

4. 4CONFIDENTIAL
INTRODUTION
Takes much time
• long-running process (> 5 min)
• modify database significantly
Integration processes

5. 5CONFIDENTIAL
INTRODUTION
time
count
… integration
processes are
invoked many
times

6. 6CONFIDENTIAL
INTRODUTION
How to speed up?

7. 7CONFIDENTIAL
INTRODUTION

8. 8CONFIDENTIAL
CONSOLIDATED
APPROACH

9. 9CONFIDENTIAL
Testcase
Arrangement
test data
(create e-mail)
Sync processes
(Set data in DB)
Build-
model
Verify
data
REAL EXAMPLE
Testcase
Arrangement
test data
(create e-mail)
Sync processes
(Set data in DB)
Build-
model
Verify
data
… Testcase
Arrangement
test data
(create e-mail)
Sync processes
(Set data in DB)
Build-
model
Verify data
many times with unique test data

10. 10CONFIDENTIAL
REAL EXAMPLE
Could we use parallelization?
(setup scope of environments and run TCs on each)
No.
• We are unable to run build-model against one server
• Only one process must run at the same time
• Therefore we need setup whole environment on each VM
• It takes about 20 minutes (some installation scripts execution
require
• So no sense
• Moreover we don’t have so much free VMs

11. 11CONFIDENTIAL
REAL EXAMPLE
Execution time of 1 test case: 10–15 minutes
Number of test cases: 70
Total running time: 875 minutes (14.5 hours)

12. 12CONFIDENTIAL
SOLUTION
Sync processes
(Set data in DB) 0
Build-
model
Testcase
Arrangement
test data
(create e-mail)
Testcase
Arrangement
test data
(create e-mail)
…
Testcase
Arrangement
test data
(create e-mail)
Testcase
Verify
data
Testcase
Verify
data
…
Testcase
Verify
data

13. 13CONFIDENTIAL
CONSOLIDATED
FRAMEWORK

14. 14CONFIDENTIAL
CONSOLIDATED FRAMEWORK
Test data csv-file should be
created/updated before
running consolidated Test case
by executing
ParameterExportByTestCaseID
test method.
Test class
implementationStructure
[TestMethod]
public void MasterTestSuite()
- testCaseData
- <variables to store test data>
...
<Test class>
CNTestCaseBase
+ setup()
reads test data from the
appropriate csv file into and
stores it to local dictionary and
variables of method instance
testCaseData =
CNTestSuiteExten
sions
.LoadTestCaseDa
ta();+ exec()
create and setup test data
needed to TC like create mails,
contacts, meeting, etc.
+ validate()
method that perform validation
steps (after processes execution)
+ cleanup()
clean up environment after TC
execution
Test case
CSV file:
../../../CNAutomation
Project
/TFS.Parameters/
<test case id>.csv
[TestMethod]
<Test
class>()
Start
Create TC…
classes
instances
For each instance
execute setup();
execute exec()
Execute sync- processes
Execute build-model
process,
iisreset
For each instance
execute validate()
End
For each instance
execute cleanup()
[TestClass]
TFSParameterExportByTestCaseID
[TestMethod]
+ ParameterExportByTestCaseID()
Gets test parameters data from TFS for
specified test cases and stores them in
appropriate csv-file

15. 15CONFIDENTIAL
CONSOLIDATED FRAMEWORK
0
Build-
model0
CRM
sync
process
Sync to
CRM
example 2

16. 16CONFIDENTIAL
CONSOLIDATED FRAMEWORK
Test class
implementation
Structure
[TestMethod]
public void BatchExecution_TestMethod ()
+ ctor <Test class>
Invokes setup()
<Test class>
TestCaseBase
+ setup()
reads test data from the
appropriate csv file into and
stores it to local dictionary and
variables of method instance
[ + Validation1() ]
if required
[ + Validation2() ]
if required
[TestClass]
<Test class>()
Start
Create TC…
classes
instances
Execute
Process 1
End
Implementation 2
For each instance
Exec Validation 1
Execute
Process 2
For each instance
Exec Validation 2
…Execute
Process N
For each instance
Exec Validation N
…[ + ValidationN() ]
if required

17. 17CONFIDENTIAL
PROS AND CONS
Advantages Disadvantages
significantly reduced TC run time complex test case structure
able to manage execution order test data should be unique
test data is read from local CSV huge test log
difficult to add new methods
MS test methods like Class/Test
initialize or cleanup are not executed
hard to adopt existing TCs to
consolidated approach

18. 18CONFIDENTIAL
NEXT STEP

19. 19CONFIDENTIAL
1. Make test case more flexible
• ability to change testing workflow process by managing pool of executable
methods
2. Add execution of ClassInitialize, TestInitialize, ClassCleanup, TestCleanup
accordingly
NEXT STEP