Experience in run an iterative and incremental test process for different projects platform

1. 22nd IFIP International Conference on Testing Software and Systems – ICTSS 2010
QUALITY
Federal University of Amazonas - UFAM VALIDATION
Nokia Technology Institute - INdT
Bug
MANTIS
INdT
Iterative Software Testing Process for Scrum and
Waterfall Projects with Open Source Testing Tools
AGILITY
Experience
SELENIUM
Verification TestLink
Eliane Collins & Vicente Lucena
SCRUM Natal – RN , Nov 2010

2. Summary
1. Introduction
2. Environments Context
3. Software Testing
Process
4. Execution Results
5. Conclusion
Company Confidential

3. Introduction
Company Confidential
3

4. INTRODUCTION
Different projects with different development process
request Test activities.
The Software Testing process must be agile for all projects.
Project 1 used Scrum. It is an agile development process
focusing on teamwork .
The test activities must follow the scrum and they may be
executed efficiently and fast .
Company Confidential
4

5. INTRODUCTION
The Project 2 used Waterfall software development process.
The system test activities must be done for the whole system
in the test phase.
Software Testing automation is one of the main approaches
that has been applied to decrease testing costs and time.
Test Case
Test Case Test Case
Test Case
Test Planning
Test Planning Defect Report
Defect Report
Specification
Specification Execution
Execution
Sequential software testing process
Company Confidential
5

6. Environments Context
Company Confidential
6

7. Environments Context
Nokia Technology Institute – INdT
It is an independent and nonprofit institute committed to conducting
research and development of technological solutions through
application development, new technologies and concepts.
The INdT area where the experiment was designed is called Product
Manufacturing Technologies (PMT).
Company Confidential
7

8. Environments Context
Project 1
A system to provide an online customer satisfaction survey result to
the development team by e-mail.
The system registers the projects, leaders, customers, team members
and the customers’ answers.
The system sends the reminders to customers online to answer the
questions on the survey and forward the results to the development
team.
The project was developed under the Scrum process.
Web Platform: MVC Architecture, Ruby and Java languages, Rails
framework and MySQL database.
Company Confidential
8

9. Environments Context
Project 1 - Scenario
In the schedule there wasn’t any time planned for test activities.
No formal Software Testing process was done.
Just one test analyst.
Short time for test execution.
No financial resources available for this activity.
Company Confidential
9

10. Environments Context
Project 2
A software project to provide a graphic analysis of factory production
tests in real time in order to advise professionals to support
manufacturing on the quality trends for production line failures.
The project development used the Waterfall process approach.
Software platform : Client-Server, Java Swing, IDE Eclipse environment
and Oracle database.
The System has twelve screens, where six of them were used to show
and analyze graphic charts and the other six had the data inputs,
reports and user configuration.
Company Confidential
10

11. Environments Context
Project 2 - Scenario
In the schedule there wasn’t any time planned for test activities.
No formal Software Testing process was done.
Outdated documentation of the system requirements.
Functionalities implemented but not tested
No financial resources available for this activity.
4 weeks for a tester and a test analyst to do all test activities for the
whole system
Company Confidential
11

12. Software Testing Process
Company Confidential
12

13. Software Testing Process
Iterative Software Test Process
Test Inputs: Project Plan, Requirements or Backlog, Development
Environment.
Test Deliveries: Test Plan, Test Cases Specification, Test Case Execution
Report and Summary Defect Report.
Functional Test Automation for basic test cases.
Exploratory Testing.
Create Test Cases derived from Backlog stories or Requirements.
The tester was a scrum team member.
Interaction with the Development Team
Company Confidential
13

14. Software Testing Process
Iterative Software Test Process
Start
Environment
Environment Criation/ Update
Criation/ Update
Planning
Config. Test Cases
Test Cases
Config.
NO
Summary
Summary End of
End of Automated
Automated
Defect Report
Defect Report Iterations?
Iterations?
YES Test Scripts
Test Scripts
End
Regression Test
Test
Tests Execution
Execution
New System
New System Defect Report
Defect Report
Version
Version
Report
Report
Generation
Generation
Company Confidential
14

15. Software Testing Process
Open Source Automated Test Tools
TestLink to automate the managing, planning, test cases specification
and test execution reports.
Mantis for Defect Tracking tool, friendly user interface to interact with
developers and can be integrated with TestLink.
Company Confidential
15

16. Software Testing Process
TestLink:
Specification:
Reports:
Company Confidential
16

17. Software Testing Process
Web Functional Open Source Automated Test Tools
Selenium IDE and Selenium Core.
Aproach rec-and-play for Mozila Firefox and Internet Explorer
browsers.
Test scripts and Test Suite created in HTML code.
Company Confidential
17

18. Software Testing Process
Automated Testing Tools:
For functional test automation a new open source tool was chosen to
test the java swing desktop application: MARATHON.
Approach rec-and-play recording the user actions creating test scripts
in Jython language and run them later in the tool .
Company Confidential
18

19. Execution and Results
Company Confidential
19

20. Execution– Project 1
Sprint Test Activities in First week:
Test Environment Configuration
Automated test scripts creation. The test scripts were implemented in
the browser Mozilla Firefox.
These scripts were edited and gathered in a test suite, and reused in
other test cases;
the test cases were done according to the acceptance criteria
emphasizing test cases creation for exceptions.
The system analyst should review the test cases to ensure coverage of
all functionalities;
Company Confidential
20

21. Execution– Project 1
Sprint Test Activities in Second week:
The test scripts were executed for Firefox and Internet Explorer
browsers. Manual Exploratory tests were also executed by the tester.
At Mantis and TestLink, the defects were registered, described and
assigned to the developer who was notified by e-mail.
The Reports were automatically generated by TestLink.
When the defects had been fixed, the entire test suite was executed by
Selenium doing the regression test.
It ensures that another part of the system was not affected by code
changes.
If the acceptance criteria were met, the same activities of the process
will begin again for the next Sprint.
Company Confidential
21

22. Execution– Project 2
Goal: To ensure in a short time the main features stressed.
Same test process for the scrum, but with some adaptations:
A meeting with the development team to understand the project and
verify the requirements documentation updates.
The testers asked for the prioritized list of the project features where
the most important features had higher priority (As a backlog list).
The test iterations were planned having the most priority features
being tested first.
Test Activities executed in 4 iterations (4 weeks to test all
applications).
Company Confidential
22

23. Execution– Project 2
Test Activities for iteration:
Plan the test cases for the priority features in the TestLink Tool.
Automated test scripts creation for test cases in Marathon.
Execute the automated tests and the exploratory manual tests too.
The defects found are registered in Testlink and Mantis.
When the developer fixes the defect, the code changes, so all
automated tests are executed to do the regression tests.
The Automated reports are generated and sent to the Project leader.
The development team releases another system version and the
same activities of the process will begin again for the next test
iteration.
Company Confidential
23

24. Results
General Results:
100% of the functionalities from backlog were covered in Project 1.
65% of the system defects were from browser incompatibilities in
Project 1.
77 defects were registered in Mantis, where 30% were interface
defects and 70% were functional defects in Project 2.
The regression tests showed defects in the functionalities that were
considered done.
Good evaluation of the projects.
Company Confidential
24

25. Results
Waterfall Results Comparison
Test
Test Process Coverage Regression cicles Defects
Cases
Manual and
65% 40 0 27
Sequential
Iterative and
100% 75 3 77
Automated Process
Company Confidential
25

26. Conclusion
Company Confidential
26

27. Conclusion
Lessons learned:
Plan Test Activities and engage the test team from the beginning of
the project.
Automated test scripts had to be reworked and updated with every
backlog change for each sprint.
The scrum developer team had to estimate more time to defects
fixing because of the large amount of problems found during the
regression tests.
Exploratory tests found the usability and performance problems.
The test script creation has become faster because of the reuse of the
script code.
Company Confidential
27

28. Conclusion
The test activities broken into iterations made it possible to
stress the most important features, making the system attend
to what was requested by the client.
Search for new automation tools for other development
platforms for further projects in the company.
Search for new techniques to improve agile test process as
Test based on Models to reduce the rework.
Company Confidential
28

29. THANK YOU!
Company Confidential
29