QA Services for Configurable Software

Rhonda Software

Quality Assurance (QA) Services
Configurable software quality assurance process:
From requirements acquisition through testing and configuration management
to release and maintenance activities

Quality Assurance – to Serve Your Needs
“Quality is never an accident; it is always the result of intelligent effort.”

John Ruskin
“Software never was perfect and won’t get perfect. But is that a license
to create garbage? The missing ingredient is our reluctance to quantify
quality.”

Boris Beizer

“Testing is an infinite process of comparing the invisible to the
ambiguous in order to avoid the unthinkable happening to the
anonymous.”

James Bach

© 2013 Rhonda Software: www.rhondasoftware.com

2

QUALITY
BUILDING BLOCKS
PEOPLE
talented & skilled
professionals

PROCESS
lightweight &
mature practices

TECHNOLOGY
flexible & powerful
tools


3

Better Safe Than Sorry
There is no secret that it is impossible to develop software product
without any defects in it. And the more complex product you create –
the more errors it will demand.

EXECUTE

What?

PLAN

CONTROL

Quality Assurance (QA), as we see, is the process that should be part of each
and every stage of SW product development . It is a well chosen number of best
practices taken from CMM/CMMI and our 18-year experience to ensure high
quality of developed SW product. In the middle of QA is a set of testing activities
integrated into entire production cycle.

QUALITY

Why?
Because escaped defects (errors left in released product) may cost too much. Too much money to fix
and too much reputational damage to live with.

When?
The sooner the better. Everybody knows that the later a defect is found, the higher its impact and
price will be. However, it is never too late to think about quality – our QA process can be easily
adopted and integrated into any customer process on any project stage.

4

The Cost of Error
Every year, software errors cause massive amounts of problems all over the world. A 2002 study
commissioned by the National Institute of Standards and Technology found that software bugs cost
the US economy $59.5 billion every year (imagine the global costs). The study estimated that more
than a third of that amount, $22.2 billion, could be eliminated by improved testing.

Here is a couple of relatively fresh examples:
March, 2011. Japanese bank Mizuho Financial Group's clients

experienced a software glitch that collapsed its ATM network and
internet banking systems. The result was $1.5 billion in salary
payment delays and $9 billion in unprocessed payments.
August, 2012. In less than an hour, Knight Capital's computers
executed a series of automatic orders that were supposed to be

spread out over a period of days. Millions of shares changed
hands. The resulting loss, which was nearly four times the
company's 2011 profit, crippled the firm and brought it to the edge
of bankruptcy. Knight only survived after a group of investors
swooped in with a $440 million lifeline.

5

Integrated QA Process
PROJECT
MANAGEMENT

Product/Feature
management

QUALITY
ASSURANCE

Identified
requirements

Task tracking

REQUIREMENTS
MANAGEMENT

Specifications

Resources

TEST
MANAGEMENT

Scope/Planning
Monitoring

TEST
PREPARATION

Design

Reporting

Unit testing

CODE
DEVELOPMENT

Code best
practices
Reviews

Requirements
traceability

Automation

TEST
EXECUTION
Test passes
Defect tracking

Escaped
defects
analysis

Version control
Continuous
integration
Release
management


CONFIGURATION
MANAGEMENT

6

QA Core Components
QA Dependencies - Project Management
As any other project activity, QA should be carefully
considered in planning stage to define quality goals &
criteria, identify applicable QA/testing procedures and
pre-assign necessary resources (including personnel,
hardware and infrastructure environment).
In the large scale projects, there is an additional practice that
have major impact on resulting product quality.

PROJECT
MANAGEMENT

Product/Feature
management
Task tracking

Resources

Product/Feature Management
• When there is an entire line of products built on a common code base
• When there is incremental product evolution where new functionality is released in
new product versions, especially only in select product tiers
• When there are a lot of features: to add something new, to change existing behavior,
to incorporate functionality from other product in the line, to exclude obsolete function,
and so on...
Products and features need to be managed very neatly. Especially since scope and
essence of on-project QA activity is directly connected with selected product feature set.

7

QA Core Components
QA Dependencies - Requirements Management
Requirements, as customer / user expectations regarding
developed SW product features and capabilities always exist
on a project in some form… even just as a set of ideas in
a head of a project manager.

Identified
requirements

For the quality sake, it is better to keep product requirements
in a well organized way.

REQUIREMENTS
MANAGEMENT

Specifications

Requirements Specifications
• To identify all important aspects of functionality and receive customer’s confirmation
• To define restrictions and exceptions avoiding a case when assumed, but
undocumented behavior was never implemented
• To establish common understanding across the entire project team
• To provide basis for project estimates, code development and validation processes
• To be a starting point for effective communication and change management during
product evolution process
Thoroughly identified software requirements, organized as a document may become one
of the major instruments during testing process since Requirements Coverage validation
is the powerful tool to verify that a developed product fulfills all given expectations.

8

QA Core Components
QA Dependencies - Configuration Management
Same as Requirements, Configuration Management (CM)
is always present on any software development project,
though sometimes only as a single code version without
any history.

Version control
Continuous
integration

Release
management

CONFIGURATION
MANAGEMENT

Needless to say that a well-organized tool-based CM is vital
for any project that is more complex then “Hello world!” app.

Version Control & Continuous Integration Environment
• If you have a distributed team simultaneously working on the same code base
• If you need to keep your code baseline free from breakages introduced by bug fixing
• If you want to know code version where new error appeared for the first time, and to
identify what change had introduced it
• If you value the possibility to test isolated code deltas knowing exact scope of changes
• If you realize all advantages of automated testing on each and every code version
published
One cannot overestimate the impact of mature CM on resulting product quality,
especially when appropriate tools are selected and efficient CM practices are used by the
entire team to store on-project deliverables and track features, tasks and defects found.

9

QA Core Components
QA Dependencies – Code Development
There are a lot of aspects in a mature Code Development
process that will influence product quality either directly
or indirectly.
Besides personal skills of talented developers, it is always
helpful to apply unified code production process to support
overall code quality during implementation phase.

Unit testing

CODE
DEVELOPMENT

Code best
practices

Reviews

Unit Testing, Coding Best Practices & Code Reviews
• For unified coding style suitable for further product maintenance and updates without
single team member lock-in
• For earliest breakage detection and the most cost effective bug fix
• For test-oriented code design, especially helpful in embedded and remote testing
• For precise error localization and minimal impact on dependent product components
• For knowledge sharing among the development team members and personal skill
improvements
Well-established and regularly applied best code development practices allow to introduce
more effective product testing, since the test team can concentrate on behavioral defects
while architectural weaknesses and logical bugs will be filtered out on early stages.

10

QA Core Components
Testing Process Components
Testing is the core process among all
other QA activities.

TEST
MANAGEMENT

Scope/Planning

As a product matures, it proceeds
Monitoring
through test subprocesses and
Reporting
activities until it is presented to
customers for their evaluation. Each test
activity adds a unique value to find defects as early
as possible in the product development life cycle.
Testing is implemented in separate and/or
overlapped test phases/types:

TEST
PREPARATION

Design
Requirements
traceability

Automation

TEST
EXECUTION
Test passes
Defect tracking

Escaped
defects
analysis

• Testing on different product levels:
unit, feature, features interaction, system;
• Testing during product maintenance:
smoke, sanity, regression;
• Testing of special aspects:
bring-up, stress, stability, performance, interoperability, compliance;

11

QA Core Components
Test Management
Testing is a complex set of many different test activities closely
related to other project processes. All of them require to be
planned / monitored (and re-planned during the project life, if
it is required) thoroughly and competently.

TEST
MANAGEMENT

Scope/Planning
Monitoring
Reporting

The Test Management subprocess is performed from the
very beginning through the end of a project to achieve goals of improved
product quality.

Major Objects of Test Management’s Attention
•
•
•
•
•
•
•
•

Testing Scope / Strategy
Approaches and Constraints
Features / Components / Functional Areas to be or not to be Tested
Test Item Pass / Fail and Test Cycle Suspension / Resumption Criteria
Test Tasks and Deliverables, Environmental Needs
Responsibilities, Staffing and Training Needs, Collaboration with other project groups
Schedule, Risk and Contingencies
Reporting and Quality metrics

12

QA Core Components
Test Preparation
Test Preparation subprocess defines tests and
test environment required for each test phase/type in order
to fully verify and validate a product according to test plans.

Basic Test Preparation Activities

TEST
PREPARATION

Design
Requirements
traceability

Automation

• Analysis of functional requirements, specifications and
functional areas' principles of work
• Development of Requirements Traceability Matrix (RTM)
• Design and development of the test cases for all planned phases/types. The following
approaches and techniques are used to ensure effective and optimal test suites:
• Various test approaches (Black-box, White-box)
• Effective test techniques (Functionality, Equivalence classes, Boundary, Negative/Positive,
Combinatorial, Model-based, Etc.)

• Development of additional tools/utilities and Test automation
• Development and setup of specific test environment (quite often it requires a very
complex hardware and software solution which in turn requires very high engineering
skills from test engineers)
Obviously, test products themselves must have good quality and thus all project test
deliverables undergo a careful review and validation process.

13

QA Core Components
Test Execution
The goal of the Test Execution subprocess is straightforward –
apply test definitions elaborated during the Test Preparation
phase to the software products.
Test Execution activities are either scheduled according to
defined test plans or initiated on demand on an event-driven
basis.

TEST
EXECUTION
Test passes
Defect tracking

Escaped
defects
analysis

Usual Test Execution Activities
•
•
•
•
•

Preparation of developed test environment
Execution of test cases
Deep analysis of execution results
Collecting/Recording of test execution results and failures found
Support development team efforts to reproduce and to analyze failures, as well as
validation of fixes
• Analysis of escaped defects
• Update test cases/test environment according to results of test execution
• Update test cases/test environment according to results of Escaped Defects Analysis

14

QA Core Components
QA & Automation
Test automation, especially within Continuous
Integration (CI) development cycle, has proven to be
the most cost-effective approach while increasing
the testing speed and quality, eliminating human
errors and reducing testing expenses.
It is a useful instrument in a long-lead product
evolution and maintenance. In some cases
test automation is the only method when test
engineers can keep pace with software
developers and verify each software version with
the same reliability and quality level as previous ones.

CONTINUOUS
INTEGRATION

However, it is necessary to understand that automated testing
is powerful aid to improving the return on the testing investment when used wisely. Only
competent project management, test preparation, including requirements analysis,
automated test tool selection/development, and test scenarios development can really
increase testing efficiency.

15

QA Core Components
QA Assets & Tools
As per our experience, Testing is performed in a more
effective manner when suitable tools are used on the
project. They allow to keep all QA-specific information
in a definite common accessible places, perform
planning / monitoring / reporting more effectively and
collect valuable QA metrics.

Different types of QA management assets and tools used are:
•
•
•
•
•
•
•
•
•

Project / Test Plans, Activities and Tasks
Environment and Hardware Resources
Requirements
Requirements Traceability (test cases vs. requirements)
Test Cases
Product Builds & Code Versions
Test Results
Failures Found (Defect tracking)
Project / Test processes progress metrics

16

Customizable QA
Seamless Integration into Customer Process
Here at Rhonda Software we know that each company
has its own SW development process with unique
characteristics and dedicated practices / activities
adapted for specific project needs.
There is no point to change things that are already good
enough and work as a well-oiled machine. Instead let’s
incorporate beneficial QA practices and improvements inline with yours development
framework where they give most noticeable effect.

Our goal is the lightweight and effective QA process and seamless integration
of improvements into customers’ existing SW development process
Our customizable QA services are:
• Core testing activities – test preparation, execution, management
• Test automation
• Requirements elicitation and management
• CM and Continuous Integration
• Introducing effective practices for using Task/defect tracking tools

17

Rhonda Software QA Services
Case Study – Multi-network Test Environment
Good technical skills helped our SW test
engineers to prepare a multi-network environment
and successfully provide testing services to one
of our customers to test dual modes mobile
products which support cdma2000, 1xEV-DO
and Wi-Fi networks. This very complicated and
labor-consuming task was done in a short period
of time without any external trainings or help
available to our team.
CDMA/Wi-Fi lab was successfully installed by
our engineers based on Agilent 8960 Series 10 Wireless Communications Test Sets and
Linksys Wi-Fi routers.
This test lab has since been used to verify mobile applications that require CDMA and
VoIP voice calls (including inter-networks calls), MO and MT SMS over CDMA and Wi-Fi
networks, 1xEVDO data services and data over Wi-Fi (browser, audio/video streaming,
Java applications), CDMA and Wi-Fi handoffs, etc.

18

Case Study – Computer Vision (CV) Testing
These are some of CV systems developed by Rhonda:
• myAudience demographics solutions
• Traffic Lights monitoring system
• Face Detector IP core for FPGA

In order to assure the quality of system we used
different levels of testing:
• Local modules unit-tests which can also be run on CI
server (Jenkins)
• Functional testing in user scenarios
• Web-portal UI acceptance auto testing (using Codeception framework)
• End-to-end product auto-testing (scenarios start from device with camera and run till calculated
data is returned via API from web-portal)

In addition to usual testing practices, development of CV products required additional
specific testing tasks.
• Evaluate system performance metrics: recognition rate, classification rate, detection rate, counting
precision
• Support optimal parameters calibration:
• CV algorithms often depend on multiple parameters that can not be easily found
• One way to find good parameters is to run the system on different combinations, measure quality and
select the best ones

19

Case Study – QA in Cloud-based SaaS
One of current Rhonda Software clients provides
all-in-one cloud-based solution for disaster recovery,
backup and data storage. And it is a quite challenging
task to sustain best quality, performance and reliability
during continuous evolution and development cycle.

Specific QA activities are applied to ensure
continuous defects free production
• Iteration-based Scrum agile process
• Test-driven development approach with JUnit, RUnit unit tests
• Continuous Integration environment with automated testing – functional, integration
and performance for each component
• Additional UI auto-tests for Web-console user interaction component
• Two-step deployment process after successful integration / auto-test:
• Mandatory staging proof cycle that includes complementary feature / bug fix & regression
testing in production-like test environment
• Post deployment regression testing on production servers for escaped defects control

• Continuous system health remote monitoring

20

QA Team – Highlights and Areas of Expertise
There is no doubt that all aforementioned QA practices and
processes are very helpful and wholesome. However, there is
one more very (maybe the most) important QA process
component – our QA Team.
Practical skills and technical background allowed our QA
engineers to conduct testing on most modern and complex
equipment (especially in the telecom domain) on the customers’
side and to install and setup own test laboratories.

EMBEDDED
NETWORKING

MULTIMEDIA
COMPUTER VISION
DRIVERS

• Strong communication skill set proven by interaction
PROTOCOLS
experience with many of our partners around the world
• All engineers have Master’s degrees in Computer Science
USER-LEVEL APPS
• Successful on-site testing to support testing process
whenever and wherever needed. Many of our QA engineers visited customers’
facilities in the USA, Great Britain, Japan, Italy, Germany, France, Israel, South Korea
and Taiwan
• Deep knowledge of different platforms, programming languages and technologies
• Experience in custom test tools and frameworks development, allowing to complete
sophisticated testing tasks

21

QA Team – Experience & Customers
Our QA engineers have participated in numerous projects for different customers, for
example:

Motorola
•
•
•
•
•
•

Android Kernel/BSP
Multimedia & DRM
Email client
i-mode
CDMA/WiFi DMH
MDS Test process improvement

CSR (former Zoran)
• Embedded solutions for digital camera processors
• Image processing (still capture, video recording)
• Networking

Exalt
• Microwave Digital Backhaul radios

Mentor Graphics (Embedded Group)
• Testing of Linux BSP ports on new silicon

22

QA Team – Instruments & Tools Skill
Rhonda Software as QA service provider deals
with various tools and instruments working on
customer’s projects.

Tasks

Tools

Job
Done

Tools for QA planning and test management
• MS Project – Commonly used tool for project and test
activities / tasks planning
• JIRA – Tracker tool used to capture and organize issues, assign work, and follow team activity
• Rational DOORS – Requirements management solution
• Assembla – Set of cloud-based task and issue management tools for software developers
• TestRail – Test case management software for QA and development teams
• Selenium – Suite of tools to automate web applications for testing purposes across many platforms
• TeamCity – All-in-one pre-integrated solution for Continuous Integration
• RSpec – Behavior-driven development (BDD) framework for Ruby programming language
• Cucumber – Tool for running automated acceptance tests written in a BDD style
• JUnit – Unit testing framework for Java programming language
• Mockito – Open source testing framework for Java
• EasyMock – Library that provides an easy way to use Mock Objects for given interfaces or classes
• TestNG – Testing framework inspired from JUnit and NUnit with new useful features

23

Thank You!

Andrey Mischenko
CEO, Rhonda Software
Mobile: +7(423)257-1008
E-mail: ceo@rhondasoftware.com

Denis Pomogaev
VP of Technology Innovations
Mobile: +1(224)715-1154
E-mail: dp@rhondasoftware.com


24

QA Services for Configurable Software

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a QA Services for Configurable Software

Semelhante a QA Services for Configurable Software (20)

Último

Último (20)

QA Services for Configurable Software