1. Abstract: Many organizations have turned to outsourcing to cut
costs. But how can you gain a strategic advantage from an
outsourcing initiative, not just cost savings? This white paper will
provide executives and managers of software product and service
companies with a ―How To‖ guide and best practices for integrating
independent Quality Assurance and testing across the product
lifecycle to shorten software development cycles.

4. Technology companies are challenged to innovate faster than their competition, to deliver their applications to
market first, and to re-balance their development priorities in order to accelerate revenue generation.
Business leaders often view software quality as a luxury - something that can be sacrificed, if necessary, for more
functionality, faster development, or lower costs. However, in practice, software development organizations that have
a firm commitment to quality can actually improve development cycles, reduce costs, and add new features with
greater ease. Organizations that develop low-quality software, whether for sale or for internal use, are always looking
backward, spending time and money on fixing defects in "finished"
products. In contrast, an organization that builds-in product quality
from the beginning can be forward-looking and innovative; it can A study commissioned by the U.S.
spend its resources on pursuing new opportunities. Department of Commerce's National
Institute of Standards and Technology
According to U.S. Department of Commerce, software bugs cost
(NIST) found that software defects
the U.S. economy an estimated $59.5 billion per year. $22.2 billion
cost the U.S. economy almost $60
could be eliminated with improved testing and earlier identification
billion annually. $22.2 billion could be
of errors.
eliminated with improved testing and
earlier identification of errors.
Customer expectations of flawless execution have become par for
the course. In today’s competitive economy, no software vendor
can afford the tarnished reputation or unhappy customers that
result from preventable software | defects. But when it comes down to traditional methods for assuring software
quality, companies are finding it increasingly difficult to give up the valuable time and strategic focus that get
consumed by resource-intensive internal test efforts—a problem that grows larger as development projects advance
in both scope and complexity. It’s no surprise that many companies are turning, with increasing popularity, to
outsourced methods of software quality assurance as a way to ensure the integrity of their products, the satisfaction
of their customers, and the long-term success of their growing development initiatives—all while keeping costs down.
As a software company, you face a unique set of development challenges— more platforms and configurations to
support, more releases to sustain, more testing to be completed, and more products to integrate.
Historically, QA is managed at a discrete project level and not coordinated across the product portfolio. As a result,
there are several challenges:
 Limited strategic direction for QA at an organizational level
 No standardization of tools/processes/test environment
 No centralized reporting or service execution
 A limited view of function level progress/quality
 Product-release delays due to excessive testing time

5.  Lack of quality testing resources, preventing quick application delivery
 Product-release delays due to inability to quickly ramp-up QA team when needed
 Absence of real-time data for managers to evaluate progress of testing
You need tools and practices that accelerate the development process to maintain and extend existing code as well
as to develop and deploy new code—while tightly managing the costs associated with numerous members of your
development team spread out across multiple locations. Your bottom line is dependent on your development team’s
ability to produce quality software.
When it comes to product lifecycle management, there are several primary objectives:
 Shorten time-to-market timeframes
Software companies should rapidly narrow the gap between what the customer needs and what the
software delivers by increasing functionality, adaptability, usability, and manageability.
 Deliver increased product feature sets
There should be a strategic release management plan that optimizes usage of the product
engineering and QA team.
 Improve product quality
Software companies may use the testing and The emerging technology leaders are
validation services of an independent team with constantly improving speed, quality,
specialized skills that stays focused on delivering and cost parameters at each step of
a zero-defect product. the product lifecycle.
 Constantly increase software development
efficiencies
To succeed in the next economic cycle in the software business, companies have to increase
efficiencies and productivity, and create and consolidate software assets into reusable components
and blocks.
 Increase market share and customer/partner retention byfocusing on the strategic activities
and the core business
Software companies can leverage a thoughtful and balanced offshore strategy across the entire
product lifecycle to keep key people focused on strategic activities such as product management,
marketing, sales, and customer service.

6. The Informatics Computer School's Software Engineering course-text defines Software Quality as:
 Conformance to clearly stated functional and performance requirements
 Conformance to clearly documented development standards
 Conformance to implicit characteristics that are expected of all professionally developed software
For any organization, defining what you mean by "quality" is an important first step. Too often, software development
organizations operate with a loose, general notion of quality and
tolerate defects that most engineering disciplines would not allow.
In contrast, a solid definition of quality that all team members
On average, developers make 100 to
understand and accept promotes thoroughness and attention to
150 errors for every thousand lines of
detail. In the domain of business applications, we can best define
code they write. Even if only a small
quality in terms of the target audience: the software users.
fraction—for example 10 %—of these
Development organizations with a quality focus know that a
errors are serious, then a relatively
"quality" application must do more than simply provide correct
small application of 20 thousand lines
results without crashing. Does the application meet stakeholder
of code will have roughly 200 serious
requirements? Is it usable? Secure? Scalable? Reliable? Easily
coding errors.
maintained? Easily extended? Simple to monitor?
James Juran's definition provides a good starting point: quality is
"fitness for use." Juran goes on to say that a product is not high-quality unless it adds value for both the consumer
and the manufacturer.
Put slightly differently, quality encompasses added value plus attention to detail. Both a luxury car and an entry-level
car will get you from point A to point B. But the luxury car offers features and capabilities that go beyond the
essentials of transportation: usability, safety, comfort, reliability, and so on. Product quality also reflects the process
behind the product. In the software world, a high-quality process can keep development organizations from losing
time reworking, re-factoring, and rewriting software. These organizations can produce more innovative and creative
products because they have more time to think about adding value and quality details.
Achieving quality results requires applying a high-quality process throughout development, integration, and testing.
This applies equally to projects involving packaged or homegrown applications, upgrades or greenfield development
work, and extending, integrating, and modernizing legacy applications (see Figure 1).

7. Figure 1: All modes of software development require the same attention to quality.
 Design
New  Develop
Development  Test
 Deploy
Integrated  Customize
 Extend  Expose
Legacy Packaged
 Expose 
Systems Applications Test
 Upgrade  Deploy
But the bottom line of software quality is conformance to the user's needs⎯or requirements,, or wants. The point is,
any definition of software quality must take into consideration the end-user's perspective. An application may do all
that it was designed to do, and do it according to professional standards, but if users hate using it, find it difficult,
counterintuitive, or error-prone, then its quality is at best questionable.
Being free of defects is not enough. For a product to achieve quality, it must also provide exceptional functionality,
usability, and compatibility.
Software Quality Assurance cannot be treated by executives, managers, or engineers as one additional "Task" on
the way to the final launch, or a thing that can be taken care of
as the project proceeds, along the development and release
process. Team members closely associated with the application
"Rigorous inspections can remove up
are less likely to find the problems that an external party would
to 90% of errors before the first test
find easy to spot and report. Independent testing efforts ensure
case is run, but rigorous inspections
that your product is tested by individuals who will be just as
should not replace testing" - Robert L.
objective and unforgiving as your customers in the case of a
Glass, "Facts and Fallacies of
potential defect.
Software Engineering"
This is simply a fact, and not a judgment of or reflection on the
developers' talents. An individual who spends the most time
working on a project intended for an audience cannot simply step back and look at it as though through that
audience's eyes. The creator is too familiar with his work to view it objectively.

8. But this is why Quality Assurance experts exist⎯just asprofessional proofreaders, copy editors, test audiences, and
software do. These independent parties bring much-needed objectivity to the appraisal of a software development
project.
It is crucial that software Quality Assurance be independent of project management for it to be completely effective.
Outsourcing software Quality Assurance to an independent, unbiased source is the most effective way to achieve the
objectivity that a project needs.
When it comes to the practice of independent verification and validation, an independent QA team should be the one
final check-off point before allowing a system to go into operation; how it is done systematically and organizationally
varies, however.
The standardization of software engineering processes, margin compression, and competition are compelling
technology-centered companies to face the task of delivering more complex application development with limited and
decreasing time frames. Software leaders are looking at outsourcing processes across the entire software
development lifecycle, from engineering, to quality assurance and testing, to maintenance and porting.
Software Quality Assurance outsourcing can take a number of paths, testing for different capabilities and performing
different analyses, depending on the demands of the project, the users, and the software itself. But an outsourced
Quality Assurance and Testing initiative carried out by seasoned QA professionals will confer certain benefits:
1. Reduced Development Cycles
2. Improved Customer Satisfaction
3. Reduced Cost of Development
4. Reduced Cost of Maintenance
Coupled with an outsourced business model and its benefits, outsourcing software QA and testing became a
strategic R&D initiative for many technology companies. The expected benefits include:
5. Improved focus of in-house resources on strategic activities and innovation
6. Reduced Total Cost of Ownership (TCO)
7. Reduced technology risks
According to Gartner (2005), “20% of
So what is the best way to leverage QA and testing to shorten
outsourcing deals do not produce cost
development cycles? Build your own QA and testing department?
savings, with 50% of all outsourcing
Create a hybrid of in-house and outsourced resources? Or
projects falling short of delivering the
outsource entirely?
expected value and being deemed
unsuccessful".
As a software vendor, innovation is the foundation of your
success. The earlier in your software development process that
you apply world-class testing and quality assurance practices, the
more efficient your process will be. Advancing other critical path activities, such as internationalization and

9. localization, further upstream in the product lifecycle will also serve to improve efficiency. Software makers who
manage their product lifecycles as carefully as the process-mature manufacturers in, say, the automotive industry,
are sure to be winners as the high-tech industry moves into the next economic cycle.
Outsourcing the testing activity, properly applied, can address many of these issues while relieving management of
many of the burdens of process development, staffing, and ongoing control.
The reasons for outsourcing are many and varied among potential customers. Depending on the nature and maturity
of the company, these could range from the need for limited special purpose testing to taking over quality
management of a product/application portfolio. The former might be done to address specific gaps in the testing
framework, such as a lack of environments for conducting load testing, or areas of particular risk such as security.
The latter may be advisable if your internal resources are unable or have limited capabilities to justify, explain, or
quantify existing internal testing activities. As with any business decision, the choice to outsource and/or offshore
software testing must be viewed in terms of risk and reward. While this will vary from company to company, a few
basic principles should be applied when considering such a solution.
The first step should be to articulate the goals of the initiative. The basic rationale for undertaking this should be
stated explicitly and frame the entire process. A formal strategy statement may also be valuable here to lay out a
roadmap for proceeding, articulating critical risks and assumptions, estimating resource requirements, and
determining measures of success.
Some of the possible reasons for deciding to outsource include:
"...the average enterprise will
1. Deficiencies in the specialized skills and ultimately outsource 60% of
processes required to support testing application work offshore (circa
2008/09)."— META Group
2. Problems related to poor test practices such as
excessive expenditures on maintenance and
critical failures in production
3. Lack of test infrastructure including test environments and test beds
4. Insufficient testing activity to warrant a full-time testing staff because the portfolio is too small or
there are only a limited number of releases during a year
5. Lack of facilities or expertise to conduct special purpose testing
6. Independent validation and verification of development work performed by other vendors
One other area to note where an organization might gain significant long-term benefits from outsourcing is in
implementing test automation. Manual testing and tracking is a complicated enough task and, although using
automated test tools can help improve efficiency and control, they are not an immediate panacea. Such tools must
be carefully selected to match the technology and environments used by a system. The tools must be set up, scripts
should be written and maintained by experienced professionals who know how to configure the package, set up
databases, design test artifacts, determine and enforce privileges, establish library control procedures, train staff,
and enforce standards. By engaging a service provider who knows how to set up and manage these tools, a

10. company can gain valuable experience in using them without some of the pain associated with starting up such an
initiative internally.
The final aspect of making a decision about whether or not to outsource Quality Assurance and testing is to
determine the organization’s readiness to support testing. An evaluation of existing processes, skill sets, and
infrastructure is recommended in order to set the stage for planning a test outsourcing initiative. An important part of
any such assessment is how the existing QA or testing organization will interact with the vendor.
The chapter entitled, ―Five Phases of Integrating QA Into the Entire Global Software-development Cycle" provides
step-by-step guidelines for how to design and manage your Quality Assurance and testing program.
Based on the goals articulated by management, a formal framework for moving forward should be developed. Some
potential models for outsourcing that might be considered include:
1. Total off-site outsourcing of all testing functions for the entire portfolio including offloading the
existing testing staff to the vendor.
2. Conducting outsourced testing on a pilot basis with the intention of developing internal processes or
introducing automation.
3. Partial outsourcing of special purpose testing (e.g. performance or security) or testing level (e.g.
outsourcing system integration testing but not unit testing or user acceptance testing).
4. A vendor-managed internal testing program in which all work takes place on site, applications
remain in the customer environment, test beds are kept in repositories controlled by the client, but
the test management and staffing is provided by the vendor.
5. Third party validation of outsourced products, involving independent testing of applications built by
another vendor.
Total off-site outsourcing has the advantage of reducing internal staff expenditures while improving the efficiency and
effectiveness of the testing function. It leverages the knowledge of the vendor to the largest extent. This model tends
to work best for customers who have well-documented requirements and well-educated stakeholders. In the absence
of those conditions, there may be a ramp-up period in which the vendor will have to spend time organizing
requirements and familiarizing internal staff with their roles. In addition, the relationship between the vendor and
developers must be managed carefully; delays in the delivery of software will affect testing schedules and the
amount of testing, which can be performed.
The final consideration in a pure outsourcing arrangement is the fact that the product will be under the control of the
vendor during testing. Arrangements must be made to assure the security and integrity of programs and data during
this time. A risk worth mentioning with a pure outsourcing approach is that the client can become out of touch with
the quality management of its own products. It is incumbent on the customer to review, interpret, and act on test
artifacts and test results even if the entire testing operation is run by an off-site test team.

11. The major benefits of running an outsourcing pilot are that targets and costs can be controlled selectively and future
strategies can be modified based on the outcomes and findings from the pilot. When choosing pilot applications, it is
usually wise to find those of enough significance and size to make the activity meaningful but not those which are
business-critical. This allows clients to learn important lessons about testing applications in their domains without
putting core systems at risk. Establishing predetermined criteria for the success of the pilot is vital to extracting the
proper lessons from it.
Partial outsourcing of special purpose tests or by testing level can be an extremely effective way for an organization
with existing test infrastructure to enhance its quality management
program.
By engaging vendors with extensive
A vendor-managed internal testing program is a hybrid business
experience in a particular specialty,
model, which can serve multiple needs. In this scenario, all work
such as performance testing, a
takes place on-site and applications remain in the customer
company can leverage its knowledge
environment. There is greater control and much less risk of
and environments, reduce investment
compromising systems or data. The vendor may design and
in hard to find skills, minimize the
create testing procedures or use existing processes, but they can
need for expanding test infrastructure,
bring additional value to the organization by working with internal
and receive independent verification of
best practices and enhancing them. The vendor may also develop
results.
processes for controlling test beds, however it is recommended
that the repositories are designed and managed by the client’s
infrastructure team. Since the test management and staffing is
provided by the vendor, the customer does not have to invest in finding the skills needed to establish the test
infrastructure and staff the test teams. One other approach that is gaining popularity is third-party validation of
applications built by another vendor. This trend is pronounced with mid-market clients who have decided to
outsource most of their IT function. It can be a cost-effective option for organizations with little experience with
software development and testing or for those who have very few or small software projects. While it does not
contribute to the client developing its own testing processes, it is a good model for those who do not wish to invest in
that infrastructure. In addition, the test cases and test data are the property of the customer and can be reused in the
future should the outsourcing relationship be discontinued.
One issue that may arise when adopting this approach is contention among vendors. Performance of the testers can
be affected negatively by developers who do not provide good test requirements, cannot provide feedback on tests
in development, or fail to deliver on time. The client in this situation must maintain control of the process and must
take steps to assure that the vendors are working in a manner that supports the company’s goals.
At the end of the day, how does an executive determine the value of outsourced testing and whether it is worthwhile
to undertake? Potential buyers should realize that the benefits to be derived from test outsourcing may go beyond
those gained through outsourcing other functions.
Those who have survived in the tough conditions of the recent IT economy are well aware of quality issues and the
need to deliver fully tested products. The financial motivation to find and fix bugs before product release is not a new
notion, however. A study by noted industry analyst Capers Jones back in 1996 clearly showed the staggering costs

12. associated with post-release bug detection. Post-release bug detection usually occurs when software companies
decide to cut costs by reducing testing schedules in early phases of development.
Figure 2. Cost of repairing defects across application lifecycle
The fact that closer attention to testing is vital to keeping overall long-term costs down is often obscured by near-
term budget constraints. Here we run into a paradox: more testing resources and enhanced QA requirements are
paramount to a company's success, but their budgets simply cannot support increased spending.
There is a solution to this seeming contradiction. As Steven Brier, senior analyst and managing director of research
at Roberson Stephens, notes, the secret to success in the IT landscape these days is to focus on running the core
business while effectively using outsourcing options. "It's often cheaper and faster to give it to someone else to do,"
says Brier {UPSIDE, June 2001).
In terms of quality assurance, this means that developers can stay within their core competencies by unit testing with
highly specialized debugging tools, such as jtest® from ParaSoft®, while outsourcing functionality, compatibility, and
integration testing. This allows software engineering organizations to capitalize on what they do best, i.e. write code,
while delegating the task of testing how their product functions in the complete solution stack to an outside team with
proven expertise in quality assurance.
Additionally, in many companies, because of repetitious test processes, the turnover rate for test engineers is very
high. This means that an in-house test lab must continuously spend time and money to seek and recruit skilled
workers and train them for its purposes. In comparison, outsource test organizations offer clearly defined career

13. paths to their employees, reducing turnover. Plus, due to the nature of the business, projects in outsourced testing
labs are always in flux, which keeps career test engineers interested in working with and discovering upcoming
technologies. As a result, these engineers are highly trained, enthusiastic, possess a wide variety of core
competencies, and are efficient in carrying out complex test projects.
It is important to understand that the savings realized through more effective testing don’t automatically fall to the
bottom line. Even if an outsourcing arrangement does directly reduce the cost of testing, most IT organizations will
probably still spend their up to their budget limits. The savings can be and usually are spent on development and
other productive tasks in support of core business functions. But some of the payback of instituting an effective
testing program is derived in other ways, including:
 More efficient staff and resource management
 Ability to investigate test automation in a low-risk setting
 Reduction of time spent on rework and repair
 Minimizing and managing product risk
By enabling the software engineering department to make more effective use of the internal resources, both testing
and development benefit. Developers can focus on the job of developing the software without worrying about
planning, designing, and executing tests. This can shorten lead times to get products to market and result in better
systems because developers will be focused on just one task - engineering - which has to be the only core
competency. Dedicated, trained testing resources who understand the overall framework of quality management will
be much more effective in constructing and implementing an organized approach to validating software. The testing
process will be more efficient and effective. First of all, highly trained specialists will be better at this task.
Effectiveness will also be enhanced since an outsourcer is more likely to provide an unbiased assessment of product
quality simply because they are operating independently from the developers. Finally, the costly and time-consuming
task of setting up and managing test environments will no longer be a factor in determining resource allocations. A
good provider of testing services should either have the appropriate hardware and software infrastructure in place or
be able to generate it at less cost and in less time.
The management of special-purpose testing will also be enhanced. Performance and stress testing can be done with
more breadth and depth. Regression testing can also be enhanced. The key to effective regression testing is to
maintain existing test scripts so that they can be re-run when changes are made. However, most companies don’t
have experience in determining which scripts to keep, how to store them in an organized fashion or when to re-run
them. A qualified service provider will have these skills on staff or make them available, when needed.
Outsourcing to a service provider who has extensive experience with automated test tools also provides a buyer the
benefits of working with a partner knowledgeable in this field. They can assist with defining the tool set to be used,
installing and configuring the product, managing it on an ongoing basis, and training staff in its use. Automated tools
can also be used to improve regression testing, as they provide facilities for storing and managing test scripts and
test data. When one considers that a significant proportion of test tools purchased are never used effectively by the
buyers due to lack of experience, the return on investment is often gained simply by using the very products they
purchase. Then there are the benefits of better product quality. Put simply, it is advantageous and economical in the
long run to test software systems effectively to assure that they meet their requirements rather than correcting them
later.

14. The amount of time and money spent by software engineering departments on application ―maintenance‖ can
exceed 50% of the annual budget. This calculation also does not take into account the overall cost of poor quality
which includes the time and resources required to identify defects, determine their location and cause, repair them,
retest them, and promote them again into the production environment. Finally, there is what may be the largest cost
of poor quality: business lost and the company’s reputation tainted by exposing end-users and customers to
systems, which cannot perform their functions accurately and dependably. This could result in inaccurate order
processing and billing, unexecuted transactions, miscalculations, inability of customers to do business during high
volume periods, generation of inaccurate reports, etc. Any one of these circumstances can cause customers to
discontinue their relationship with a company, and they may very well convince their acquaintances to do so as well.
The last major benefit to be derived from working with a vendor to establish a sound testing program is better
information to support management decision-making. Prioritizing requirements provides a sound basis for evaluating
product risk and determining areas for concentrating quality management or improvement programs. Measurements
of test completion and coverage represent necessary information for determining if and when products go live. Root
cause analysis of incident reports can lead to significant improvements in the product definition and development
process. A vendor can make recommendations for making the development effort more efficient and improving the
quality of its deliverables. It is this benefit, which is not immediately quantifiable in monetary terms, which can
contribute to the long-term success of the enterprise by helping senior management assess risk, set priorities,
establish product improvement programs, and generally provide a real-world basis for evaluating investments in
technology.
The bottom line? In some cases, development organizations prefer having their own test departments for reasons
that go beyond financial considerations. For one, they believe close coordination between test and development
resources requires proximity. In these instances, a hybrid solution of outsourced test specialists working on-site,
backed by external lab-based testing, may be the optimal solution. In either case, today's budget-minded and high
velocity organizations avoid missed time-to-market opportunities and enjoy lower over¬all costs of quality assurance
through testing outsourcing.
Whether you are a software product company and sell software or develop software to use internally, it is important
to begin with the discovery of business opportunities across the application lifecycle.
Choosing the testing processes that software companies want to outsource is a critical component of their
outsourcing initiatives and can have a tremendous impact on their success. The stakeholders should take the current
inventory of development and testing processes, resources, and tools before they start identifying objectives,
expectations, and metrics to measure the success of the company.

15. To forge a successful outsourcing strategy, companies should conduct an outsourcing ―readiness‖ assessment. The
assessment should be equally applied to companies considering outsourcing processes across the entire product
lifecycle, or to those who wish to outsource QA and testing only.
The results of the assessment will provide an organization with much of the information required to make key
decisions and to focus management effort. The assessment can help examine the ability of a software development
organization to outsource by looking at the processes maturity, experience, tools, and ROI objectives.
Data for assessment may be gathered from:
1. Questionnaires
2. Interviews
3. Discussion with product managers and application owners
4. Study of product roadmaps, test plans, test coverage, test scripts inventory, and application/code
base inventory
Deliverables may include:
1. Outsourcing Readiness Report
2. ROI analysis
3. Application/Project Sequencing Plan
4. Engagement Roadmap
When reviewing test plans, test coverage, and test scripts inventory, the following criteria are recommended to
assess readiness of specific processes:
1. Skill alignment with current/future quality assurance and testing direction
2. Current cost of testing/quality assurance
3. Application maturity
4. Alignment to business (core, context, utility)
5. Business value
6. Functional completeness
7. Technical completeness
8. Rate of change
9. Test coverage
10. Perceived risk profile

16. Successful completion of this phase will provide the company with insight to areas of the software quality assurance
and engineering, which have room for improvement. Gap analysis naturally flows from the results of the
Assessment/Business Case phase and involves determining, documenting, and approving the variance between
business objectives and current capabilities.
At this stage, companies should analyze gathered materials with the goal of comparing its actual performance with
its potential performance, and of finding gaps that represent opportunities for improvement.
The objective is to study the differences between the current and desired state/best practices of the quality
assurance and testing processes, for the purpose of determining how to get from one state to a new state. Gap
analysis is undertaken as a means of bridging that space. Among the various methodologies used to perform gap
analysis is IDEF, a group of methods used to create a model of a system, analyze the model, create a model of a
desired version of the system, and to aid in the transition from one to the other.
With that information in hand, a company can pinpoint and address the quality/process gaps. The ultimate goal is to
identify and compare a list of initiatives, which you can begin implementing to improve the speed, quality, and cost
parameters at each step of the product lifecycle.
In this step, companies define a high-level project plan and a test strategy for reaching the desired QA destination.
The following steps outline the major components of a typical outsourced QA project.
1. Overview –Includes the objectives for QA on this project and background information.
2. Scope – Contains what will be covered and what will not be covered in the QA effort.
3. Assumptions – Includes any assumptions made in developing the QA Strategy.
4. Project Organization – Contains the project organization chart and how the QA Team fits within.
Roles and responsibilities respective to QA and other stakeholders. Specify Project Team
responsibilities for test infrastructures and data.
5. Project Steps – Address how the QA effort will be approached. Include findings from Proof of
Concept. Include one or more schematics, along with written descriptions of the proposed QA
workflow. Describe types of data and validation methods.
6. Levels of Testing – Define what types of testing will be performed such as Unit, Integration,
System, Data Validation, Performance, GUI and the entrance and exit criteria for each.
7. Testing Groups - Testing groups provide a breakdown of major functionality. List the testing groups
and clearly state what they include and what they do not include.
8. Technical Assessment – Content depends on current status of development, technical
infrastructure, and type of technologies and platforms. In this section, the following should be done:
Assess compatibility of data structures along with any code or software specific issues. Address
feasibility of test automation with recommendations for using home-grown or vendor tools at certain

17. stages of the development cycle. Include recommendations regarding what should be automated for
testing.
9. Test Infrastructure Requirements - Assess existing infrastructure processes and needs. Discuss
test infrastructure considerations and requirements for loading converted data and ongoing data load
processes. Include environment sizing considerations and depth of test data. Clearly state how the
test infrastructure is and is not analogous to the production infrastructure. Note any exceptions or
limitations.
10. Processes and Tools – Describe processes and tools to be used for change control, configuration
management, and discrepancy reporting. Describe what test automation will or may be used and
how it will or could be performed.
11. High Level QA Project Plan - Schedule with tasks and associated resources for successfully
completing testing.
After implementation of the project, companies have to return to their initial assessment and gap analysis to compare
the results of the project with the initial expectations.

18.  What to Automate  Quality of Software
 What Tool to Use  Impact of Failure  Nature of System
 Manual Tools  Likelihood of Failure  Strategic Significance
Test Tools Risks Application
Test Test
Objectives Infrastructure
Test
Strategy
 Demonstrate Usability  Code Control
 Intrfaces  Enviroments/Progression
 Coverage  Hardware
 Stress/Load  Software
Review Test
Resources
Strategy Requirements
 Money  Walkthroughs  Completeness
 Time  Inspections  Accuracy
 People  Desk Checks  Stability
 Skills
 Priorities
 Hardware
 Software

19. During the execution phase the following metrics are used:
1. Planned test execution versus actual: are we on track? (could identify a risk for being on time)
2. Number of successful versus not successful tests (too low means inefficient testing due to low
quality of the test object)
3. Trend in outstanding number of defects (going up could mean a risk for being on time)
4. Burn rate of the test hours per phase (too high could identify a budget issue)
The metrics #1, #2, and #3 are essential to the success of an outsourced QA process and should be tracked weekly.
These metrics are mandatory for Quality Assurance projects with a duration of the execution phase longer than two
weeks.
By applying criteria to the metrics during a running test project, important signals can be derived as to whether the
outsourced testing project is going well and whether the quality of the test object is reasonable.
Towards completion of the project it is advisable to add the following metrics:
1. Has the test coverage reached sufficient level?
2. Is the percentage of successfully executed tests high enough?
3. Are the remaining defects below a certain level?
These metrics play an important role in the acceptance process.
When the execution phase is finished some additional metrics can be determined:
1. Number of hours spent on test execution per test case
2. Number of hours spent on test execution per defect found
3. Number of hours spent on testing as percentage of the whole project
When these metrics are collected for a number of comparable projects, they play an important role for supplier
management purposes (how efficient and effective is a test supplier) and in general to drive improvements.
When the project is over, it is recommended that companies analyze defects found after the deliverables were
accepted:

20. 1. Number of defects found after acceptance compared to before acceptance
2. Test specification progress per week (number of test cases written) versus planned
3. Defect solution turn around time (are all showstoppers solved within a week?)
4. Defect root cause analysis in all phases (are there particular weak spots in build?)
The following mistakes are the most common when it comes to outsourcing Quality Assurance and testing activities:
1. No ownership of product quality.
2. No overall test program design or goals.
3. Non-existent or ill-defined test plans and cases.
4. Testing that focuses narrowly on functional cases.
5. NO ad hoc, stress, or boundary testing.
6. Use of inconsistent or incorrect testing methodology.
7. Relying on inexperienced testers.
8. Improper use of tools and automation, resulting in lost time and reduced ROI.
9. No meaningful metrics for tracking bugs and driving quality back into development.
10. Incomplete regression cycle before software release.
Future State, and Required Roadmap
Software vendors often make decisions about what and how to outsource too quickly, without detailed expert
assessment of their current state, desired future state, and required path on how to get there. It is critical to plan
outsourcing strategy by considering a company’s current software engineering processes maturity, in-house team
size, available documentation, and experience with distributed software development, cultural fit, and executive and
staff support. It is recommended to access applications, source code base, and business processes, and to lay out a
matrix of how and in what sequence applications and processes can and/or should be outsourced, kept in-house, or
retired.

21. The cost expectations that determine the savings that organization can leverage as a result of outsourcing are often
over-estimated. Although they can be based on ROI projections, software vendors should consider the learning
curve for the offsite team to become productive. For several years now, the business press has been claiming that IT
work costing $40-$80 an hour in the United States can be done for $15-$25 an hour in India or Russia. If those
figures sound too good to be true, that’s because they usually are. An offsite team often will not become productive
(by onshore standards) for at least three to four months, or even longer for more complex projects. IT executives
should expect to pay an additional 5 percent to 15 percent on managing an offsite outsourcing program, at least
during the first year. The transition phase will add costs, too — planned expenses should cover due diligence,
transfer of knowledge, and extensive program management.
Once a company has decided to outsource, it usually rushes to get the outsourcing deal done and put the
transaction on a fast track. To do that, executives from the company and the outsourcer meet, agree, and then let
the next level of management sort out all the details. Since an outsourcing process is a complicated one, it is
advisable that various elements of a company remain involved in the original meetings and have a say in the
agreement. Top-level executives alone are rarely sufficient to review such a process effectively.
The essence of outsourcing is that a company transfers the process of application engineering to the outsourcer, and
then buys the results of that process. The quickest and most certain way to destroy an outsourcing process is to
dictate to the outsourcer how the process should be done. When a company tells the outsourcer what to do, it
removes from the outsourcer the ability to add value to the process. As the result, the outsourcer cannot deliver
quality software.
Many companies choose not to transfer the ownership of the outsourcing process to the outsourcer, and instead opt
to keep the process under company control. When the company assigns someone to oversee the process, this
person usually becomes overwhelmed and is unable to manage it, because it is an area that he or she has very little
knowledge about. As a result the outsourcing process fails.
The longer your contract terms, the more deep you get in it. Information-driven enterprises and ASPs should
construct a contract that will last a long time, but that will allow the service term's agreements to change during short
periods of time. It is preferable for a company to develop a close relationship with an outsourcing company, but to
sign short-term contracts that can be easily negotiated and, if necessary, renegotiated or even broken.

22. Companies new to the outsourcing process often do not assign the right people to manage the process. They often
assume that since all the details of an outsourcing contract are discussed and the contract is signed, that they do not
have to worry about the outsourcing process any longer. The right person for the job is someone who understands
the process, is able to focus on soley on the process, and who can interpret the results with the best interest of the
company in mind. He or she should have a different way of thinking, in respect to most business managers, and the
oversight to recognize any problems or mishaps before they surface.
Far too often, the company forgets that the outsourcer is a business asset and must be treated as such. Team
members across the globe are vital parts of a successful and prosperous business. If the outsourcer is treated
differently— for example, like one of many suppliers or one of many customers—then the one thing that the
outsourcer adds to a company will be lost.
 It is preferred that companies inspect in person the physical premises where the software is to be
developed. This is an opportunity to check the security firewalls of their vendor’s buildings and work
area, the organization culture, the functioning of their networks, etc.
 It is preferable that companies interview the team members who will work on their project. This will
help companies to judge the employees' level of reliability.
 Companies should also check the offshore development company’s employee retention rate, and
also determine if the outsourcing vendor is working with any competing organizations. If this is the
case, companies should ensure that the teams working on their competitors’ projects do not have
access to their project information.
 Any method of information transfer, such as e-mails, fax, electronic file exchange, instant
messenger, on-line meetings, paper documentation etc, should have parameters for usage clearly
defined.
 All activities that will have to accompany the end/termination of the contract should be defined while
negotiating the contract terms and conditions. This includes the retrieval of any methods and
procedures, documents, source and executable code, company proprietary security or development
standards, code libraries, and data stored offsite.
 Contracts should be framed, such that the offshore company takes responsibility for the actions of its
employees.

23.  Companies should ensure that any project-related work is not subcontracted without approval. This
will help to protect a sensitive application or data within the application.
 Companies should ensure that the vendor agrees not to use any of the company’s confidential
information for purposes like sales, marketing, or demo without prior approval.
 Companies should ensure that only data related to the performance and reliability of the system is
transmitted over the Internet. Information from the database should not be disturbed during any part
of the project.
 Companies should make sure that the system experts make modifications to the system only after
obtaining prior permission.
 Exchange of passwords and other critical information should be made secure by encrypting them.
 Companies should ensure that the data used during testing does not expose the real information of
the customers.
 Unwanted data should be destroyed.
 Companies should ensure that the vendor reports on replacements of team members, if any.
 Companies should ensure that an original copy of the source code is maintained.
The business benefits of including quality-oriented activities in all phases of your software development lifecycle are
both broad and deep. These measures not only facilitate innovation and lower costs by increasing predictability,
reducing risk, and eliminating rework, but they can also help to differentiate your business from its competitors. Most
importantly, continuously ensuring quality will always cost less than ignoring quality considerations.
Renat Khasanshyn is CEO of Altoros Systems, LLC. www.altoros.com

24. 1. James M. Juran and Frank Gryna, Juran's Quality Control Handbook. McGraw-Hill, 1988
2. Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement, Jeff Tian
(Wiley-IEEE Computer Society Press, 2005)
3. Managing Successful IT Outsourcing Relationships, Petter Gottschalk and Hans Solli-saether (Jul
2005)
4. "Software Quality Assurance: Principles And Practice" by Nina S. Godbole Alpha Science
International, Ltd (2004)
5. Test Outsourcing: Risks and Rewards, 2006 Rich Yannetti 545 Fifth Avenue 4th Floor, Ste 401 New
York, NY 10017
6. Metrics for Outsourced Test Activities, Kees Blokland / Polteq IT Services B.V. (NL) 2004
7. Data Warehousing QA Challenges, Denise Copening, Olenick & Associates
8. http://en.wikipedia.org/wiki/Gap_analysis
9. Software Quality Assurance Improvement Plan: EPIcode Gap Analysis; May 2004, U.S. Department
of Energy
10. Facts and Fallacies of Software Engineering; Robert L. Glass; Addison-Wesley (2002)
11. See Watts S. Humphrey's article entitled "The Quality Attitude," published in the Software
Engineering Institute (SEI ) newsletter, (2004, no.3): http://www.sei.cmu.edu/news-at-
sei/columns/watts_new/watts-new.htm