As software becomes more integrated into our daily lives, companies are finding that visibility into the systems that run their business has many benefits: reduces business risks, increases revenue, and improves IT spending.
This whitepaper provides a framework for capturing the impact of software analytics on your business and a worksheet to help you create your own business case. Leaders that can clearly articulate this value are more successful than their peers in obtaining strategic support and funding for software analytics.
Learn the Fundamentals of XCUITest Framework_ A Beginner's Guide.pdf
ย
The business case for software analysis & measurement
1. White Paper
The Business Case
for Software Analysis
and Measurement
2. The Business Case for Software Analysis and Measurement
Page 2
Executive Summary
The notion of measuring application development has long been a
controversial one. Yet as application development and maintenance
(ADM) matures as a discipline and measurement capabilities evolve
organizations are finding that the ability to effectively measure
application development output can lead to many benefits:
Objective visibility into application development output
A rational basis for ADM improvements decisions
Accountability based on real data not subjective judgments
The identification of ADM best practices
Improved IT spending decisions
Comparison of ADM performance over time
or against outside organizations
Prediction of future ADM behavior based
on historical performance
But measurement cannot be for measurement sake -- when
implemented properly measures should prompt or compel an action.
Those actions can include: increasing oversight, verifying results using
another data source, investigating root cause, changing processes,
shifting priorities, mitigating or remediation actions, and sometimes
software intensive business, at the forefront of those actions are ones
that help reduce cost, improve productivity, and mission effectiveness.
The challenge; however, is that while businesses can quantify the cost
of their system failures, they struggle to build a business case to
justify investments in the systems needed to identify and prevent these
failures.
This paper presents a framework for capturing the impact of
a software analysis and measurement system and illustrates how it
improves executive visibility; helps prevent business risks,
can support revenue growth, and improve ADM spending.
3. The Business Case for Software Analysis and Measurement
Page 3
a collection of
measurements and
indicators in three
areas: the relevance
of the functionalities,
the intelligence of
the development,
Pierre Dulon, CIO
Crรฉdit Agricole CIB
Compelling action through
software analysis and measurement
While most IT organizations have some measurement in place the
vast majority of metrics are focused on the application development
project processes and IT service levels. While it is important to
understand how IT services are performing, these measures (number of
severity 1 incidents, number of help desk tickets, uptime) are reactive
and therefore only provide a historical view of what has happened.
Process metrics such as project hit rates and budget adherence only
tell part of the story. Both sets of data simply inform us about the
current situation with little insight into the root cause that led us here or
insight into what we can expect to happen in the future.
Introducing measurement of the output of the application development
process the software itself - with existing measures of the IT service
levels and project performance creates insight in three areas:
Visibility into the risks in critical business systems and
understand how they may impact your ability to execute
business or service your customers.
Gain a root cause understanding into the technical risks
and system complexity and how they impact your project
new features on time.
Ability to measure ADM team productivity and vendor
performance and to effectively measure the overall quality
of their output.
4. The Business Case for Software Analysis and Measurement
Page 4
Combing Project Data with Software
Analytics enables the objective measurement of ADM
output and team and vendor productivity. It also helps
predict ADM costs, creates transparency into vendor
cost and identifies potential risks to project team
performance.
Marrying IT Service data with insight into the systems
that support these services identifies the technical
risks or vulnerabilities in the systems that can disrupt
business.
Figure 1 illustrates how analysis of the software enhances IT Service and Project data to create new meaningful insight
Shift from cost to value
The inclusion of product-based metrics into your analysis framework
creates a fundamental shift in thinking; one that shifts us from a cost
control mindset to a business value focus.
The traditional view of IT as a cost center has ingrained a cost control
culture within organizations. By incorporating metrics that describe
ADM output in business relevant terms such as the amount of functions
we shift our
t
improve IT services in ways that add value to an organization --
Coupling IT Service, Project Data with
Software Analytics enables early identification of
development risks, root cause of project risk and
complexity and makes the effective measurement of
ADM throughput possible. IT SERVICE
DATA
SOFTWARE
ANALYTICS
PROJECT
DATA
5. The Business Case for Software Analysis and Measurement
Page 5
"We need precise
measurements that
enable us to easily
define the issues
where we can deliver
the most value, and
also to identify the
teams who are
showing some
momentum in terms
Olivier Morbรฉ,
Associate and
Managing Director
Boston Consulting
Group
rather than simply reduce cost.
However, this transformation from cost to value is only possible
once we include an objective, standards-based assessment of the
code itself, which is often a critical input into the ADM process
and is always the most important ADM output.
Software analysis and measurement is the intelligent use of insight
into the state of an application to improve IT investment decisions,
operational performance, and customer outcomes. Generating software
metrics involves the analysis of the software artifacts created or
managed by a development or maintenance team. It includes insight
about the structural quality of application, metrics about the
and functional or technical size.
Description Prevent
Transferability
Determines how easily a new team or
team member can be productive when
assigned to work on the application
SME Dependency, ramp up delays
Delivery inefficiency, reduced output
Time-to-market delays
Changeability
(ISO)
Determines how easily and quickly an
application can be modified
Correction and evolution delays
Late delivery of new features
Inability to resume services
Robustness
Determines the risk of failures or
defects that could occur in production
Operational downtime
Application outage
Inability to test source code updates
Performance
Determines the risk of performance
issues of an application
Application Degradation
Response time degradation
Denial of Service, Logic issues
Security
Determines the risk of security
breaches for an application
Damaging Business & Operations
Security failures
Determines the cost and
difficulty/ease to maintain
Figure 2 provides a description of the characteristics of software systems and their impact on IT services.
Maintainability
(SEI)
Organizations such as the Consortium for IT Software an application
Quality publish standard definitions of software characteristics.
Drifts on maintenance costs
6. The Business Case for Software Analysis and Measurement
Page 6
The value of software analysis and measurement
When coupled with traditional measures, software analysis metrics
provide value across an organization from creating executive visibility to
early warning of on-coming business risks to actionable insight to fine-tune
the ADM organization.
INCREASE REVENUE
โข Improve organization agility
โข Improve time to decision
(opportunity cost per day)
โข Improve ADM throughput
REDUCE ADMSPEND
Software analysis and measurement prevents business risks
Visibility into the vulnerabilities of critical systems enables business unit
leaders to identify issues that will disrupt IT services, impact customer
satisfaction, and potential impact company brand image. Software
analysis enables business unit leaders to understanding whether a
PREVENT BUSINESS RISKS
โข Identify risks that disrupt
business processes, impact
customer satisfaction and
company brand
โข Enhance your reputation
as analytic-driven leader.
โข Get more with same
โข Do same with less
(cut wasted resources, underperforming
projects, underutilized applications)
โข Negotiate better with business unit leaders,
IT finance / CFO for IT budgeting
7. The Business Case for Software Analysis and Measurement
Page 7
imperative for CIOs to
resting state and
establish productivity
performance
indicators that help
organizations assess
Paul Dottle, CTO and
acting CIO, American
Express
client facing system has potential performance and stability issues
before being deployed to production. This helps prevent loss of
revenue and customer satisfaction -- a major problem for U.S.
companies who are losing $26.5 billion in revenue due to downtime
each year.
This visibility reduces production issues by finding these weaknesses
fast and reducing downtime cost -- which has reached an annual cost
of 127 million hours lost. Similarly, companies are now able to avoid
regulatory compliance violations due to outages while improving
customer satisfaction with better uptime and system performance.
Business risks often present the easiest case for quantifying quality
benefits, since the cost of quality problems can be expressed as the
loss of known current or future revenue, the cost of underutilized
workers, liquidated damages, and other costs based on historical
business data.
Consider the loss due to an outage in a commercial application such
as a customer order system. The costs involve lost revenue, effort for
the business to recover, spikes in help desk inquiries, liquidated
damages, and other costs that may be unique to the specific area of
business. Although not all costs may be triggered in each outage, they
should be considered to ensure they do not remain hidden in ongoing
business activities.
Gaining visibility into system vulnerabilities has lead to drastic
improvement to system uptime and availability. Our clients experience
a minimum reduction of 10% of their production outages through the
effective use of software analysis and measurement. Recently, a global
software analysis and measurement as part of an Application Quality
Management initiative across 20 teams with the aim of reducing system
outages, break / fix cost and production incidents due to change
requests.
8. The Business Case for Software Analysis and Measurement
Page 8
The client established the software analysis improved overall quality
of output over time and that software analysis identified 30% of
application issues that lead to production issues. This visibility into
ADM output led to significant cost avoidance of over $500,000 annual
savings in cost to remediate defects in QA and production.
Software analysis and measurement increases revenue
The revenue impact of software analysis is twofold: improve
organizational agility and increase ADM throughput.
The agility of a business in responding quickly to opportunities or
threats is strictly limited by the systems the business relies on to
service its customers. The more needlessly complex the architecture
or code the longer it takes to add functionality, verify its correctness,
and deliver it into operation. Worse, unnecessary complexity induces
more mistakes and rework, lengthening the time to develop and
transfer new functionality into operation.
By identifying the areas in the system that prevent teams from
operating at peak performance, organizations are left with a code
base that is more easily changed or extensible to new features and
integrations. This is achieved through structural quality analysis and
a standard assessment of the systems changeability, robustness, or
transferability. This assessment creates the visibility needed to navigate
decisions to improve the overall product thus enabling teams to
perform at optimal levels.
Secondly, improving ADM throughput means that more features,
enhancements, upgrades can be delivered at the same ADM spend
levels. The ability to objectively measure output using function points
and quality enables organizations to optimize their output resulting in a
reduction of rework, more efficient QA, and improved developer or
vendor productivity.
9. The Business Case for Software Analysis and Measurement
Page 9
The value of software analysis and measurement to business agility is
in terms of lost opportunity cost. It represents the lost revenue or
market share experienced when competitors can respond more quickly
when the respons
importance of business agility cannot be overstated for long term
business viability and growth.
Typical improvement of time to market ranges from 10-20% depending
on the industry and product function. Similarly effective productivity
measurement programs supported by software analysis improve
developer productivity by 20% or more.
In this case, an international retailer needed to improve time to market
of new features. An analysis of the system and development team
determined that 65% of the developers time was spent understanding
the existing code thus prolonging their ability to make modifications or
fixes. As a result, CAST was able to blueprint the system architecture
creating system level visibility. This enabled the architect and
development team to perform better impact analysis and reduce non-coding
time by 60%, resulting in a 30% reduction in time to market.
Developer
Productivity Impact
Before CAST
After CAST
60%
Less time
required
Time to Market
Impact
6
18
Months Months
10. The Business Case for Software Analysis and Measurement
Page 10
Software analysis and measurement reduces ADM spend
The value to improving the software development lifecycle is a worthy
discussion yet merits its own conversations. Therefore, we will limit our
discussion to the value that software analysis has at a management
level -- that is a solution that creates visibility into the murky view of
develo
cost associated with it.
Vendor spend
When vendors manage applications, oversight and control of that work
is relegated to service level agreements. As such, due to the objective
nature of SLAs, visibility into the actual work performed, the state of
not possible.
However, by applying standard assessment of the work and quality
delivered by vendors has demonstrated significant value to early
adopters of structural quality measurement. The ability to understand
how much work was actually completed in a given time frame, how that
compares to internal team or industry performance is a powerful
executive weapon.
Coupling that data with an objective assessment of the risk and
complexity of the delivered code empowers organizations to better
understand service provider charges and eliminate unnecessary
disputes or over-charging. Service providers are able to see more
clearly the complexity of the work they are taking on, reducing the risk
premium for estimates. Typically, our clients are experiencing 5%
improvement to vendor spend based on better estimates and clarity
into change order effort.
For example, a major consumer services institution deployed software
11. The Business Case for Software Analysis and Measurement
Page 11
analysis across three strategic application service providers which
represented over $2 billion in spending. Through benchmarking and
able to drive down maintenance cost by as much as 12% as well as
improve release hit rates and mean time to repair.
Productivity
Cost Effectiveness
Mean Time to Repair
Quality
Internal application development and maintenance spend
Introducing software analysis as an assessment of internal team
effort provides many benefits as well. At the most basic level, software
analysis and measurement creates:
Visibility into system vulnerabilities that disrupt
project team performance
Standards-based analysis that acts as a rational
and fair basis of comparison of team performance
The ability to understand the technical condition of the systems they
support helps ADM management reduce maintenance cost and rework
by proactively allocating resources to high risk areas and shifting
priorities to remediate complexity and technical risk prior to system
Best in
Class
Good
Average
3.01
Technical Code Quality
2.77
CAST Quality
3.02 2.96
2.66
3.31
Average TQI
February 2010 โ June 2012
Release Hit Rate
Throughput
74%
Release Data
February 2010 โ June 2012
75%
76%
84%
85%
87%
Feb 11 June 11
Grp 1 Grp 2 Grp 3
Defect Density
Quality
Pre Production Defect Density
February 2010 โ June 2012
Pre Production Mean Time to Repair
February 2010 โ June 2012
Cost per Function Point | Enhancement
February 2010 โ June 2012
Productivity
Cost Effectiveness
Cost per Function Point | Maintained
February 2010 โ June 2012
0.70
0.86
0.06
--
0.23
0.18
0.07
0.17
2011.02 2011.06
Grp 1 Grp 2 Grp 3 Grp 4
2.85
16.75
14.32
30.18
4.29
12.09 11.80
14.30
2011.02 2011.06
Grp 1 Grp 2 Grp 3 Grp 4
$1,989
$3,320
$957
$2,603
$3,132
$1,901
$1238
$2,193
2011.02 2011.06
Grp 1 Grp 2 Grp 3 Grp 4
$58
$9 $6
$95
$55
$8 $6
$78
Q1 2011 Q2 2011
Grp 1 Grp 2 Grp 3 Grp 4
Grp 1 Grp 2 Grp 3 Grp 4 Grp 5 Grp 6
๏ง Benchmarks development team
productivity, quality and hit rates
๏ง Measures against other towers, and
trending over time
๏ง Release hit rate up by 10%
๏ง Maintenance cost down by 5-12%
๏ง Defect densities down by 3x
๏ง MTTR down by >25%
12. The Business Case for Software Analysis and Measurement
Page 12
deployment. This provides a residual benefit that reduces QA / testing
efforts, reduces developer rework rates and reduces total ownership
cost of the system over time.
A fair and objective basis of comparison enables benchmarking
and monitoring of team performance, helping management identify
underperformers and non-standard performance for improvement
or elimination.
Lastly, this visibility, industry benchmarking, and ability to prove
performance improvements arms ADM leaders with the insight and
facts needed to improve their dialog with business unit leaders and
finance departments. The ability to secure funding for additional
resources or to prove team effectiveness is an intangible benefit that
leads to true alignment between IT and the business.
A major insurance institution had launched a continuous improvement
initiative across their internal development teams. By analyzing one of
their critical systems, claims processing system, the company realized
improvements to the stability of the system, improved their delivery
times, and reduced maintenance cost by 20% over three years.
56% reduction in
defects in 4 years
๏ง Maintainability stabilized despite
40% increase in code over 4 years
๏ง Reduced delivery time by 60%
๏ง Reduced costs 20% over 3 years
Early Defect
Detection Impact
Year 1 Year 2
Defects / KLOC
Year 3 Year 4
13. The Business Case for Software Analysis and Measurement
Page 13
Summary
The impact of software analysis and measurement is based on how
systems and the scope with which they are deployed.
Organizations that claim that they are too immature for measurement in
fact have the most to gain. Regardless of process maturity, businesses
are constantly changing and managing that change is hard. Measuring
transformation programs is even harder but what you can't measure
you can't manage so it's worth the trouble - especially if you need to
prove it to those outside IT.
The inclusion of application development output metrics in your
executive reporting or dashboards is invaluable to driving IT
transformation as they are normalized denominator for software
development productivity and quality.
Understanding this value is important; however, leaders must master
the ability to communicate value in business-oriented language
in terms of its contribution to their business.
Leaders that clearly articulate this value are more successful than their
peers in obtaining strategic support and funding for software analysis
and measurement that increase business agility and reduce cost and IT
risks to the business.
14. The Business Case for Software Analysis and Measurement
Page 14
About CAST
CAST is a pioneer and world leader in Software Analysis and Measurement, with
unique technology resulting from more than $120 million in R&D investment.
CAST provides IT and business executives with precise analytics and automated
software measurement to transform application development into a management
discipline. More than 650 companies across all industry sectors and
geographies rely on CAST to prevent business disruption while reducing hard IT
costs. CAST is an integral part of software delivery and maintenance at the
Founded in 1990, CAST is listed in NYSE-Euronext (Euronext: CAS) and services
IT intensive enterprises worldwide with a network of offices in North America,
Europe, and India.
North America Europe
321 W. 44th Street, Suite 501 3, rue Marcel Allegot
New York, NY 10036 92190 Meudon - France
Phone: +1 212-871-3330 Phone: +33 1 46 90 21 00
www.castsoftware.com
15. The Business Case for Software Analysis and Measurement
Page 15
Whatโs the Value of Software Analysis and Measurement?
Use the worksheet to learn more about Software Analysis and Measurement.
PREVENTBUSINESS RISKS YOUR DATA HERE
INDUSTRY
AVERAGE
ANNUAL DOWNTIME Hours 87 Hours Gartner Research
COST PER HOUR OF UNPLANNED DOWNTIME $ $250,000 Gartner Research
TOTAL ANNUAL COST OF DOWNTIME $ $21,750,000
Annual Downtime X Cost
Per Hour
PERCENT OF OUTAGES DUE TO SOFTWARE FAILURE % 30% Capers Jones
ANNUAL DOWNTIME COST DUE TO SOFTWARE $ $6,525,000
Annual Cost of Downtime
X Percent of Outages Due
to Software Failures
REDUCTION OF PRODUCTION OUTAGES % 20%
CASTClients Experience
10-30% Improvement
EXPECTED BENEFIT $ $1,305,000
INCREASE REVENUE YOUR DATA HERE
INDUSTRY
AVERAGE
EXPECTED REVENUE GROWTH
From new features/enhancements delivered to market
$ $60,000,000
NEW REVENUE AT RISK
Due to troubled, late, under delivered or cancelled projects.
% 60% Standish Group
TOTAL REVENUE AT RISK $ $36,000,000
Revenue Growth X
Revenue at Risk
Percentage
NEW REVENUE AT RISK PER WEEK $ $692,308
Total Revenue at Risk /
52 Weeks
AVERAGE TIME TO MARKET DELAY 12 weeks
TIME TO MARKET IMPROVEMENT
Identify troubled projects early, improve developer and QA efficiency, eliminate technical risk in critical systems
% 15%
CASTClients Experience
10-20% Improvement
TIME TO MARKET BENEFIT 1.8 weeks
Average Time to Market
Delay X Time to Market
Improvement Percentage
EXPECTED BENEFIT $ $1,200,000
REDUCE ADM SPEND YOUR DATA HERE EXAMPLE
APPLICATION DEVELOPMENT AND MAINTENANCE BUDGET $ $50,000,000
PERCENT OF BUDGET ALLOCATED TO IT VENDOR 70%
INTERNAL ADM SPEND IMPROVEMENT
Identify underperforming teams, better resource allocation, Hawthorne Effect on developer and QA productivity.
% 10%
CASTClients Experience
5-20% Improvement
VENDOR SPEND IMPROVEMENT
Better estimates on enhancements and change requests, objective dialog with vendors, Hawthorne Effect.
% 5%
CASTClients Experience
5-15% Improvement
EXPECTED BENEFIT $ $3,250,000