Towards an Effective Process Improvement Platform: Spago4Q and the QEST nD Model

EuroMed-SPI 2010 Conference
Sofia (Bulgaria), October 19-20 2010

Towards an Effective Process
Spago4Q and the QEST nD Model
Improvement Platform

Luigi Buglione
Gabriele Ruffatti
Sergio Oltolina
Engineering Group - Italy

Fulvio Frati
Ernesto Damiani
Mauro Regoli
Dipartimento di Tecnologie dell’Informazione
Università degli Studi di Milano – Italy

www.eng.it

Engineering Group At a glance

_ The first Italian ICT player
_ more than 730 M/€ revenues Research and PA & HC Finance Industry TELCO Utilities
Development
_ 1000 clients
_ 6,300 IT specialists System Int. &
Consultancy
% 46 70 54 80 80

Outsourcing % 35 10 27 10

Software
% 19 20 19 10 20

ERP IT Security ECM

Plant Management
Managed Operations Broadband & Media
System

www.eng.it

2 www.eng.it

SESAR Lab Università degli Studi di Milano

• SErvice-oriented Software Architecture Research Lab within the Università
degli Studi di Milano (Italy) headed by Prof. Ernesto Damiani

• The research activities mainly focused on:
 Software Engineering
 Knowledge Management
 Business Modeling
 Open Source Development paradigms
 Semantic Web
 Enhanced Security Architectures
 Mobile Nets interaction and Geolocation
 Distributed Systems
 Net Protocols

• The staff is composed by full time Professors, Researchers, Post-Docs, PhDs
and Research Collaborators.
sesar.dti.unimi.it

3 EuroMed-SPI 2010 – Sofia, October 19-20, 2010 www.eng.it

SPAGO4Q & QEST nD Goals of the presentation

 G1. Discuss the relevance of a SPI program properly set up
 G2. Propose an OSS based-solution joining a multidimensional
performance management framework
 G3. Present a case study describing the implementation of a
multidimensional performance model in a context of Application
Management Services


Spago4Q & QEST nD Agenda

• Introduction
– A bit of humor…
– Need for SPI tools
• Our Open Source Solution
– QEST nD model
– Spago4Q platform
• Multidimensional Analysis Framework for SPI
– Dimensions of Analysis
• Case Study
– Description
– Results
• Conclusions & Perspectives
• Q&A


Introduction A bit of humour…

URL: www.dilbert.com


Introduction Some initial questions…

Q: What does it mean SPI in your ‘organizational
language’?

Q: How (and from who) it has been perceived right now?

Q: Is there a way to measure it?

Q: Which possible improvements for achieving it?


Introduction Need for SPI tools

• The continuous improvement of software process has become a goal of
paramount importance for organizations focused on software development

• Such a scenario of integrated information is known as Business Intelligence
and encloses all the business processes and tools used by organizations for
data acquisition

• In order to achieve this goal, a number of commercial and open source
frameworks has been developed and distributed implementing mathematical
schema aimed at measuring process performance and suggesting process
areas needing improvements.

• Tools have to be tightly coupled with formalized SPI framework to foster solid
and reliable analyses

Introduction Related Works
• A fundamental problem facing software process engineering is verifying and
ensuring that the mandated process is actually carried out and improved, if
necessary
 Requires monitoring of performed process steps and produced work products
• Needed information can be obtained
 Monitoring actual projects as they are carried out
 Using a combination of manual and automatic recording
 Exploiting a post-mortem analysis of process enactment records
• Early efforts in this direction included TAME (Tools for a Measurement
Environment), and SoftMan projects
 Focused on single-process, single-project data collection
• Holkar project produced a prototype quality monitoring system
 Used XML as means of both capturing data from sources and storing results in a
repository
• Commercial tools such as Polarion and 6th Sense Analytics do not support cross-
project comparisons and are not meta-model driven
• Spago4Q fills the gap introducing a complete multi-process multi-project
monitoring framework providing data for a complete process improvement
analysis


• Introduction
– QEST nD model
• Case Study
– Description
– Results
• Q&A


Our OS Solution Introduction

Our work is aimed at showing the integration of
• QEST nD model, a conceptual framework for measuring process
performance based on multiple analysis dimensions (e.g. economic,
social, and technological dimensions)
 www.semq.eu/leng/modtechqlm.htm

• Spago4Q, the open source platform to measure, analyze and monitor
quality of products, processes and services
 www.spago4q.org

Source: Buglione L. & Abran A., QEST nD: n-dimensional extension and generalisation of a Software Performance Measurement Model, International Journal of
Advances in Engineering Software, Elsevier Science Publisher, Vol. 33, No. 1, January 2002, pp.1-7


Our OS Solution QEST model

Method: Performance is expressed as the combination of the specific ratios
selected for each of the 3 dimensions of the quantitative assessment
(Productivity - PR) and the perceived product quality level of the qualitative
assessment (Quality - Q)

Performance = PR + Q

Model: QEST (Quality factor + Economic,
Social & Technical dimensions) is a
“structured shell” to be filled according
to management objectives in relation to
a specific project
Such a model has the ability to handle
independent sets of dimensions without
predefined ratios and weights - referred
to as an open model
Source: Buglione L. & Abran A., QEST nD: n-dimensional extension and generalisation of a Software Performance Measurement Model, International Journal of
Advances in Engineering Software, Elsevier Science Publisher, Vol. 33, No. 1, January 2002, pp.1-7


Our OS Solution QEST model – Geometrical Indicators

Target: measuring project performance (p) using 3 distinct viewpoints
Input Data: list of weighted ratios for each dimension and quality
questionnaires
Output Data: an integrated normalized value of performance

It is possible to measure performance considering at least 3 distinct geometrical
concepts:
• Distance between the tetrahedron base
center of gravity and the center of the
plane section along the tetrahedron
height – the greater the distance from 0,
the higher the performance level;
• Area of the sloped plane section – the
smaller the area, the higher the
performance level;
• Volume of the lowest part of the
truncated tetrahedron – the greater the
volume, the higher the performance level.


Our OS Solution QEST model – Key Features

• Integrated quantitative and qualitative evaluation from 3 concurrent
organisational viewpoints
• A 3D geometrical representation at a single project phase (usually after
the project is completed)
• Use of de facto and de jure standards (e.g. ISO/IEC 9126 for the Quality
Factor)
• Performance Measurement Model to use for consolidating Balanced
Scorecard (BSC) measurement outcomes
• Extension of the original 3D model to n possible dimensions-
perspectives  QEST nD through the simplex as the mechanism to solve
the problem from the 4th dimension on


Our OSS Solution Spago4Q Platform

Spago4Q (SpagoBI for Quality) is:
• An open source platform for the continuous monitoring of software quality (
www.spago4q.org)
• A vertical adaptation of SpagoBI, the Open Source Business Intelligence suite
(www.spagobi.org)
• Pure open-source (no commercial version), with commercial support by
Engineering Group
• Recently adopted by the European Commission –
Directorate General for Regional Policy (DG-REGIO)

Main features
 Multi-process multi-project monitoring
 Data Collection run in a fully, transparent way
Equipped with extractors specific of most-common software process
environments (IDE, workflow management, text editing, …)


Our OSS Solution Spago4Q Structure


Our OSS Solution Spago4Q Metamodel


Our OSS Solution Definition of the Integrated Environment

• The definition of a QEST nD model is a multisteps procedure fully
supported by Spago4Q

• The procedure is coherent with the PMAI (Plan-Measure-Assess-Improve)
cycle:
 PLAN, defining a set of metrics, based on the GQM approach, and
possible dimensions of analysis (perspectives) characterizing the analysis
 MEASURE, including the collection of data, and the computation of
metric values and global performance value
 ASSESS, presenting results through dashboards and reports
 IMPROVE, analyzing in detail each value below expected thresholds in
order to find possible problems or bottlenecks from a process based
viewpoint

Source: Buglione L., Misurare il Software. Quantità, qualità, standards e miglioramento di processo nell’Information & Communication Technology, FrancoAngeli, 3/
ed, 2008, ISBN 978-88-464-9271-5


Our OSS Solution Step 1. Metrics & Model Definition

Declaration of a complete GQM, with the definition of
 the analysis dimensions
 the concepts to measure
 the metrics to apply to project’s workproducts


Our OSS Solution Step 2. Weights & Thresholds Definition

Couple each metric with the
respective weight
 Indicates the importance that
such a concept plays in the
dimension it belongs to

Define the specific thresholds
 Evaluates the value with respect
to organization policies

Assign (if provided) QF to each
dimension
 Give to each dimension a quality
evaluation


Our OSS Solution Step 3. Data Gathering

Measures are taken directly from Spago4Q data warehouse
 The DB is filled by data automatically collected by extractors
accessing process work-products (code package, text documents,
project information, …)
 Supported RDBMS: MySQL, ProstgreSQL, Oracle 10g, etc…

Metrics are described in terms of:
Name of the model to which the metric is assigned to
Default value
Minimum and maximum values (for normalization)
KPI computation algorithm


Our OSS Solution Step 4. Overall Performance Calculation

• Overall and dimension-wise performance indexes are computed
as KPIs that take in input configuration data and results of the
metrics

• The performance value of each dimension is calculated as the
weighted sum of each selected measure by its assigned weight
for that dimension


Our OSS Solution Step 5. Reporting

• Sets of reports and dashboards could be defined and
configured to satisfy reporting and managerial needs

• Spago4Q provides methods and interfaces to directly
configure and create new reports using all the facilities
provided by Eclipse BIRT open source reporting tool



• Introduction
– QEST nD model
• Case Study
– Description
– Results
• Q&A


Multidim. Analysis Dimensions of Analysis

• Four analysis dimensions:
1. Economical (E)
2. Technical (T)
3. Resource Usage (RS)
RS
4. Customer Satisfaction (CS)
CS

• Each dimension is characterized by a specific metrics set for
process evaluation

• Performance values for each dimension allow to identify
process areas that need improvements



• Introduction
– QEST nD model
• Case Study
– Description
– Results
• Q&A


Case Study Project description

• The case study is based on a three-year project
• Application Management (AM) services
 Software Maintenance (Corrective, Adaptive, Perfective, Preventive)
for a large mission-critical system in a Finance Institute

• Services started in 2006
 Analysis period : January 2008 – June 2010

• Data was collected from tools supporting development
environment
 Data gathered with the Polarion (www.polarion.com) Application
Lifecycle Management (ALM) tool


Case Study Objectives

 Verify QEST nD applicability and results in a context of AM Services
 Define a QEST nD model aligned to the AM services goals
 Monitor the effectiveness of improvement action with specific goals and
metrics
Example of adopted goals and improvement actions

 Goals
 EC-G3 Reduce the rework (intended as impact of defects in UAT or production environment)
 TE-G1 Improve the deploy process
 TE-G5 Improve effectiveness of peer reviews

 Improvement actions
 Deploy process automation and automatic analysis of source code
 Progressively increasing of the number of peer reviews on critical work products
 Specific tasks were included in Impact analysis phase at the aim to:
 Classify and identify critical Work Products to be reviewed
 Assign an owner to solve complex defects impacting on different development streams
 Root-cause analysis of the recurring defects


Case Study QEST Model - Dimensions & Metrics

The model defined for the case study take into account four analysis
dimensions and goals as follows:
1. Economical (E)
E.G1 Reduce the effort of corrective maintenance
E.G2 Improve the number of delayed deliverables
E.G3 Reduce the rework (intended as impact of defects in UAT or production environment)

1. Technical (T)
T.G1 Improve the deploy process
T.G2 Reduce the resolution time for defects and technical issues
T.G3 Improve quality of documents and source code
T.G4 Reduce the rework (intended as impact of defects during development phase)
T.G5 Effectiveness of peer reviews
T.G6 Improve non regression test

1. Resource Usage (RS)
RS
RS.G1 Reduce impact of human resource management issues
RS.G2 Improve hardware system availability

1 Customer Satisfaction (CS)
CS
CS.G1 Improve user satisfaction 2010 – Sofia, October 19-20, 2010 and application services
about training courses www.eng.it
29 EuroMed-SPI

Case Study QEST Model - Dimensions & Metrics
Dimension Metric Description Formula
Economical (E)
E Incidence of Corrective Maintenance Effort w.r.t. maintained code size Corrective Maintenance Effort/ KLOC
Ratio Corrective Maintenance Effort - Adaptive Maintemance Effort Corrective Maintenance Effort/ Adaptive
Maintenance Effort
Incidence of Delayed Deliverables w.r.t. total number of Deliverables no. Delayed Deliv. / no. Deliv.
Incidence of Defects after system test w.r.t. total number of Defects no. Defects in UAT or production / total no. of
Defects
Resource Usage Human Resources management issues w.r.t. total number of issues admitted for no. HR issues / no. Issues for group size
(RS
RS) working group size
Hardware System Availability Percentage System Availability
Technical (T)
T Technical management issues w.r.t. total number of issues admitted no. Technical issues / no.issues admitted

Issues Mean Resolution Time Total Res. Time / no. Issues
Document quality: respect of document quality standard Percentage of positive response to a
checklist
Software Complexity Results of automatic static code analysis
Coding rules non-conformity level Results of automatic static code analysis
Software Maintenability Results of automatic static code analysis
Incidence of Peer Reviews w.r.t. total number of Deliverables no. Peer reviews / no. Deliverables
Number of Defects discovered by peer reviews w.r.t. total number of Defects no. Peer review defects / total no. defects
no. Defects / FP
Incidence of Defects Due to Design Phase w.r.t. total number of Defects no. Defects(Design phase) / Total no.
Defects
for any phase p
Test coverage w.r.t. Requirements no. Test Cases / no. Requirements

Production Defects Mean Resolution Time Total Res. Time / no. defects

Customer Training Services Questionnaire results
Satisfaction (CS
CS) User Satisfaction Questionnaire results


Case Study Results - QEST dashboard


Case Study Results – Dimensions trend analysis

Last results for each dimension

Trend for each dimension


Case Study Results – Global and technical % increasing


Case Study Detailed Analysis Results (example)
example

[AM-EC-M.04] Defects reduction in UAT and production environment
[AM-TE-M.11] Defects mean resolution time reduction


example

[AM-TE-M.01] Technical issues reduction: specifically related to deployment process
[AM-TE-M.02] Technical issues mean resolution time


example

[AM-TE-M.07] Number of peer reviews actually executed vs. number of critical Work
Products
[AM-TE-M.08] Defects or potential defects discovered during peer reviews
[AM-TE-M.09] Incidence of defects due to design phase



• Introduction
– QEST nD model
– Dimensions of analysis
• Case Study
– Description
– Results
• Q&A


Spago4Q & QEST nD Conclusions

• Need for SPI tools
 Huge attention from Management for low-cost technical solutions for a better monitoring
& control both at the project and organizational level  OSS
 The right combination should include both methods and tools
• QEST nD
QEST is multidimensional performance model taking into account several perspectives
from interested stakeholders
QEST can be exploited also at the SLC level with LIME (see www.semq.eu)
• SPAGO4Q
Spago4Q is a flexible platform for software process/product monitoring applicable to any
SPI activity
 Current version (v2.3.1) is freely downloadable @ www.spago4q.org
• Case study - Provided evidence
Clear Benefits on measurement process
Greater effectiveness of peer reviews
• Difficulties in achieving process improvements
 Time pressure
 Workload and urgent activities disqualify the team to provide sufficient effort to SPI
activities
 SPI is usually considered as a low priority process
 Continuous commitment is needed

Spago4Q & QEST nD Perspectives

• Spago4Q new services will support the implementation measurement model based on
standard models
Product level: ISO 9126-1 ( ISO 250xx), ISO 25012
Process level: CMMI-DEV, CMMI-SVC, ISO 15504-2, ISO 20000-4, etc., …

• Spago4Q possible enhancements towards newest and more complete analysis
frameworks

• Enlargement of the Spago4Q set of tool extractors to enlarge the set of extracted data

• Creation of a Correlation Matrix from base measures retrieved from the various
systems to make more derived measures available to analyzers and include them in
the project measurement plan (GQM-R)
GQM-R
o Example: having the number of Unadjusted FP (UFP) and the project effort (in
mm/h), the typical derived metrics are
 Productivity, defined as UFP/effort
 Productivity Delivery Rate, defined as effort/UFP

• Implementation of an Automatic/Dynamic (re)calculation of KPI thresholds from
historical data


Spago4Q & QEST nD Q&A

Благодаря ви за вниманието!
Thanks for your attention!

Thanks for your Attention !

We care of your problems and we have in mind a solution

Luigi Buglione, Gabriele Ruffatti, Sergio Oltolina Ernesto Damiani, Fulvio Frati, Mauro Regoli
Engineering Group Università degli Studi di Milano
{luigi.buglione | gabriele.ruffatti | sergio.oltolina } {ernesto.damiani | fulvio.frati} @unimi.it
@eng.it mauroregoli @tiscali.it
@


Towards an Effective Process Improvement Platform: Spago4Q and the QEST nD Model

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (17)

Semelhante a Towards an Effective Process Improvement Platform: Spago4Q and the QEST nD Model

Semelhante a Towards an Effective Process Improvement Platform: Spago4Q and the QEST nD Model (20)

Mais de SpagoWorld

Mais de SpagoWorld (20)

Último

Último (20)

Towards an Effective Process Improvement Platform: Spago4Q and the QEST nD Model