This presentation was given at GRC Conference in Boston (October 2010) and explains the importance of measuring performance for real value. It goes into the world of metrics and balanced scorecards

1. Strategic Governance
Performance Management Systems
Ramsés Gallego
CISM, CGEIT, CISSP, SCPM, CCSK, ITIL, COBIT(f), Six Sigma Black Belt
General Manager
Entel Security & Risk Management
rgallego@entel.es

2. Program
Cause-effect analysis IT investment
Strategy KPIs
Outcomes
Processes Maturity models
Reliability
Alignment DMAIC Efficiency
Data Business goals Normalization
Real-time
Portfolio Proactive
Dashboards
Govern Lean Management
Mapping Indicators Balanced
Scorecard
KGIs Improvement
Metrics
Effectiveness
2

3. The need for IT to reinvent itself
Despite the projections of renewed economic health, the business
will continue to expect IT leadership to show strong financial
competencies, that IT projects realize tangible business value, and
that the IT organization demonstrates competitive effectiveness.
“..IT organizations that rise to the challenge will be rewarded with
substantial opportunities to develop a new type of service
organization. Those that don’t will face a grimmer future”
Gartner – CIO Update - 2009
4

4. We will be talking today about...
Some quotes and definitions
The myths on metrics
The power of Performance Management Systems
Metrics: characteristics & classification
What are CSFs, KGIs and KPIs?
Examples of governance indicators and KPIs
Process and architectures for Performance Systems
The SMART side of metrics & indicators
5

5. Let’s think about this
• ‘Measure what is measurable and make measurable what is not
so’ - Galileo Galilei (1564-1642)
• ‘If you cannot measure it, you cannot improve it’ - William
Thomson (Lord Kelvin), (1824-1907)
• ‘You cannot control what you cannot measure’ - DeMarco, 1982
• ‘Even when it is not clear how we might measure an attribute, the
act of proposing such measures will open a debate that leads to a
greater understanding’ - Fenton and Pfleeger, 1997
5

6. Definitions
Governance: “The set of responsibilities and
practices exercised by the board and executive
management with the goal of providing strategic
direction, ensuring that objectives are achieved,
ascertaining that risks are managed appropriately
and veryfing that the enterprise’s resources are
used responsibly”
6

7. Definitions: what is a performance
management system?
• Tools designed to facilitate decision-making and improve
performance and accountability through collection, analysis and
reporting of relevant performance-related data
• Metrics are simply a standard or system of measurement. In this
case, it is a standard for measuring the value driven by IT, an
organization’s value delivery posture. Although there are some
published standards for measuring the IT dimension, ideally
measurements should be adjusted and tuned to fit a specific
organization or situation
7

8. Goals of this effort
Develop a strategic governance performance
framework that allows management and other
stakeholders to assess their business
improvements (time-relevant), guide their
governance thinking and aid in the assessment
for their environments
8

9. Some myths
• #1 - a little data goes a long way
– Fact: you can only improve what you can measure
• #2 - measurement is for punishing the guilty
– Fact: metrics are for problem solving and identifying opportunity areas
• #3 - we can’t measure what we cannot control
– Fact: measure what you can influence
• #4 - metrics are for measuring people
– Fact: measure the team contribution. They are an organizational tool
• #5 - we must measure everything
– Fact: keep it simple so that everybody understands it
9

10. The power of metrics
• It’s not in the details but in their clarity
• Metrics allow executive management to:
• Measure achievement
• Drive performance
• Improve and realign (towards goals)
• Metrics should provide a holistic and balanced view of the
business
• Need to talk about RoI
10

11. Metrics: what is needed?
• The 7 attributes of Information criteria (also known as the “IC
Profile”)
Key conditions before defining a framework:
• Having a pre-defined business process
• Having clear goals/performance requirements
• Having quantitative/qualitative measures for
the business process
11

12. Characteristics & classification
• Process
Secure coding standards in use
• Objective/Subjective Avg. time to correct critical vulnerabilities
• Quantitative/Qualitative • Vulnerability metrics
By vulnerability type
• Static/Dynamic
By ocurrence within a software development life
• Absolute/Relative cycle phase
• Management
• Direct/Indirect
% of applications that are currently accepted by
business partners
Trending: critical unresolved, accepted risks
12

13. Let’s be specific
• Name of the metric
• Description of what is measured
• How is the metric measured
• How often is the measurement taken
• Range of values considered normal for the
metric
• Best possible value of the metric
• Units of measurement
© Source: Vicente Aceituno’s presentation for the FIST conferences in Madrid, 2008
13

14. CSFs, KGIs, KPIs: what are they?
• CSFs: Critical Success
Factors or “vital elements”
• KGIs: Key Goals Indicators or
“what” has to be accomplished
• KPIs: Key Performance
Indicators or “how well” the
process is performing
14

15. KPIs and KGIs reflect organizational goals
15

16. Example of metrics and KPIs
• % reduction in repeat security incidents
• Increased number of secure assets from risk analysis audits
• % reduction of blank passwords on critical systems
• % improvement on time-to-access applications
• Improved bandwith use due to only-professional web surfing
• % reduction in the unavailabilty of services and components (linked with
corporate infrastructure management)
• % efficiency improvement based on number of RFCs processed
regarding vulnerabilities
• % reduction in installed software not taken from DML
16

17. Where do we show metrics?
Dashboards and BSCs
• Single point of information for governance
• Help to make decisions and provide real-time answers to managers
• Talk about the business, not about figures!
• Need the involvement of the business and operations to be
developed/designed in order to provide value
• Web and role-based so as to get the right data (becoming the tool that
consolidates siloed information)
17

18. Some dashboards examples
© Business Objects. Crystal Xcelsius dashboard from www.xcelsius.com
© Business Objects. Crystal Xcelsius dashboard from www.xcelsius.com
18

19. Some dashboards examples (II)
© Business Objects. Crystal Xcelsius dashboard from www.xcelsius.com
19

20. Monitor vs. Manage
R A GE
ITO Refine, observe, MA
N
MON analize and
classify data
provided by
Value (and cost)
Act with business
systems knowledge, in a
Centralize single place
according to
access to data Apply business business needs
content and relevance to the
applications information to
determine
business
priorities
DATA INFORMATION KNOWLEDGE ACTION
Level 1
Level 1 Level 2
Level 2 Level 3 Level 4
21

21. The road to manage IT information
Management
Alarm Escalation, Invoke Management
ACTION Response
Console, Response Model
Management/Alert
● email ● Pager ● Cell ●
Presentation
Event Manage/Report Event Display, Trend Analysis, Security
Reports, Performance Reports, Security
KNOWLEDGE Pattern Discovery System Health, Assigning Ownership
Prioritization
Event Correlation
Event Prioritization, Event Associations,
Security Modeling
Event Aggregation
Data Normalization and Log Data Reduction, Event Matching,
Monitoring
Reduction
De-Duplicating Events
INFORMATION
Data Filtering
Data Repository Event Monitoring, Third-Party Integration,
Data Collection/Capture Protocol Support
DATA ● Syslog ● SNMP ● API ●
21

22. Process & Architectures
Query
Policy Reporter
Policies Events Manager
Collector Management Portal
ts
er
Al
Router
Load Balancer Router SunOS Mainframe Windows
X.500 DB IDS/IPS
Switch Directory AIX
Proxy
Network Identity Applications /Hosts Security
Systems Systems Information systems Systems
© 2006 CA - All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.
22

23. Across Business Applications
Division
Customer Order Entry
Customer
Division
Shared Invoicing &
Service Marketing
Receivables
Supplier Vendor Managed
Inventory
Outsourced Shipping
Collections
23

24. To understand the business performance
24

25. To understand the business performance

26. To understand the business performance

27. Using IT in the real world
27

28. From technology...
28

29. ...to what really matters
29

30. The Business side
31

31. The IT Operations side
32

32. How we link them
33

33. What can be achieved
• KPIs that are a measure of how well a process is performing
• The capability of predicting the probability of success or failure in
the future
• KPIs that are business-focused, process-oriented but IT-driven
• KPIs that are expressed in precisely measurable terms
• KPIs that, when acted upon, will help to improve the process
• FOCUS on what is really important and has impact
33

34. The SMART side of metrics
• First business needs, then processes,
then metrics, then tools
• Keep them simple
• Use “as is/to be” & “is/is not” lists
• Metrics should be S-M-A-R-T
34

35. THANK YOU
Strategic Governance
Performance Management Systems
Ramsés Gallego
CISM, CGEIT, CISSP, SCPM, CCSK, ITIL, COBIT(f), Six Sigma Black Belt
General Manager
Entel Security & Risk Management
rgallego@entel.es