Endasol Streamlining Bi Implementation, Euci March 2008

Considerations for Streamlining Your
Business Intelligence Implementation
March 2008

© 2008 Endasol, Inc. All Rights Reserved.

Brief Introduction - Endasol (www.endasol.com)
Focus Areas
Data Warehouses/ BI Efforts
Lead Roles in PJM Enterprise-Wide Data Warehouse prior to forming Endasol
Called back to PJM to help develop a separate Market Monitoring Data Mart
Just underway with helping to identify business requirements and early design for
a major data mart at an ISO.
Business Analysts for a couple different energy companies’ major data
consolidation & reporting projects
Developed Comprehensive Solution for Wholesale Power P&L + Other KPIs
Systems Integration – ETRM, Scheduling, Settlements, Generator
Management, Fuel Inventory
ISO Operations & Interfaces - PJM, MISO, NYISO, ISO-NE
Developed Streamlined Interfaces to many of the PJM eSuite Applications
Direct Experience in estimating Major Settlement Line Items
Business Process Development/ Refinement


Streamlining BI Implementation - Topics

Components of Business Intelligence

Methodologies for Streamlining a BI Implementation

Relationship to Business Process Management


Components of BI Systems


BI System Integration & Information Layers
Data Warehouse
Data movement from Sources to Target Tables
Usually scheduled in batched ETL processes
Staging Tables or Operational Data Store often an intermediate step
Data Model promotes business user reporting & querying – Dimensional
Data Propagation
Data movement among Systems
Usually in Real Time or near Real Time
Use of Enterprise Application Integration (EAI) tools to publish & subscribe to data
Data Warehouse can be a Subscriber
Data Federation
No data movement
Pre-established Views are often constructed to relate data across systems
More suitable for highly decentralized organizations
A sensible approach may involve elements of each method

BI System Integration Layer

Pre-established Reports
Routine reports designed & developed in advance – often scheduled
Typically ‘locked down’ and controlled by a couple of individuals - often IT
May build intermediate tables, views and/or calculated fields to feed reports
Self Service Reports and Queries
Business users should be able to do reporting & querying without the need for IT
Business users do not always know exactly what they want until they start
using the data in ways not previously possible
Requires a sound foundation (typically dimensional data model), data
dictionary or other metadata, and training
OLAP Cubes
OLAP provides interactive analysis and reporting using multidimensional views of
data
Users can slice, dice, drill-down and roll-up data – more than Excel pivots
Cubes are often pre-built and refreshed right after batch ETL processes so that they
are immediately available to users


BI System Intelligence Layer

The value of the BI System is often not realized until deploying this
Layer

Advanced Visualization, including Dashboards & Scorecards
Enable rapid assessment of Business Metrics or Key Performance
Indicators
Dashboards display information in tables, charts, gauges, and other
indicators
Scorecards display metrics against pre-established thresholds grouped into
red, yellow, green or numerical weightings
Ultimately, should have the capability to drill down into information
feeding the dashboard and scorecard items


BI System Intelligence Layer

Advanced Analytics
Exploratory Data Mining – seeking new insights through examining the
information with statistical analysis or other techniques
Predictive Analytics – formulating rule-based models to predict what
will happen under various scenarios with high degree of confidence

Many BI efforts are abandoned before reaching this Layer
Why? Because implementation of the other layers have been overly
costly with mediocre results


Predictive Analytics; Extending the Value of Your Data Warehousing Investment,
2007 TDWI Best Practices Report, Wayne W. Eckerson

Streamlining BI Implementation
Employ methods that get to the higher value BI layers sooner:

Establish Fundamentals Early

Consider using a Packaged Solution or Starter Model

Consider Agile Development Approach

Consider Hybrid Approach

Considerations for the Interactive Layer

Considerations for the Intelligence Layer


Establish Fundamentals Early for Any Phase
Implement a Data Governance Program Upfront
Assign Data Stewards from the Business responsible for quality and use of data

Establish Monitoring and Methods for Correcting Issues

Consider a Master Data Management Program
Managing your Reference Data, such as customer, generators
Ensuring consistency and dynamic mapping among systems
Some examples of the need:
How many systems or databases house customers (individuals or organizations) and how
many different identifications, abbreviations, or alias’s are used?
How are you tracking who owns whom? Frequently changes - if not careful, you may
have excess credit exposure to entities or entities may be exercising unfair market power.
Various technologies are available
Ensuring consistency among systems regarding Master Data will streamline BI
implementation, but not absolutely needed since the data warehouse will ultimately
solve this issue for users of the data warehouse

Establish Fundamentals Early for Any Phase

Set realistic expectations from both business & IT perspectives
Define precise & measurable objectives
Ensure that the BI system will be scalable to accommodate other corporate
initiatives in the future
More importantly, align the project with overall corporate BI objectives
Minimize scope creep
Well-defined business requirements
Establish Change Control policies upfront – know how changes will be
processed and who needs to approve these
If you don’t have the expertise in-house, then hire professionals
Ensure they have experience in the phase you are embarking on
Ensure those on the team have experience with your business
Be involved – you still need to ‘own’ the program & business must be involved
BI systems are not static – you will need skills to advance & maintain your system
If not prepared to adopt these & other fundamentals, then don’t start!

Use of Packaged BI Solutions & Starter Models

Packaged BI Solutions: 3rd party products fulfilling a slice of your BI
needs

A Packaged BI Solution should:
Provide the business with a clear definition of the end results – what you
will get is visible from the start!
Integrate and merge data across all appropriate systems
Enable easy access to underlying data and the information it provides
Serve as a starting point for other analysis and reports
Build on an existing data warehouse as appropriate



A Packaged BI Solution is typically not:
A front line application, such as an Energy Trade & Risk Management
System (ETRM), or a Customer Information System (CIS) - too many
try to elevate these transactional systems to be an enterprise data mart
for reports & analysis

Are there Packaged BI Solutions available in our industry?
For wholesale power, including financial transactions, we have our
TM
PREMIR Solution
There may be solutions available for the Retail, and
Transmission/Distribution Sectors
Consulting Companies focused in your business may offer at least
Starter Models


Endasol’s Flow Diagram

premir Solutions
TM

Our present focus is the Wholesale Power & Financial Transactions Data Models and Reporting


Advantages:
Requirements/Vision already defined – you see what the end result looks like
Data Model provided and can be built upon
Key Reports may be provided and others can be added
KPI’s & Dashboard may be provided and can be expanded
In any case, they can provide a head start enabling quicker deployment
Potential Pitfalls:
Don’t expect a packaged solution or even starter model to meet all of your
objectives – expect that some customization will be needed
Compatibility with existing data warehouse, data marts must be considered
Compatibility of technology used – database, reporting tools – must be
considered
Capability to expand the solution to other areas of the enterprise must exist
Conclusion: A packaged solution or starter model can significantly expedite
your BI implementation, but do your homework first


Consider Agile Development for BI - Background
Waterfall Methodology – Traditional Approach
From Wikipedia: “The waterfall model is a sequential software
development model (a process for the creation of software) in which
development is seen as flowing steadily downwards (like a waterfall)
through the phases of requirements analysis, design, implementation, testing
(validation), integration, and maintenance.”
Often characterized by lengthy requirements, design and signoff before any
development is allowed to begin.

Agile Methodology – Currently gaining acceptance
From Ralph Kimball: “…In general, it (Agile) refers to a group of
methodologies, including Extreme Programming, SCRUM, Adaptive
Software Development and others, which share a common focus on iterative
development and minimizing risk by delivering new functionality in short
timeframes, often measured in weeks.”
Less structured with emphasis on deliverables, not documents & signoffs


Consider Agile Development for BI
Agile Development is gaining interest - Agile is now included in training
classes and conferences
Scrum is one structured methodology for Agile Development
Product/Project broken into deliverables, called ‘Sprints’ of 2 to 4 weeks.
Sprints established to bring priority, but achievable results within time frame.
Sprints typically locked down with high resistance to changing scope once started.
Product Owner - The Product Owner represents the voice of the client. He/She ensures
that the Scrum Team works with the right things from a business perspective.
ScrumMaster (or Facilitator) - Scrum is facilitated by a ScrumMaster, whose primary
job is to remove impediments to the ability of the team to deliver the sprint goal. The
ScrumMaster is not the leader of the team (as they are self-organizing) but acts as a
buffer between the team and any distracting influences. Empowered teams without
traditional Project Management.
Scrum Team - A small team of 2+ people with cross-functional skills to do the actual
work (designer, developer etc.).
Daily meetings among players – IT & Business.
Development proceeds immediately and documentation often lags.


Consider Agile Development for BI Implementation

Biggest Issue with many DW/BI projects
Changes during the project!
Source system changes
Requirements
Data quality issues – some sources are just not very good.
Change in business sponsor / business owner
Market changes
ISO rule changes
Subscription services
Business Priorities



Advantages of Agile Development
Design at the beginning of each sprint - Learn from each sprint
Phased / iterative approach – Useable tools delivered quicker
Determine next sprint based on priorities.
Working closely with the business users during data cleansing and
development of reports, cubes and dashboards
Increase user satisfaction and ‘ownership’
Decrease ‘do-overs’
Building what the business needs overrides building just to meet
documentation



Disadvantages of Agile Development
Budgeting difficulties due to minimal upfront analysis & design
Hardware needs
Staffing – how long?
When is the project considered finished?
How does the present sprint fit into the solution?
Some rework may be needed
Have a risk of delivering isolated silos of data which meet individual releases, but
don’t integrate well with each other
New to the organization and not well understood by IT and Business.
Users are confused about their role.
Users may not have sufficient time.
Minimize risk by using company standards & following corporate SDLC process
Documentation – Agile methodology minimizes documentation.
What about support and future enhancements?


Hybrid Approach for BI Implementation

Optimum Approach may be a Hybrid between Waterfall & Agile
Take time upfront to develop overall BI plan/ roadmap
Spend some time on how the overall BI solution fits into the enterprise picture
Identify subject areas for potential phases
Develop framework of data model
Set standards
Select hardware / software
Break into focused phases
Phase 1 may be 4 – 6 months due to infrastructure, overall requirements, design
& documentation
Focus mainly on current phase subject area but know how it fits with the other
areas.
Other phases may decrease in length as more and more subject areas are added.
Take each phase all the way through user reports, OLAP cubes, dashboard
& scorecard
Include support with the delivery of each phase


Hybrid Approach for BI Implementation

Establish Project High Level Design Infrastructure
•Subject Area Data •Hardware
•BI System Objectives
•Subject Area Relationships •Software/ Licenses
•Business Requirements
•High Level Data Model •Set Up Environments
•Prioritize Phases
•Set Standards
•Project Plan

Detailed
Requirements
Phase
1
Detailed
Phase
Design
2
Phase
ETL &
3
Test
Repeat Each
Phase
for Phase
each may get 4
User
Phase shorter
Reports

OLAP
Cubes

Dashboard/
Scorecard


Streamlining Interactive Layer

Choosing Reporting and Querying Tools
Business Ease of Use

Deploying a mixture of tools

What about Excel?
MS Excel will continue to be the tool of choice for many business users
Even with well managed data warehouses, end users still resort to Excel
But this may not be bad as long as the source is the data warehouse and
not direct pulls from source systems

How important is OLAP?


Streamlining Interactive Layer

Give Business Early Access to Data
Standard reports should be pre-established using the corporate reporting
tool & ‘Locked Down’
Underlying structure/ data model should be such that business users can
easily understand
Allow them to develop their own reports and analysis - Stress using
corporate reporting tools, but be flexible as long as they are sourcing
from the data mart


Streamlining Intelligence Layer - KPIs
Restrict Key Performance Indicators (KPIs) to a small number (6 –
10) specific to the organization’s goals
Develop different set of KPIs for each business unit, sub-unit
The business unit’s KPIs should relate to corporate KPIs
The business unit’s KPIs should also include what’s important to bonuses
Most effective use is often a mixture of financial & non-financial
KPIs as in the Kaplan and Norton Balanced Scorecard approach
KPIs should follow the SMART test:
Specific, Measurable, Achievable (Actionable), Realistic, Timely
Once implemented, continue to refine KPIs
Revise as corporate and business unit objectives change
Monitor who is accessing and how often –retire KPIs not being used
Ensure individuals can easily access KPIs and understand what they are
viewing

Streamlining Intelligence Layer – Visualizing

Choosing Technology
Numerous products available with nice dashboard & scorecard features

Some are limited to specific technologies, such as using MS Analysis
Services (cubes) to drive dashboard components

Most effective is ability to drill down – start with the picture & drill to
underlying information

Don’t need to go overboard on dashboard/ scorecard technology
If can’t justify expense upfront, then consider publishing output of certain
reports & OLAP such as graphs or charts in a web-portal – this can be
automated
Once base functionality is working, can always buy advanced technology
later


Streamlining Intelligence Layer – Visualizing
Implementing Dashboards & Scorecards

Keep it simple – don’t clutter the viewing screen

Consider allowing viewers to construct their own ‘myBI’ page
Some items may be required viewing such as important metrics
Others may be selected from a list of available dashboard reports,
graphs, charts

Continue to monitor usage and effectiveness & continue to refine


Streamlining Intelligence Layer – Data Mining

Determine Data Mining or Advanced Analytical Methodology
Early
Exploratory – looking for new insights and/ or
Predictive – developing models to predict behavior/ market
opportunities
Different methodologies are available for each focus
Methodology may require very specific data model or structure
Data model used for advanced analytics may conflict with that optimal
for reporting and user-queries
If you know this upfront, you may be able to build a hybrid model that
can serve both purposes, or
You may be able to implement a model structure that can be more easily
transformed into that needed for the advanced analytic application


Align Your BI System with Corporate Objectives
Ultimately, your BI System should be aligned to corporate objectives
and provide monitoring & feedback for continued alignment

Corporate KPIs should trickle down to corresponding KPIs pertinent
to individual Business Units

Business Unit KPIs should link to corporate & adjusted as corporate
objectives change

The above is part of the broader initiative referred to as Business
Performance Management (BPM), Corporate Performance
Management (CPM), or Enterprise Performance Management (EPM)

There are advancement in software tools which support
implementation & monitoring

Streamlining BI Implementation - Summary

Align your BI System with Corporate Objectives

Value of your BI System is often not realized until deployment of
Visualization and Advanced Analytics

Dashboards & Scorecards to track KPIs & other important measures

Drill-down into information driving displays is an important capability

Data Mining for discovering new insights

Predictive Analytics to gain edge in market


Streamlining BI Implementation - Summary

Objective is to get to these value BI Layers as soon as possible:
Use Packaged BI Solutions or Starter Models if available and if
applicable
Use a Hybrid Approach of Agile & traditional Waterfall Development
Do some detailed requirements & design upfront
Break Project into short phases of 1 – 6 months covering selected subject
areas
Each Phase should include all BI Layers - Data Model, User Reports,
Cubes, Dashboard & Scorecard items

Technology – decide what’s important to you from a budget and
capability perspective, be flexible and don’t get stuck in analysis
paralysis.


Endasol Streamlining Bi Implementation, Euci March 2008

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Endasol Streamlining Bi Implementation, Euci March 2008

Similar to Endasol Streamlining Bi Implementation, Euci March 2008 (20)

Recently uploaded

Recently uploaded (20)

Endasol Streamlining Bi Implementation, Euci March 2008