Guru4Pro Data Vault Best Practices

Data Vault
Best Practices
BI Practice, September 2010

No. 2© Logica 2010. All rights reserved
• 17:30 – 17:40 Introduction
• 17:40 – 18:10 Data Vault Innovation
• 18:10 – 18:30 Best Practices at Large Telco
• 18:30 – 18:45 Coffee break
• 18:45 – 19:05 Best Practice at Health Information Broker
• 19:05 – 19:45 Panel discussion
Agenda

© Logica 2010. All rights reserved No. 3
Speaking about Data Vault
Dan Linstedt
Founder of the Data Vault
modelling Methodology
Data Vault Innovation
Aytekin Keskin
Senior Data Architect
Liesbeth Hozee
Senior Information
Architect
Sharing Data Vault best practices at
Large Telco
Health Information Broker
Panel Discussion

© Logica 2010. All rights reserved
• A strong relationship with the founder
of Data Vault for over 3 years now.
• Supporting your business with 40+
certified consultants.
• Incorporated as the preferred
Enterprise Data Warehouse modelling
paradigm in the Logica BI Framework.
• Satisfied customers in many countries
and industry sectors
No. 4
Logica and Data Vault

Dan Linstedt
Aytekin Keskin
Liesbeth Hozee
Senior Information
Architect
Large Telco
Health Information broker
Panel Discussion

What is a Data Vault?
Customer
Sat
Sat
Sat
F(x)
Customer
Product
Sat
Sat
Sat
F(x)
Product
Order
Sat
Sat
Sat
F(x)
Order
Elements:
•Hub
•Link
•Satellite
Link
F(x)
Sat
Records a
history of the
interaction
Hub = List of Unique Business Keys
Link = List of Relationships, Associations
Satellites = Descriptive Data
© Dan Linstedt, 2010

EDW: The Case for Business Vault
Source 1
Source 2
Source 3
Business
Rules
HR Mart
Sales
Mart
Finance
Mart
Source 1
Source 2
Source 3
Virtual
Mart
Cube
Mining
Collection
OLD WAY!!
Compliant Way!

Business Vault – My Definition
• Master Data System + History
• Data defined, cleansed, aggregated one time
• Master consolidation…
• One Potential View = MASTER DATA MART
• Might be a Modified Data Vault Model…
• Can be ANY Data Model style that best fits the
business
• Loaded with High Quality Business Data

Business Data Vault – Question…
When you buy a car, do you buy it JUST for it’s engine? Or Do you buy it because
of it’s looks, it’s gas mileage, it’s horsepower, and it’s price? (Perceived Value?)
Raw Data Vault
Business Vault
Business Vault

Business Vault Logical Architecture
Executes
Business Rules
Re-Defines
Uses
Updates
Uses
Applies To…
Business Term Ontology

Future Business Vault Direction
Business Rules
• Dynamic Rules
• Defined by Business
• Owned by Business
• Checked in/out
• Changed Without IT
Business Vault
• RAM Based or SSD Based
• Cube Rebuild
• Appliance Based (columnar / NoSQL based)
• Ontology Managed
• Versioned
• Secure Accessed

Business Vault Future… (One Example) – Dynamic BI
Server Level
Access Control
Virtual Business Vault
Data Write Back
Business Rules Creation
Insert New Versions
Data RetrieveRetrieve Latest Copies
ING Real Estate
TODAY….

• Data Warehouse Components:
• Dynamic Data Warehouse
• Unstructured Data in the Data Vault
• Operational Data Vaults (cloud computing)
• Secured or Classified Data Vault (cloud computing)
• Please Take Note!
• Physical Data Modeling losing ground
• Columnar MPP Appliances Growing
• The Future… The Trends…
• Data Vault / Warehousing As a Service leads to…
• Data As A Service
No. 13
Additional New Developments

Dynamic Data Warehouse
No, it is NOT the first self-adapting data model (was done in 1970’s)
Structure
Common Queries
Web Services
Auto Changes
To Structure
Mining Activities
Produce New
Links!
Common Query Activity
Produces NEW
PIT & BRIDGE tables
*May produce Marts*

Operational Data Vault / Secure Access
Customer 1
Data Vault
SQL View Layer
Mart
1
Mart
2
Mart
3
Customer
Login
Internal
Login
Encrypt Key
Web-Services and Flat File Delivery
Data Exchange/Sharing Through Code Only
Internal Global
Good for Private Cloud Deployment
Encrypt Key
Data Vault
Tracking Pointers
Admin
Login
Employee
Login
Encrypt Key Encrypt Key
Customer 2
Data Vault
SQL View Layer
Mart
1
Mart
2
Mart
3
Customer
Login
Internal
Login
Encrypt Key Encrypt Key

1. Makes Private Clouds Possible
2. Keeps Data Safe with PGP encryption/decryption at disk level
3. Provides Web-Interface SERVICE access (data as a service…)
4. Keeps individual customer information separate in different virtual
machines
5. Keeps employees from “accessing” data sets of customers without
knowing the customer keys.
6. Keeps customers from “hacking” other customer data sets
7. Global Environment tracks security and pointers to universal data,
or “authorized shared data” according to customer
No. 16
Benefits of Secure Layout

• Columnar Databases and NoSQL will change some things
• Unstructured based warehousing (NoSQL)
• Physical Data Models will become less important
• Relational DB’s are good for specific tasks
• MPP computing
• Tomorrows’ vendor will offer “seemless” hardware
• self-organized appliance
• Data Vault thrives as a Logical model solution going forward
• because of it’s focus on the business…
• Operational Data Warehousing is here
• Dynamic is around the corner…
• Cloud Warehousing is good
• but only for secure access
• Note: Normalized & Changed Data will keep costs down
• Data as a Service is the next paradigm shift…
No. 17
Thoughts…

Dan Linstedt
President, CEO, Empowered Holdings, LLC
http://empoweredHoldings.com
26 Prospect St
Saint Albans, VT 05478 USA
Tel: +1 802.524.8566
E-mail: DanL@DanLinstedt.com
Home of the Data Vault Blog (FREE!)
http://danLinstedt.com
Twitter: dlinstedt
Linked-in:
http://www.linkedin.com/in/dlinstedt
How to Reach Me…
Check out my interactive
coaching/e-learning:
http://danLinstedt.com/my-coach
I offer consulting, one-on-one
mentoring, assessments,
performance and tuning, training, and
architecture (design/review)
Register for my blog before October
2nd 2010 and receive advance notice
of my Technical DV Modeling book in
December.

Dan Linstedt
Aytekin Keskin
Liesbeth Hozee
Senior Information
Architect
Large Telco
Panel Discussion

A large scale data warehouse initiative…
Sources:
Logical 44
Physical > 130
– Access Control System
– Billing System
– Call Center
– Campaign Management Tool
– Content Billing Gateway
– CDR Systems
– CRM Systems
– Device Detection System
– Partners and competing operators
– Data of Organisation
– Demographic data on Individuals
– Finance
– Sales
– Site Catalogue
– Network System - Support for
Planning and Network
Implementation
– Interconnection with Others
Networks
– Broadband Management
– Order Handling System
– Porting Handler
– Unsubscription
– Product Management
Targets:
10 Subject Areas
– Customer Base Management
– Financial Management
– Sales Channel Management
– Service delivery Management
– Supply Chain Management
– Billing Management
– Customer Campaign Management
– Customer Operational Management
– Product Lifecycle Management
– CDR operational store
Locations:
14 Countries
8 Time Zones
– Sweden
– Norway
– Russia
– Germany
– Austria
– Baltics
– Croatie
–
– … Plus Some, Minus Some!
No. 20
Over 3000 business requirements
Over 300 TB estimated data volume

• Design and implement a global BI/DW solution
• Supporting country business functions
• Supporting corporate functions
• Improve BI & DWH practice
• Less costly and faster processes
• A enterprise-wide DWH based on common definitions
• Higher information quality and more usage
• Improved technical solution
• Ensure Adaptability & Flexibility
• Ensure High-Performance
• Replace existing DWH platforms
• Migrating historical data
• Migrating existing reporting and analytics
No. 21
Challenging ambitions to comply with…

A comprehensive architecture design is needed…
No. 22
Combining the Logica BI Framework and Data Vault

Requiring sophisticated data modelling method…
One Subject Area Only: x 14!
Let’s look at some data
modelling challenges

Business key uniqueness...
Customer
DB
Extern
Contracts
Customer
DB
Location A, System X
Location B, System Y
Customer
Hub
Customer
Satellites
Large Telco:
-Origin part of
Hub Key
-What should
Origin contain?
Customer
Satellites
Customer Id: #144
Customer Id: #144
No. 24

Record sources…
…
…
…
xxx_<name>_LDTS
xxx_<name>_RSRC
What to set in Record Source?
•Originating system?
•Originating Load Process/Run?
•System Derived Data?
Hub, Satellites, Links
“Data Vault supports corporate compliance
through traceability and auditability”
Dan Linstedt
“Record Sources are system
generated/managed elements;
they are metadata about
WHERE the data in that record
came from.”
Dan Linsted
No. 25

Raw and Business Data Vault…
Raw
Data
Vault
Business
Data
Vault
Source Data Sets
Business Rules
H_Customer_SKEY
CustomerId
CustomerId_Origin
H_Customer_LDTS
H_Customer_RSRC
Customer Hub
CDW
Staging
DM
Staging
Src Sys
Src Sys
Src Sys
Src Sys
DM
DM
DM
Error
DM
100% Data, 100% Time!
No. 26

Unit of work…
H
H H
L
L
H
H
HL
Customer
Product Supplier
Customer
Product
SupplierModelling a Shipment…

Finding right Satellites…
H
S
S
H
S
S S
OR OR
10 Product Usage Satellites!
S_ProdUsageItem_Gen
S_ProdUsageItem_Call
S_ProdUsageItem_Data
S_ProdUsageItem_Content
S_ProdUsageItem_Email
S_ProdUsageItem_Internet
S_ProdUsageItem_SMS
S_ProdUsageItem_MMS
S_ProdUsageItem_Pos
S_ProdUsageItem_TV
Plus
9 BDV Satellites!

Handling weak entities…
Cust
Hub
Billing
Addr
Sat
Mailing
Addr
Sat
Internet
Addr
Sat
Cust
Hub
Addr
Hub
Link
Cust-
Addr
Sat
OR

Data Vault Dates:
•LoadDateTimeStamp (Hubs, Satellite, Links)
•LoadEndDateTimeStamp (Satellite)
vs.
Business Dates:
•EffectiveFrom
•EffectiveThru
•StatusDate
No. 30
Load Dates vs Business Dates...
Product Subscription:
BindingPeriodStartDate
BindingPeriodStartEnd
Market Segment:
ActiveFrom
ActiveUntil

De-activating Relationships...
Cust
Hub
Addr
Hub
Link
Cust-
Addr
Sat
On 28 Sep 10 Rec:
CustXYZ on Old Street 18, 1097 KL, Amsterdam
On 29 Sep 10 Rec:
CustXYZ on New Street 18, 9710 LK, Eindhoven
Customer moved on 29 Sep!

Reference data…
CustId
Origin
…
Name
Tel
Status
Type
Segment
…
What to do
about it?
Keep as
reference
code?
Keep as
reference
table?
Keep as
Hub-Satellite
without link?
Keep as
Hub-Satellite
with link?
Customer Satellite

Data Vault Colors…
• Which Data Vault Colors?
• Which Data Vault Rules?
• Which Data Vault Standards?
• Which Data Vault Conventions?
H
S
H
S
L
No. 33

Data Vault in Perspective…
Data
Vault
Workload
Time
People
Project
Architecture / Design
BI Framework
Business
Data Modelling

• Look for alternative solutions, multiple correct answers
• Think context sensitive, no one-size-fits-all solution
• Follow Data Vault Standards, Principles, Conventions
• Follow the Best Practices
• Consider Data Vault in all Perspectives
• Train your resources well
• Communicate intensively and effectively with All parties
• Adapt your Project Planning accordingly
No. 35
Lessons learned…

Dan Linstedt
Aytekin Keskin
Liesbeth Hozee
Senior Information
Architect
Large Telco
Panel Discussion

BI Cycle, a cyclist’s view on BI
Policy
Business
Goals

• Enabeling the organisation to take over maintenance
• Working within fixed DEADLINES in a learning organisation
• High visibility due to impact on society
• Raise awareness dataminers that changing data by hand
destroys trace ability
• Selecting the right business rules (quality checks, correcting
values, adding calculated values)
No. 38
Organisational Challenges facing the project

Dan Linstedt’s view on business rules

Solution
Storing data
no changes
Apply
data quality
rules
Create
business
views
(Validate
for)
research

• DV Modeling technique improves flexibility and scale ability
to ensure continuity in the future
• Implementing programmed business rules (business view)
secure trace ability and transparancy and help to minimise
data handling
• Results of quality rules in a separate datamart support
dialogue on data quality with data delivering parties
• Methodology with strict building standards ensure resilience
How does Datavault help?

• There is a learning curve in the beginning
• Adapting to the new modeling technique takes time and is
especially hard for people used to working with flat data sets.
• Confront your model with real Test cases before realisation
• Performance:
• Integrate parallel processes using the same tables
• Key has no descriptive data? No need to create a HUB
• Yes there are a lot of tables, a generator for standardised ETL
could be of value
No. 42
Data Vault at Lessons Learned

Transition:
Client: Project management, Change management,
Architecture, Building, Project management, Acceptance testing
Logica: Logical design, Technical design, System testing
Satisfaction:
“Architecture that’s as solid as a house”
“We now benefit from the efforts defining the architecture”
Size of the system : ca 7,000,000,000 records, 339 tables
Size of the datavault : ca 4,000,000,000 records, 97 tables
No. 43
What is the situation now…

Dan Linstedt
Introducing the Data Vault
Methodology concepts
Aytekin Keskin
Liesbeth Hozee
Senior Information
Architect
Large Telco
Health information Broker
Panel Discussion

• Data Vault and Industry Data Models are complementary…
• Data Vault enables Master Data Management…
• Data Vault enabler for BI in the Cloud…
• Data Vault enabler of post merger data migration…
• Data Vault allows phased shutdown of existing DWH…
No. 45
Panel Discussion

Dan Linstedt
Aytekin Keskin
Liesbeth Hozee
Senior Information
Architect
Large Telco
Panel Discussion

• Data Vault and Industry Data Models are complementary…
• Data Vault enables Master Data Management…
• Data Vault enabler for BI in the Cloud…
• Data Vault enabler of post merger data migration..
• Data Vault allows phased shutdown of existing DWH…
No. 48
Some statements to get you started…

Guru4Pro Data Vault Best Practices

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Guru4Pro Data Vault Best Practices

Semelhante a Guru4Pro Data Vault Best Practices (20)

Mais de CGI

Mais de CGI (20)

Último

Último (20)

Guru4Pro Data Vault Best Practices