Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (20)
Semelhante a Experiences from a Data Vault Pilot Exploiting the Internet of Things
Semelhante a Experiences from a Data Vault Pilot Exploiting the Internet of Things (20)
Último
Último (20)
Experiences from a Data Vault Pilot Exploiting the Internet of Things
- 1. Business Analytics For All BA4All – Insight Session April 29th 2014 Guy Van der Sande – Vincent Greslebin Unlocking the Value within the Data Vault
- 2. Fifthplay : Architecture Smart Homes Platform Data Warehouse Data Vault Data mart Gebruikers Marketing & SSC Utility Portal ETL Dag - 1 - Controle data kwaliteit - Toepassing business rules - Aggregatie - Filtering Facility Portal
- 3. Fifthplay : Why Data Vault ? • Pattern based design which allows agility to take place • Easy to add new data sources making it future proof. This allows Fifthplay to stay innovative • Large volume of data • Build up history that is not available in the operational system • Possibility of performing analysis on raw data (cfr quality checks) • Development speed (Pilot : 37 working days)
- 4. Data Vault ?
- 5. Data Vault ?
- 6. Data Vault ? The Data Vault is a detail oriented, historical tracking and uniquely linked set of normalized tables that support one or more functional areas of business. It is a hybrid approach encompassing the best of breed between 3rd normal form (3NF) and star schema. The design is flexible, scalable, consistent and adaptable to the needs of the enterprise.
- 7. Standard architecture The centerpiece of the Enterprise Data Warehouse History is build-up Granularity as ‘detailed’ as possible No use of business rules Use of business keys that are horizontal in nature and provide visibility across lines of business A new layer which has the benefits of the RAW Data Vault, but with the business data embedded In the Business Data Vault the data has been altered, cleansed and changed to meet the business rules Downstream of the raw data vault Starting point for Master Data Management Metadata is absolutely vital
- 8. The Data Vault Model exists of 3 basic entity types • Hubs : contains a unique list of business keys • Links : associations across or between business keys • Satellites : holds descriptive data (about the business key) over time Component parts of the Data Vault model
- 9. • Represents a Core Business Concept • Is formed around the Business Key of this concept • Is established the first time a new instance of that business key is introduced • Must be 1:1 with a single instance • Consists of the business key, a sequence id, a load date/time stamp and a record source. Component parts - Hub
- 10. • Represents a natural business relationship between business keys • Is established the first time this new unique association is presented • Can represent an association between several Hubs and sometimes other Links. • maintains a 1:1 relationship with the unique and specific business defined association between that set of keys. • Consists of the sequence ids from the Hubs and Links • Contains sequence id, a load date/time stamp and a • record source. Component parts - Link
- 11. • The Satellite contains the descriptive information (context) for a business key. • A Satellite can only describe one key (Hub or a Link). • The Satellite is the only construct that manages time slice data (data warehouse historical tracking of values over time). Component parts - Satellite
- 12. Fact Dimension 1 Dimension 3 Dimension 2 Dimension 4 Data Vault – Why ?
- 13. Fact Dimension 1 Dimension 3 Dimension 2 Dimension 4 Data Vault – Why ?
- 14. Fact Dimension 1 Dimension 3 Dimension 2 Dimension 4 Fact Dimension 5 Data Vault – Why ?
- 15. Data Vault – Why ? DV DM
- 16. DV DM S S S S H S L H H H Data Vault – Why ?
- 17. DV DM S S S S H S L H H H Dimension Fact Data Vault – Why ?
- 18. Data Vault – How did we do it with Fifthplay ?
- 19. HubServicePartner HubCustomer HubHomeAreaManager HubSmartPlug HubDeviceGroup HubEnergyLogType LinkServicePartnerCustomer LinkCustomerHomeAreaManager LinkHomeAreaManagerSmartPlug LinkCustomerDeviceGroup LinkDeviceGroupSmartPlug LinkDeviceSubGroupSmartPlug LinkSmartPlugApplianceEnergyLogT ype HubCityLinkHomeAreaManagerCity HubCountry LinkCountryCity HubSatServicePartner HubSatCustomer HubSatHomeAreaManager LinkSatHomeAreaManagerCity LinkSatCountryCity HubSatCountry HubSatDeviceGroup HubSatSmartPlug HubAppliance HubSatAppliance LinkSatSmartPlugApplianceEnergyL ogType HubSatHomeAreaManagerAddress SeqServicePartnerPK ServicePartnerID LoadDateTime RecordSource SeqCustomerPK CustomerID LoadDateTime RecordSource SeqHomeAreaManagerPK HomeAreaManagerNumber LoadDateTime RecordSource SeqSmartPlugPK SmartPlugID LoadDateTime RecordSource SeqDeviceGroupPK DeviceGroupID LoadDateTime RecordSource SeqEnergyLogTypePK EnergyLogName LoadDateTime RecordSource SeqServicePartnerCustomerPK SeqCustomer LoadDateTime RecordSource SeqServicePartner SeqCustomerHomeAreaMan ager PK SeqCustomer LoadDateTime RecordSource SeqHomeAreaManager SeqHomeAreaManagerSmar tPlug PK SeqHomeAreaManager LoadDateTime RecordSource SeqSmartPlug SeqCustomerDeviceGroupPK SeqCustomer LoadDateTime RecordSource SeqDeviceGroup LoadDateTime SeqDeviceGroupSmartPlugPK LoadDateTime RecordSource SeqDeviceGroup SeqDeviceSubGroupSmartPl ug PK LoadDateTime RecordSource SeqDeviceGroup SeqSmartPlug SeqSmartPlug SeqSmartPlugApplianceEner gyLogType PK SeqEnergyLogType LoadDateTime RecordSource SeqSmartPlug SeqCityPK CityPostalCode LoadDateTime RecordSource CityName SeqHomeAreaManagerCityPK SeqCity LoadDateTime RecordSource SeqHomeAreaManager SeqCountryPK CountryIsoCode LoadDateTime RecordSource SeqCountryCityPK SeqCity LoadDateTime RecordSource SeqCountry SeqSatServicePartnerPK SeqServicePartner LoadDateTime RecordSource LoadEndDateTime ServicePartnerCode ServiucePartnerEmail ServicePartnerCustomerCon tact SeqSatCustomerPK SeqCustomer LoadDateTime RecordSource LoadEndDateTime CustomerEmail CustomerFirstName CustomerLastName CustomerLanguage SeqSatHomeAreaManagerPK SeqHomeAreaManager LoadDateTime RecordSource LoadEndDateTime HomeAreaManagerMode HomeAreaManagerArchitec ture SeqSatHomeAreaManagerCi ty PK SeqHomeAreaManagerCity LoadDateTime RecordSource LoadEndDateTime HAMCityAddressLine1 HAMCityPhoneNumber HAMCityAddressLine2 SeqSatCountryCityPK SeqCountryCity LoadDateTime RecordSource LoadEndDateTime CountryCityRegion CountryCityState SeqSatCountryPK SeqCountry LoadDateTime RecordSource LoadEndDateTime CountryName SeqSatDeviceGroupPK SeqDeviceGroup LoadDateTime RecordSource LoadEndDateTime DeviceGroupName DeviceGroupDescription SeqSatSmartPlugPK SeqSmartPlug LoadDateTime RecordSource LoadEndDateTime SmartPlugDisplayName SmartPlugManufacturer SmartPlugModel SmartPlugIsGenerator SmartPlugHasChildren SmartPlugHasSchedule SeqAppliancePK ApplianceID LoadDateTime RecordSource SeqSatAppliancePK SeqAppliance LoadDateTime RecordSource LoadEndDateTime ApplianceCategory SeqSatSmartPlugApplianceE nergyLogType PK SeqSmartPlugApplianceEner gyLogType LoadDateTime RecordSource LoadEndDateTime EnergyLogDateTime EnergyLogValue SeqAppliance EnergyLogValueUnit Legend Hub Link Satellite ServicePartnerWebPage SeqSatHomeAreaManagerA ddress PK SeqHomeAreaManager LoadDateTime RecordSource LoadEndDateTime HomeAreaManagerAddress Line1 HomeAreaManagerPostalCo de HomeAreaManagerAddress Line2 HomeAreaManagerCityNam e HomeAreaManagerProvince HomeAreaManagerState HomeAreaManagerCountry Fifthplay Raw Data Vault Architecture
- 20. Fifthplay Raw Data Vault Architecture HubSmartPlug HubEnergyLogType LinkSmartPlugApplianceEnergyLogT ype HubAppliance HubSatAppliance LinkSatSmartPlugApplianceEnergyL ogType SeqSmartPlugPK SmartPlugID LoadDateTime RecordSource SeqEnergyLogTypePK EnergyLogName LoadDateTime RecordSource SeqSmartPlugApplianceEner gyLogType PK SeqEnergyLogType LoadDateTime RecordSource SeqSmartPlug SeqAppliancePK ApplianceID LoadDateTime RecordSource SeqSatAppliancePK SeqAppliance LoadDateTime RecordSource LoadEndDateTime ApplianceCategory SeqSatSmartPlugApplianceE nergyLogType PK SeqSmartPlugApplianceEner gyLogType LoadDateTime RecordSource LoadEndDateTime EnergyLogDateTime EnergyLogValue SeqAppliance EnergyLogValueUnit Legend Hub Link Satellite
- 21. Fifthplay : Data Vault – lessons learned • Don’t stop with data vault; A combination with classic dimensional Kimball-methodology is advised • Be creative; get out of your comfort zone, dare to walk the thine line • While setting up the data vault, operational issues where discovered early in the process • ETL-development goes very quickly because of the typical pattern design of the data vault;
- 22. Data Vault – What’s next ?
- 23. 2013 : Dan Linstedt releases Data Vault 2.0 specs History and what’s next ? Relational modeling (E.F.Codd) Bill Inmon began discussing Data Warehousing • Barry Devlin and Dr Kimball release “Business Data Warehouse” • Bill Inmon popularizes Data Warehousing • Dr Kimball popularizes Star Schema Dan Linstedt begins R&D on Data Vault Modeling Dan Linstedt releases first 5 articles on Data Vault Modeling 2012 : Dan Linstedt announces Data Vault 2.0 1960 1970 1980 1990 2000 2010
- 24. Thank You BI@USGICT.be http://www.linkedin.com/company/usgprofessionalsbe +32 3 231 94 84 www.usgict.be https://www.facebook.com/usgictbe “In the Data Warehousing/BI world, we should store the data as it stands on the source system and interpret it on the way out to the data marts. This is absolutely critical to remember.” Dan Linstedt @BICC_at_USG