O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Change data capture

2.792 visualizações

Publicada em

Change Data Capture for Business Intelligence

Publicada em: Tecnologia, Negócios
  • Entre para ver os comentários

Change data capture

  1. 1. The foundation of Business Intelligence Jim Deppen deppenjim@gmail.com 1
  2. 2.  Jim Deppen, Technology Specialist with HP Enterprise Services. Servicing as a Data Architect for the US Army Human Resources Command. 20+ years Experience in Database Technologies. Email: deppenjim@gmail.com Twitter: @jimdeppen 2
  3. 3.  Isn’t the Development Framework the most important thing for successful a BI application? How does this align with Kimball or Inmon? How does this support the Star Schema? Isn’t BI just about creating great Dashboards? 3
  4. 4. 4
  5. 5. 5
  6. 6.  BI is more than turning a traditional report into a dashboard. BI is about building a structure of data that allows for new insights to be discovered. 6
  7. 7.  Change Data Capture is a process to capture insert, update, and delete activity from a data source. Change Data Capture is the most accurate technique to detect a business event. 7
  8. 8. “Business events are individual actions performed by people or organizations during the execution of business processes”.11 Agile Data Warehouse Design by Lawrence Corr 8
  9. 9.  A strategy of capturing business event data as it occurs is the most accurate technique to build consistent business metrics. Important business metrics are many times lost due to poor Operational Application Database Design. Many Operational Applications have a lack of analytic foresight. 9
  10. 10.  Knowing the factors that led to a positive outcome will lead to insights of more positive outcomes in the future. In the end, knowing the what factors led to an outcome, is more important than the outcome itself. 10
  11. 11.  Operational Applications were not designed to capture events. The intermediate steps are not important. An Operation Application may overwrite a number of transitional steps before completing the transaction. 11
  12. 12.  The business process may cross multiple internal applications. Key data is coming from external Internet sources. 12
  13. 13. Question: How many of you have ever had a customer ask: "Hey, can you help us track some Business Events?” 13
  14. 14.  Can you help us duplicate some reports? (Bob quit. Joe accidentally deleted our Access Database. ) Can you help us turn these reports into an Executive Dashboard? We are having trouble manipulating the 10 million rows in our Excel Spreadsheet. Can you us help make it faster? 14
  15. 15.  Understanding the customer’s business process should be your top priority. This is more important than understanding their computer systems and applications. 15
  16. 16.  Conduct meetings with business stakeholders to discover the business process time lines. Brainstorm business events that lead to positive outcomes. Understand the transitions from one event to the next. Understand the whole life cycle. 16
  17. 17. 17-24 Year Old Population (QMA) 17-24 Year Old PopulationTarget Prospect Population (Army Qualified) College Bound College Bound Qualified Lead (Refined) Interest Appointments Made MSL I Appointments Conducted MSL II MSL III Test Passed MSL IV Contract CMSN Access BOLC II IET BOLC III First Unit First Unit First Term First Term 17
  18. 18. Great Reference on the subject: Agile Data Warehouse Design by Lawrence Corr 18
  19. 19.  Business Events are the Facts in the Dimensional Model. 19
  20. 20.  Database Triggers  Database detection of table Insert, Update, and Delete events.  Supported by all major databases.  Can be added to COTS applications with little or no impact. 20
  21. 21.  Data Modification Stamping (also known as Change Tracking). Involves the addition of tracking columns to a table.  Time Stamping  Status Indicator  Version Number 21
  22. 22.  Database transaction log scanning. Decode database transaction logs to detect database Insert, Update and Delete events.  Examples  Microsoft SQL Server Change Data Capture  IBM InfoSphere Change Data Capture  Oracle Goldengate  Can be added to COTS applications with no impact. 22
  23. 23.  Sales prospect becomes a customer by purchasing a product. Sales prospect clicks online ad link. Sales prospect calls TV ad 800 number to inquire about your product. Salesperson sets a follow-up date with prospect. Customer’s purchase depletes the last product. Gross Margin of a product falls below a specified level. 23
  24. 24. 24
  25. 25. 25
  26. 26. 26
  27. 27. 27
  28. 28. 28
  29. 29. 29
  30. 30. Sales Prospectbecomes a customer by purchasing a product 30
  31. 31. Gross Margin of aProduct falls below a specified level. 31
  32. 32. Sales Prospect clickson an online ad link 32
  33. 33. Sales Prospect clickson an online ad linkSales Prospect calls TV 800 Number 33
  34. 34. Sales Prospect clickson an online ad linkSales Prospect calls TV 800 Number Sales Person setsfollow-up date with prospect 34
  35. 35. Customer purchase depletes the lastproduct from stock 35
  36. 36.  Sales prospect becomes customer by purchasing a product.  A salesperson changes the Person Type from “Prospect” to “Customer” in the Sales Application.  This changes the Person_Type_Code from ‘P’ to ‘C’ on the “Person” Table  This update action overwrites the information about a persons prior “Prospect” status. 36
  37. 37.  You no longer know the time it took to convert this person into a customer. The time this conversion takes may be key to increasing sales, especially if this could be correlated to other business events. 37
  38. 38.  Build an infrastructure and a process to capture business events.  Choose a method to detect Business Events on your database. (Database Triggers, Change Columns, or Transaction Log Scanning).  Build a table to store your Business Events.  Use the Business Events as the seeds to populate your BI Fact Table. 38
  39. 39. 39
  40. 40. CREATE TRIGGER NewProspect ON PersonFOR INSERTASINSERT INTO Business_Event (Business_Event_Type_ID, Table, Column, Key_Value, Data_Value) SELECT (select Business_Event_ID from Business_Event_Type where Business_Event_Desc = NewProspect), Person, Person_Type_Code, convert(varchar, Person_ID), convert(varchar, Person_Type_Code) FROM inserted 40
  41. 41. Event Business Event Date Table Column Key Data ValueID Event Stamp Value Type ID1 1 2012-07-05 11:09 Person Person_Type_Code 100 P AM2 1 2012-07-05 12:15 Person Person_Type_Code 102 P PM3 2 2012-07-07 9:23 AM Person Person_Type_Code 102 C4 5 2012-07-07 9:23 AM Product Current_Stock 9519 05 10 2012-07-07 9:45 AM Customer_ Reference_URL 100 http://www.acme.com Contact /julyoffer6 7 2012-07-07 10:12 Order_Ite Calculated 9519 -2.76 AM ms7 12 2012-07-07 11: 11 Customer Followup_Date 960 2012-07-21 AM Contact8 11 2012-07-07 1:29 PM Customer Sales_Person_ID 987 122 Contact 41
  42. 42. Event Business Event Date Table Column Key Data ValueID Event Stamp Value Type ID1 1 2012-07-05 11:09 Person Person_Type_Code 100 P AM2 1 2012-07-05 12:15 Person Person_Type_Code 102 P PM3 2 2012-07-07 9:23 AM Person Person_Type_Code 102 C4 5 2012-07-07 9:23 AM Product Current_Stock 9519 05 10 2012-07-07 9:45 AM Customer_ Reference_URL 100 http://www.acme.com Contact /julyoffer6 7 2012-07-07 10:12 Order_Ite Calculated 9519 -2.76 AM ms7 12 2012-07-07 11: 11 Customer Followup_Date 960 2012-07-21 AM Contact8 11 2012-07-07 1:29 PM Customer Sales_Person_ID 987 122 Contact 42
  43. 43. Event Business Event Date Table Column Key Data ValueID Event Stamp Value Type ID1 1 2012-07-05 11:09 Person Person_Type_Code 100 P AM2 1 2012-07-05 12:15 Person Person_Type_Code 102 P PM3 2 2012-07-07 9:23 AM Person Person_Type_Code 102 C4 5 2012-07-07 9:23 AM Product Current_Stock 9519 05 10 2012-07-07 9:45 AM Customer_ Reference_URL 100 http://www.acme.com Contact /julyoffer6 7 2012-07-07 10:12 Order_Ite Calculated 9519 -2.76 AM ms7 12 2012-07-07 11: 11 Customer Followup_Date 960 2012-07-21 AM Contact8 11 2012-07-07 1:29 PM Customer Sales_Person_ID 987 122 Contact 43
  44. 44. Event Business Event Date Table Column Key Data ValueID Event Stamp Value Type ID1 1 2012-07-05 11:09 Person Person_Type_Code 100 P AM2 1 2012-07-05 12:15 Person Person_Type_Code 102 P PM3 2 2012-07-07 9:23 AM Person Person_Type_Code 102 C4 5 2012-07-07 9:23 AM Product Current_Stock 9519 05 10 2012-07-07 9:45 AM Customer_ Reference_URL 100 http://www.acme.com Contact /julyoffer6 7 2012-07-07 10:12 Order_Ite Calculated 9519 -2.76 AM ms7 12 2012-07-07 11: 11 Customer Followup_Date 960 2012-07-21 AM Contact8 11 2012-07-07 1:29 PM Customer Sales_Person_ID 987 122 Contact 44
  45. 45. Event Business Event Date Table Column Key Data ValueID Event Stamp Value Type ID1 1 2012-07-05 11:09 Person Person_Type_Code 100 P AM2 1 2012-07-05 12:15 Person Person_Type_Code 102 P PM3 2 2012-07-07 9:23 AM Person Person_Type_Code 102 C4 5 2012-07-07 9:23 AM Product Current_Stock 9519 05 10 2012-07-07 9:45 AM Customer_ Reference_URL 100 http://www.acme.com Contact /julyoffer6 7 2012-07-07 10:12 Order_Ite Calculated 9519 -2.76 AM ms7 12 2012-07-07 11: 11 Customer Followup_Date 960 2012-07-21 AM Contact8 11 2012-07-07 1:29 PM Customer Sales_Person_ID 987 122 Contact 45
  46. 46. Event Business Event Date Table Column Key Data ValueID Event Stamp Value Type ID1 1 2012-07-05 11:09 Person Person_Type_Code 100 P AM2 1 2012-07-05 12:15 Person Person_Type_Code 102 P PM3 2 2012-07-07 9:23 AM Person Person_Type_Code 102 C4 5 2012-07-07 9:23 AM Product Current_Stock 9519 05 10 2012-07-07 9:45 AM Customer_ Reference_URL 100 http://www.acme.com Contact /julyoffer6 7 2012-07-07 10:12 Order Calculated 9519 -2.76 AM Items7 12 2012-07-07 11: 11 Customer Follow-up Date 960 2012-07-21 AM Contact8 11 2012-07-07 1:29 PM Customer Sales_Person_ID 987 122 Contact 46
  47. 47.  A central location to drive the population of all Fact and Slowly Changing Dimension tables. Elimination for the need to flush and reload Fact and Dimension tables. Potential to process data in real-time. Ability to capture business event from status columns. Capability to retain history of business events over time. 47
  48. 48.  BI’s most important function is the analysis of Business Events. Learn the business process first. Help customer to look beyond traditional reporting. Data about intermediate steps in a Business Process are usually lost. CDC is the most accurate way to capture this business event data. 48
  49. 49. Change Data CaptureThe foundation of Business Intelligence deppenjim@gmail.com @jimdeppen 49
  50. 50. The foundation of Business Intelligence Jim Deppen deppenjim@gmail.com 50

×