Dwh lecture 13-process dm
•Transferir como PPT, PDF•
0 gostou•162 visualizações
DATA WARE house
Recomendados
Mais conteúdo relacionado
Semelhante a Dwh lecture 13-process dm
Semelhante a Dwh lecture 13-process dm (20)
Mais de Sulman Ahmed
Mais de Sulman Ahmed (20)
Último
Último (20)
Dwh lecture 13-process dm
- 1. Data Warehousing Lecture # 13 Process of Dimensional Modeling 1
- 2. The Process of Dimensional Modeling Four Step Method from ER to DM 1. Choose the Business Process 2. Choose the Grain 3. Choose the Facts 4. Choose the Dimensions 2
- 3. Step-1: Choose the Business Process • A business process is a major operational process in an organization. • Typically supported by a legacy system (database) or an OLTP. – Examples: Orders, Invoices, Inventory etc. • Business Processes are often termed as Data Marts and that is why many people criticize DM as being data mart oriented. 3
- 4. Step-1: Separating the Process 4 Star-1 Star-2 Snow-flake
- 5. Step-2: Choosing the Grain • Grain is the fundamental, atomic level of data to be represented. • Grain is also termed as the unit of analyses. • Example grain statements • Typical grains – Individual Transactions – Daily aggregates (snapshots) – Monthly aggregates • Relationship between grain and expressiveness. • Grain vs. hardware trade-off. 5
- 6. The case FOR data aggregation • Works well for repetitive queries. • Follows the known thought process. • Justifiable if used for max number of queries. • Provides a “big picture” or macroscopic view. • Application dependent, usually inflexible to business changes (remember lack of absoluteness of conventions). 6
- 7. The case AGAINST data aggregation • Aggregation is irreversible. – Can create monthly sales data from weekly sales data, but the reverse is not possible. • Aggregation limits the questions that can be answered. – What, when, why, where, what-else, what-next 7
- 8. The case AGAINST data aggregation • Aggregation can hide crucial facts. –The average of 100 & 100 is same as 150 & 50 8
- 9. Aggregation hides crucial facts Example Week-1 Week-2 Week-3 Week-4 Average Zone-1 100 100 100 100 100 Zone-2 50 100 150 100 100 Zone-3 50 100 100 150 100 Zone-4 200 100 50 50 100 Average 100 100 100 100 9 Just looking at the averages i.e. aggregate
- 10. Aggregation hides crucial facts chart 0 50 100 150 200 250 Week-1 Week-2 Week-3 Week-4 Z1 Z2 Z3 Z4 10 Z1: Sale is constant (need to work on it) Z2: Sale went up, then fell (need of concern) Z3: Sale is on the rise, why? Z4: Sale dropped sharply, need to look deeply. W2: Static sale
- 11. Step 3: Choose Facts statement 11 “We need monthly sales volume and Rs. by week, product and Zone” Facts Dimensions
- 12. Step 3: Choose Facts • Choose the facts that will populate each fact table record. –Remember that best Facts are Numeric, Continuously Valued and Additive. –Example: Quantity Sold, Amount etc. 12
- 13. Step 4: Choose Dimensions • Choose the dimensions that apply to each fact in the fact table. –Typical dimensions: time, product, geography etc. –Identify the descriptive attributes that explain each dimension. –Determine hierarchies within each dimension. 13
- 14. Step-4: How to Identify a Dimension? • The single valued attributes during recording of a transaction are dimensions. 14 Calendar_Date Time_of_Day Account _No ATM_Location Transaction_Type Transaction_RsTransaction_Rs Fact Table Dim Time_of_day:Time_of_day: Morning, Mid Morning, Lunch Break etc. Transaction_Type:Transaction_Type: Withdrawal, Deposit, Check balance etc.
- 15. Step-4: Can Dimensions be Multi-valued? • Are dimensions ALWYS single? – Not really – What are the problems? And how to handle them 15 Calendar_Date (of inspection) Reg_No Technician Workshop Maintenance_Operation How many maintenance operations are possible?How many maintenance operations are possible? FewFew Maybe more for old cars.Maybe more for old cars.
- 16. Step-4: Dimensions & Grain • Several grains are possible as per business requirement. – For some aggregations certain descriptions do not remain atomic. – Example: Time_of_Day may change several times during daily aggregate, but not during a transaction • Choose the dimensions that are applicable within the selected grain. 16
Notas do Editor
- The four steps are: but before the difference b/w er and dm should be clear. Er covers the whole business, which has number of processes like accounting, inventory, invoices or the process which do not work simultaneously. But as er by defination covers the whole business so it covers the all processes. The complexity of er process understanding is difficult. A process could be in case of exceptional conditions what should be done. But in dm we isolates the process, and this is one reason to know the dm. it identifies the processes, and then dm is developed for that process. The second point is to choose the grain of the process. Grain is high then detail will be high as well. In fact table more records or details will come in this case. The grain level b/w sys and fact table could be different (detail later) Now the third step is to check the facts or isolate them. Facts are numeric value but it shouldn’t be the key value. So fact is a non key value. So by choosing facts you made fact tables. Now the dimension tables are great in number and small in size, and you choose them. They are joined with fact table, you have to identify them.
- It is not an automated process could be done by a wizard, which needs domain knowledge and human experience which needs efforts. Business process is the major process runing in a business. These are the core business processes which are controlled by oltp systems.
- This is an animation. In this snowflake schema and it has a great number of processes which are represented as star which needs the dimensional modeling. The system which is developed against these stars are known as data marts.
- Grain is atomic level means which couldn’t be breaked. Grain corresponds to a transaction, which goes in the table. The detail level of data which is captured in business, that level and the level of data which goes in fact table may be different. E.g. in a store a trasaction could be of single item, or multiple items. It could be on the base of sales per hour, week etc. When we talk about grain with transaction, it has a factor of exprssivness. The more detail, means you know more about your business. Less data less expressive. So if the aggregation is high you will not need to invest on hardware for the performance. So the focus should be on grain, i.e if you want to run the queries in parallel you need more processing power. But high aggregation means when you need answer you will not need to run the query in parallel you just need to display the precomputed aggregate against the query.
- Grain has 2 costs. 1 grain making needs effort. E.g. you need to down the dwh, so for grain making, you need the time other than the office hours i.e 2 am. So you need support people so you need dollars. After that when you will make the grains, you need to store them. So benifitial when you need to use again and again. So the queries should be which may use the grain again and again. Aggregates are business process dependent. When you make aggregates you keep the business before you. So they are more benifitial.
- If I take the weekly aggregates I can make monthly aggregates, but if aggregates are monthly can you come back to weekly aggregates. E.g we can guess 3+5 but can not tell if 8 is given and you want to know 2 numbers. As already discuss several “w” are involved in aggregation, when you aggregate you remove one w. So suppose you aggregate on geography you remove where “w”. So now you cannot tell, this sale is either of zone1,2 etc. suppose you summarized on geography you cannot tell what actually happened in zones as geography has zones as components. The reason is aggregation is irreversible and from aggregation we mean summary. so if you aggregate on when i.e time you cannot tell when that event occurred. Aggregation is one-way i.e. you can create aggregates, but can not dissolve aggregates to get the original data from which the aggregates were created. For example 3+2+1 = 6 at the same time 2+4 also equals 6, so does 5+1 and if we consider reals, then infinetly many ways of adding numbers to get the same result. If you think about the “5 W’s of journalism”, these are the “6 W’s of data analysis”. Again it highlights the types of questions that end users want to ask and can not be answered by summary data. By definition, a summarization will consider at least one of these points irrelevant. For example, a summary across the company takes out the dimension of “WHERE” and a summary by quarters takes out the element of “WHEN”. The point to be noted is that although summary data has a purpose, yet one can take any summary and ask a question that the system can not answer.
- This is not the case of ireversibility, it means same aggregate can come by adding different numbers. But a lot of pit falls can come.
- In this example a portion of data is hide, here it represents that the average of every zone in every week. Now from aggregation, we can see that average sale is equal in all zones. Suppose a company has launched a sale compaign, and they want to know the impact of the compaign in all zones. Now aggregation has hide the information, and if we look at its graph in next sl
- Questioon is what are facts and dimensions. If in this statement when we say we need sales volume and Rs these are facts. And dimension mantioned. So it is your job to identify what are the facts and what are the dimensions.
- Every number is not fact i.e. pk. The rule is you suld be able to add them, e.g. discount is not fact, because you give discount on the base of percentage (discuss later).
- Dimensions are descriptions, which have hirarchy or you run where clause on it. You have to identify the dimensions you cannot judge them from their loud voices.
- Its mean when a transaction is recorded, during transaction the attribute which do not change its value is dimension. See the above table. Here, you went to an atm and you withdraw some money, now transaction_RS attribute will not have that money which was previously their, but there are some attributes like location, account_no they will not change. Now question is time changes but here we are concerned with time_of_day like is it morning, evening etc. so the time will remain constant because transaction will not take so long. So except transaction_rupees every thing is dimension.
- Are dimensions Always single valued: a car selling company has a workshop as well. So when car goes there they record different type of data against it. Oil filter change, plug change so maintainance has multivalues. So multi values are possible in dimension table.
- Several grains are possible but select the most appropriate grain low grains will have more dimensions ie you aggregate on day, then time of day will have several values like morning, evening etc. but if the grain is weekly then it wont have such dimensions. So low grains are more expressive.