Big data analytics and visualization techniques. Explanation of CRISP DM

1. 1
Presented by:
Daniel & Vini

2. 2
“We are drowning in data, but starving for knowledge!”
-John Naisbett

3. 3
TODAY‘S SHOW

4. 4
You will learn a few data analysis topics
Posing a question
Wrangling your data into a format you can use and fixing
any problems with it
Exploring the data, finding patterns in it, and building
your intuition about it
Drawing conclusions and/or making predictions
Communicating your findings

5. 5
What is Big Data Analytics?
Data analytics is an emerging technique that dives into a
data set without prior set of hypotheses
Accumulation of raw data captured from various sources
(i.e. discussion boards, emails, exam logs, chat logs in e-
learning systems) can be used to identify fruitful
patterns and relationships
Examining large amount of data

6. 6
Data Drives
Performance
Big Data Analytics Drives
result
Increase Revenue
Decrese Costs
Increse Productivity
Why Big Data Analytics??

7. 7
Why Big Data Analytics??

8. 8
Applications of Data analytics
Understanding and targetting Customers
Understanding and optimizing Business Processes
Improving Healthcare and Public Health
Optimizing Machine and Device Performance
Financial Trading
Improving and Optimizing Cities and Countries
Can you think of anything more??
How??

9. 9
Reference Models
CRISP-DM
Agile methodology: ASD-DM

10. 10

11. 11
Cross Industry Standard Process for Data Mining
(CRISP-DM)
The CRISP-DM reference model

12. 12
Cross Industry Standard Process for Data Mining
(CRISP-DM)
The CRISP-DM reference model

13. 13
Cross Industry Standard Process for Data Mining
(CRISP-DM)
The CRISP-DM reference model

14. 14
Cross Industry Standard Process for Data Mining
(CRISP-DM)
The CRISP-DM reference model

15. 15
Cross Industry Standard Process for Data Mining
(CRISP-DM)
The CRISP-DM reference model

16. 16
Cross Industry Standard Process for Data Mining
(CRISP-DM)
The CRISP-DM reference model

17. 17
The BIG Four
Classification Cluster Analysis
Association Rules Prediction

18. 18
Data Classification
Some Examples:
Separating Customer based on gender
Data sorting based on content type/file type,size etc
Classifying data into restricted, pubic or private data
types
"Among all the customers of Zalando, which are likely to respond to a new
offer?"
Will respond Will not respond

19. 19
Decision trees (DT)
Build classification or regression models in the form of Tree
structure
Classification Methods

20. 20
Classification Methods
Decision Trees to Decision Rules

21. 21
Classification Methods
Support Vector Machines(SVM)
Each data item is a point in n-dimensional space(n number
of features)
Find the hyperplane that differentiate the two classes

22. 22
Classification Methods
Which do you think are the separating
Hyperplanes?

23. 23
Classification Methods
Select the hyperplane which
segragates two classes better
Ans: B
Maximising the distance between
nearest data point (Margin)
Ans: C
Select hyper-plane which classifies
accurately prior to maximising margin
Ans: A
Ignores outliers
Introduce: Z=x²+y²
In original input space
hyperplane looks like a circle

24. 24
Classification Methods
Bayesian Networks
Dotted lines: Potential Links
Blue box: Additional nodes and links between input
and output
 Based on probability theory.
 Can mix expert opinion and data to build
models
 Backwards reasoning - in addition to
predicting outputs given inputs, we can
use output values to infer inputs.
 Support for missing data during learning
and classification

25. 25
Classification Methods
Bayesian Network Example

26. 26
Association Rules
Discovering interesting realtions between variables in
large DB
Example Problems
 Which products are frequently bought together by
customers? (Basket Analysis)
● DataTable = Receipts x Products
● Results could be used to change the placements of products in the market
 Which courses tend to be attended together?
● DataTable = Students x Courses
● Results could be used to avoid scheduling conflicts....

27. 27
Association Rules
Examples
 Bread, Cheese → Red Wine.
Customers that buy bread and cheese, also tend to buy red
wine
 Machine Learning → Web Mining, ML Praktikum
Students that take 'Machine Learning' also take 'Web Mining'
and the 'Machine Learning Praktikum'

28. 28
Apriori Principle illustration
If {c,d,e} is frequent then all
subssets of this itemset are
frequent
Support Based pruning illustration
If {a,b} is infrequent then all
supersets of this itemset are
infrequent
Association Rules

29. 29
Association Rules: Apriori example

30. 30
Cluster analysis
Task of grouping a set of objects in such a way that
objects in the same group (called a cluster) are more
similar (in some sense or another) to each other than to
those in other groups (clusters).
Examples
Biology: What is the taxonomy of the species?
Education: What are student groups that need special
attention?
Business: What are the customer segments?

31. 31
Clustering workflow

32. 32
Cluster analysis
Methodologies
K-Means Clustering
Hierarchical Clustering
And many more!!

33. 33
K-means clustering
k-means clustering aims to partition n observations into k
clusters in which each observation belongs to the cluster
with the nearest mean, serving as a prototype of the
cluster
Unsupervised learning algorithm
Define k centroids, one for each cluster
Take each point in the data set and associate it to the
nearest centroid
Recalculate the centroids
Repeat until the centroid doesnt move

34. 34
Hierarchical clustering
Groups data over a variety of scales by creating a cluster
tree or dendrogram.
Find the similarity or dissimilarity between every pair of
objects in the data set.
Group the objects into a binary, hierarchical cluster
tree.
Determine where to cut the hierarchical tree into
clusters

35. 35
Hierarchical clustering
Dissimilarity
measures
Grouped (B,F), less
dissimilarity
Grouped (A,E), less
dissimilarity

36. 36
Hierarchical clustering

37. 37
Hierarchical clustering
Cutting the Tree
50% similarity=50% dissimilarity
Take cluster samples below 0.5 dissimilarity
(B,F),(A,E,C,G),(D)
Creating 3 cluster labelled 1,2,3

38. 38
Clustering workflow
Which algorithm fits my data?
Which parameters fit my data?
How good is the obtained result?
How to improve result quality?

39. 39
Predictive Analytics
Make predictions about unknown future events based on
past happenings
Why now?
 Growing volumes and types of data, and more interest in
using data to produce valuable insights.
 Faster, cheaper computers.
 Easier-to-use software.
 Tougher economic conditions and a need for competitive
differentiation.

40. 40
Predictive Analytics
improve pattern detection and prevent criminal
behavior.
determine customer responses or purchases, as well as
promote cross-sell opportunities
forecast inventory and manage resources, to set ticket
prices.
Credit scores are used to assess a buyer’s likelihood of
default for purchases

41. 41
Data Visualization
Data visualization is the process of converting raw data
into easily understood pictures of information that
enable fast and effective decisions.
Visualization plays the key role in the efficient
communication of information (especially with large
amounts of information).
Visualization is used as a "check" to verify / falsify
results of automatic data analysis.

42. 42
Why Data Visualization?
Identify areas that need attention or improvement.
Clarify which factors influence customer behavior.
Help you understand which products to place where.
Predict sales volumes.
Data visualization is a quick, easy way to convey concepts in a
universal manner

43. 43
Where does Visualization fit in CRISP-DM
Visual
Reportting

44. 44
Visual Analytics Loop
Visual Analytics will foster the constructive evaluation, correction and rapid
improvement of our processes and models and - ultimately - the improvement of our
knowledge and our decisions

45. 45
Visual Analytics : Humane and Machine

46. 46
Visual Analytics vs Information Visualization
Visual analytics is more than just visualization. It can rather be seen as an
integral approach to decision-making, combining visualization, human
factors and data analysis.

47. 47
Various Data Visualization Techniques

48. 48