Slides used during the presentation and demonstration 'QlikView & Big Data' at the Business Discovery World Tour on 9 October 2013 by Mischa van Werkhoven and Michael Robertshaw.
Big Data. We've all heard about it. We all think we should do something with it. But do we know exactly what it is and how to create value from it? How reasonable are our expectations? This session focuses on the myths of Big Data, technologies involved as well as how QlikView can be used to add relevance and context to Big Data for the end user.
3. Key Takeaways
• The Most Common Purpose of Big Data Is to Produce Small Data
• Big Data is About Relevance and Context
• Know What You Want to Achieve
4. Agenda
• What is Big Data?
• Myths about Big Data
• Gartner
– Hype Cycle
– Top Challenges
• Who’s doing it?
• What technologies are they using?
• Hadoop Components
• The Bloor Group
– The Intelligent Thing
– Cost vs Benefit
• How to do it using QlikView
• Demonstration
“Big Data Analytics refers to analytics on data that is not able to be
performed on a standard relational data warehouse in a timeframe
and cost that is acceptable for its business purpose”
7. Paper Print Computer Internet
Big Data happens in every part of History
• Medium to write
ideas and
information
• Not enough writers
to disseminate
• Technology to
distribute
information
• No place to store
• Place to store
• Can’t keep up with
computing
requirements
• Distributed
computing globally
• Too many Emails
to read
We always create more than we can consume!
19. How Reasonable are your Expectations?
Notebook
HDD
Server
HDD
SSD
RAM
Hadoop
Tape
Performance
Cost
20. The Bloor Group
Hard Disk
Drives (HDD)
Solid State
Storage (SSD)
Random Access
Memory (RAM)
Speed (t/TB) 3300s 1000-300s 1s
Price $/TB $ 50 $ 500 $ 4 500
• Keep data in memory when the value obtained from processing it is high
• Leave data on disk when it is inactive or the value from processing it is low
22. The Value in Big Data Comes from Context and Relevance
Machine data, web
data, cloud data
Big Data
cluster
Operational
systems
Data
warehouse
Google
BigQuery
23. The Value in Big Data Comes from Context and Relevance
Business Discovery is about enabling the users to find their own path
through a pre-defined Dataset.
Structure needs to be defined by a QlikView document developer,
though content could be refreshed periodically (conventionally)
or impacted and triggered by the user (on demand).
24. The Value in Big Data Comes from Context and RelevanceMoreHistory
More Categories
They’re both the same number of bricks!
The same volume of data, same schema.
You choose what is relevant to your analysis.
25. Using QlikView with Big Data
1. Conventional Reloads with Document Chaining
2. Direct Discovery – Hybrid Approach
3. Reload on Demand
26. 1. Conventional Reloads
• Reload available data into
multiple QVW documents
segmented by Region and
current Financial Year
reloaded Monthly
• Entry Document contains
Details for All Regions for
Current Period only.
Reloaded Daily
• Use Document Chaining to
navigate to/amongst Region-
Year documents
• A lot of Publisher capacity
and Data Replication
27. 2. Direct Discovery
• Reload available data into
multiple QVW documents
segmented by Region and
current Financial Year
reloaded Monthly
• Entry Document provides
Trends for All Regions for
Any Period.
Dimensions reloaded Daily.
QvS generates aggregate
SQL to draw Charts
• Use Document Chaining to
navigate to/amongst
Region-Year documents
containing Detail
• Performance dependent
upon Database
28. 3. On Demand Reloads
• Entry Document provides
some Aggregate KPIs for All
Regions, but mostly just
Dimension selection.
• When User selects sufficient
criteria, a Link is enabled to
pass criteria to custom
ASPX page.
• ASPX page causes User
document to be Reloaded
with chosen criteria
• User Document contains
relevant subset entirely in
Memory
• Reload requires a little
patience but then
performance is great.
33. Key Takeaways
1. The Most Common Purpose of Big Data Is to Produce Small Data
2. Big Data is About Relevance and Context
3. Know What You Want to Achieve