Trends for Modernizing Analytics and Data Warehousing in 2019
1. Arcadia Data. Proprietary and Confidential
Trends for Modernizing Analytics and Data
Warehousing in 2019
2. Arcadia Data. Proprietary and Confidential
Featured Speakers
Joydeep Das
Sr. Director, Product
Howard Dresner
Founder and Chief Research
Officer
Priyank Patel
Co-founder and
Chief Product Officer
3. Copyright 2018 Dresner Advisory Services, LLC
Trends in Business
Intelligence
Howard Dresner
Founder and Chief Research Officer
Dresner Advisory Services
December 5, 2018
10. Copyright 2018 Dresner Advisory Services, LLC
1
1.5
2
2.5
3
3.5
4
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2012 2013 2014 2015 2016 2017 2018
Cloud BI Importance 2012-2018
Critical Very important Important Somewhat important Not important Mean
11. Copyright 2018 Dresner Advisory Services, LLC
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2012 2013 2014 2015 2016 2017 2018
Plans for Public Cloud BI 2012-2018
Discontinued No plans Plan to use next year Plan to use this year Using today
13. Copyright 2018 Dresner Advisory Services, LLC
2
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
3
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2014 2015 2016 2017 2018
Importance of Big Data Analytics 2014-2018
Critical Very important Important Somewhat important Not important
14. Copyright 2018 Dresner Advisory Services, LLC
17%
41%
53%
59%
47% 46%
36%
33%
36%
14%
11%
9%
0%
10%
20%
30%
40%
50%
60%
70%
2015 2016 2017 2018
Big Data Analytics Adoption
Yes. We use big data today We may use big data in the future No. We have no plans to use big data at all
15. Copyright 2018 Dresner Advisory Services, LLC
11% 12%
28%
32%
61%
57%
0%
10%
20%
30%
40%
50%
60%
70%
2017 2018
Big Data Analytics Future Adoption
Will adopt this year Will adopt next year Will adopt beyond next year
16. Copyright 2018 Dresner Advisory Services, LLC
1
1.5
2
2.5
3
3.5
4
Data warehouse optimization
Customer/social analysis
Predictive maintenance
Clickstream analytics
Fraud detection
Internet of Things (IoT)
Big Data Use Cases
2015 2016 2017 2018
17. Copyright 2018 Dresner Advisory Services, LLC
-6% -4% -2% 0% 2% 4% 6%
Predictive maintenance
Data warehouse optimization
Clickstream analytics
Internet of Things (IoT)
Customer/social analysis
Fraud detection
Big Data Analytics Use Cases 2017-2018
18. Copyright 2018 Dresner Advisory Services, LLC
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Presto
Couchbase
MemSQL
Kudu
Apache Drill
Neo4j
SAP Hana
Snowflake
Amazon DynamoDB
Cassandra
HBase
Google BigQuery
Azure Data Lake Store (ADLS)
Impala
MongoDB
Amazon Redshift
HDFS
Hive/HiveQL
Spark SQL
Amazon S3
Big Data - Data Access
Critical Very Important Important Somewhat Important Not Important
19. Copyright 2018 Dresner Advisory Services, LLC
-20% -15% -10% -5% 0% 5% 10% 15%
HBase
Hive/HiveQL
HDFS
Presto
Impala
Spark SQL
MongoDB
CouchDB
Google BigQuery
Cassandra
Amazon Redshift
Kudu
Amazon DynamoDB
Amazon S3
Big Data - Data Access 2017-2018
20. Copyright 2018 Dresner Advisory Services, LLC
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Qubole
IBM BigInsights
Microsoft HD Insights
Google Dataproc
MAP/R
Amazon EMR
Hortonworks
Cloudera
Big Data Distributions
Critical Very Important Important Somewhat Important Not Important
21. Copyright 2018 Dresner Advisory Services, LLC
Conclusions
• Natural language analytics and streaming data analysis
represent emerging technologies. Organizations should
explore and identify use cases to see where they can add
value.
• Cloud computing has passed the “tipping point” where
organizations should now feel comfortable moving critical data
and applications to the public cloud.
• Big data technologies and architectures have become
mainstream as an alternative to traditional database
approaches for business intelligence.
22. Copyright 2018 Dresner Advisory Services, LLC
Trends in Business
Intelligence
Howard Dresner
Founder and Chief Research Officer
Dresner Advisory Services
December 5, 2018
23. Arcadia Data. Proprietary and Confidential
Modernizing Analytics on Big, Fast and Complex
Data
Priyank Patel, Co-founder and Chief Product Officer
December 5th, 2018
24. Arcadia Data. Proprietary and Confidential
Arcadia Data Mission: To Connect Business Users to Big Data
Founding team from Teradata Aster, HPE
3PAR, IBM DB2
Architected to solve challenges around
big data analytics
Investors
Strong Performer: Hadoop Native BI Wave Report.
”Put your BI where your data is”
Recent Awards
Customers
Leader and Technology Innovation Award Winner for 2017 Big
Data Analytics Market Study
25. Arcadia Data. Proprietary and Confidential
Arcadia Delivers Analytics Scale and Speed Natively from Big Data
25
Ad tech
Enterprise-wide BI standard
3000 node deployment for
reduced risk across credit
cards and IT operations
90% reduction in spam sent
before detection using real-
time analytics
Developed a new SaaS self-
service BI platform to give
their customers better
marketing attribution
Gives global brand
managers digital
campaign intelligence
across 100+ brands
INNOVATION
REDUCE RISK
Government
Improve patient outcomes
on 10+ million members by
predicting and controlling re-
admission risk.
Real-time analytics service
for CSP operations to 40,000
premium customers
26. Arcadia Data. Proprietary and Confidential
How can we give business users self-serve BI on big/complex data?
Where does real-time and streaming analytics give us a competitive advantage?
Can we make analytics as easy as Google search?
26
3 Trends in Modernizing Analytics on Big, Fast and Complex Data
27. Arcadia Data. Proprietary and Confidential
How can we give business users self-serve BI
on big/complex data?
Data Warehouse Optimization
28. Arcadia Data. Proprietary and Confidential
Companies are Now Choosing Two BI Standards for Their Enterprise
28
RDBMS
BI Standard for
Relational Stores
BI Standard for
Modern Data Platforms
29. Arcadia Data. Proprietary and Confidential
RDBMS BI Architecture
29
BI Server
RDBMS
Analytic Process
Optimize Physical
Semantic Layer
Secure Data
Load Data
Big Data Requirements
Native Connection
Semi-Structured
Parallel
Real-time
30. Arcadia Data. Proprietary and Confidential
Native BI Architecture – The Arcadia Data Way
30
BI Server
Analytic Process
Optimize Physical
Semantic Layer
Secure Data
Load Data
Big Data Requirements
Native Connection
Semi-Structured
Parallel
Real-time
Data Warehouse
(RDBMS)
Arcadia Data was built
from inception to
run natively within data lakes
31. Arcadia Data. Proprietary and Confidential
Visual Insights To Purchase Paths
“Arcadia Enterprise is the first product we found
that provides truly on-cluster Hadoop BI…
Its execution model and user self-service approach deliver
performance at Hadoop scale and let us develop our analytics
quickly.”
Digital Marketing Use Cases
• Increase campaign effectiveness
• Measure brand recognition
• Understand and respond to customer preferences
• Incorporate insights into future products
Challenges
• Fragmented silos of applications with product and brand information
• Lack of granular insight into customer response to marketing campaigns
• Manual process to create reports requires data extraction & movement
Results
• 100s of brand managers have direct access to self-service visual analytics across all data
on the effect of digital campaigns on product performance
• Increased visibility into campaign effectiveness and brand recognition across geographies
• Marketers and product managers can leverage insights to drive campaign creation and
execution as well as product roadmap
32. Arcadia Data. Proprietary and Confidential
Data Drives Market DisruptionRetail Store Geographic Analysis
YoY Growth
metrics plotted by
county for the
chose sub-brand
Trellising allows for
quick trend analysis
across multiple stores.
Here showing store
sales vs trade area
sales to correlate
potential shifts in buying
pattern
Choose a specific
state to drill down to
county level
33. Arcadia Data. Proprietary and Confidential
Faster Supply Chain Optimization
“Supply chain optimization with visual analytics
has been transformative”
Use Cases
• Integrate financial and physical flow data
• Self-service visual analytics
Challenges
• One-off consulting project typically costs
hundreds of thousands of dollars and lasts 6-8 months.
Results
• Business analysts have instant access to all data –
no data movement necessary
• Visualizations make it easy to highlight anomalies and
potential issues
• Analysts, engineers, and data scientists all can
create stories directly on the data
34. Arcadia Data. Proprietary and Confidential
Where does real-time and streaming
analytics give us a competitive
advantage?
Log Warehouse
35. Arcadia Data. Proprietary and Confidential35
Streaming Data For Predictive Maintenance
QUOTE TBA
Use Cases
• Mining equipment performance and maintenance
• Native business intelligence on a Hadoop data lake with complex data types and KSQL for
streaming data
Challenges
• Large investment in Hadoop, but existing business intelligence solution did not support new
data types
• Growing data volumes. Expected growth of 30TB per month
• A single machine can produce 30,000 – 50,000 unique time-stamped records per minute
Results
• New visualizations for waveform and spectrum plotting alongside traditional derived metric
visualizations
• Real-time analysis around production goals of mining operations
• Support for complex data and streaming data
• Analytics across all data plus functionality beyond traditional BI
• Fast adoption for users of traditional BI
“Our former environment limited our ability to scale, grown,
and innovate.
We now provide customers with better
recommendations on machine utilization and deliver
services faster.”
— Anthony Reid
Senior Manager of Analytics
36. Arcadia Data. Proprietary and Confidential36
Reducing SMS Spam With AI and BI
“Our customers shouldn’t expect SMS spam to be
part of the cell phone experience.
With an AI spam detection solution built on
Arcadia Enterprise and Cloudera, we are greatly
reducing the negative effects of spam for
customers.”
Use Cases
• Real-time log streaming
• Operational dashboards
• AI algorithm – spammer detection
Challenges
• Spammers easily adapted to traditional rule-based solutions
• Bell needed a real-time, scalable, cost-efficient solution
Results
• 250 percent improvement in spammer detection with AI compared to
traditional rule-based solution
• 90 percent decrease in spam sent before detection
• Spam detection 24 hours faster than rule-based detection
• 90 percent reduction in manual work
Image:Apple,2018
37. • Real-time log streaming
• <3sec to dashboard
• Operational dashboards
• Single view with multiple
datasets
• Long term data retention
• 1stsolution to handle all SMS
logs (295 fields)
• ~250 mil records per day @3K
TPS
• Faster troubleshooting +
reporting (sec vs days)
• AI Algorithm – Spammer detection
• Anomaly based spammer
detection Faster and more
spammer detection than rules
based
Hadoop enables AI, Arcadia Data realtime visualization
Bell Canada Use Case from Webinar on 10/24/18:
View at: https://www.arcadiadata.com/lp/how-bell-canada-and-twilio-succeed-with-scaling-real-time-analytics/
38. Arcadia Data. Proprietary and Confidential
Data Drives Market DisruptionArcadia Data Streaming Visualizations
Data Sources
Historical
Native Access for Streaming Analytics – Real-Time + Historical
Real-Time
Advanced Visualizations
and Semantic Layer
Kudu
Kafka Cluster
Source Topics
Kudu Kudu
HDFS/S3 HDFS/S3
… …
………
……
39. Arcadia Data. Proprietary and Confidential
Can we make analytics as easy as
Google search?
Search-Based BI
48. Arcadia Data. Proprietary and Confidential
Native Architecture powering the ease of access
Native
Architecture
Smart
Acceleration
Connected
Ecosystem
AI Scoring and Recommendations
60. Apache Kudu: Scalable and fast tabular storage
Scalable
• Tested at 300+ nodes (PB-scale)
• Designed to scale to 1000s of nodes and tens of PBs
Fast
• Millions of read/write operations per second across cluster
• Multiple GB/second read throughput per node
Tabular
• Represents data in structured tables like a relational database
• Individual record-level access to 100+ billion row tables
• SQL & NoSQL access
61. Kudu + Impala vs DWH
Commonalities
✓ Fast analytic queries via SQL
✓ Ability to insert, update, and delete data
Differentiators from Traditional Warehouses
✓ Faster streaming inserts
✓ JOIN between HDFS + Kudu tables, run on same cluster
✗ Slower batch inserts
✗ Transactional data loading, multi-row transactions, or indexing
67. Social media: @arcadiadataarcadiadata.com 67
Dresner Big Data
Analytics Research
See Search-Based BI in
Action
Download
Arcadia Instant
arcadiadata.com/lp/2017-big-data-analytics-
market-study/
arcadiadata.com/product/
search-based-bi/
arcadiadata.com/instant