Managing Data Analytics in a Hybrid Cloud discusses challenges with traditional analytics approaches and proposes using shared data lakes with dynamic compute clusters. Common challenges include explosive analytics team growth leading to resource contention, and duplicating large datasets for each cluster. The proposed approach uses shared object storage to hold unified datasets accessed by multiple ephemeral analytics clusters provisioned on-demand. This allows teams independent resources while avoiding duplicate storage costs and improving agility. The document outlines example architectures and benefits of this shared data lake approach when implemented on a private or public cloud.

1. Managing Data Analytics in a Hybrid Cloud
Karan Singh
Sr. Solution Architect
Storage & Hyper-Converged Business Unit
Daniel Gilfix
Technical Marketing Manager
Storage & Hyper-Converged Business Unit

2. AGENDA
2
● CUSTOMER PAIN
● COMMON APPROACHES
● SHARED DATA LAKES
● HOW IT WORKS AND WHERE
● SUMMARY AND NEXT STEPS

3. CUSTOMER PAIN

4. INSERT DESIGNATOR, IF NEEDED4
CUSTOMER PAIN POINTS
EXPLOSIVE GROWTH
in data analytics teams
and analytic tools
MULTIPLE TEAMS COMPETING
for use of the same
big data resources.
CONGESTION
in busy analytic clusters
causing frustration and missed SLAs.
HADOOP
SPARKSQL
SPARK
HIVE
MAPREDUCE
PRESTO
IMPALA
KAFKA
NIFI
ETC.

5. INSERT DESIGNATOR, IF NEEDED5
RESULTING IN CUSTOMER CHOICES
Get a bigger cluster
for many teams to share.
Give each team
own dedicated cluster,
each with copy of
PBs of data.
Give teams ability to
spin-up/spin-down
clusters which can
share common data store.
#1 #2 #3

6. INSERT DESIGNATOR, IF NEEDED6
#3 ON-DEMAND ANALYTIC CLUSTERS
WITH A SHARED DATA LAKE
HIT SERVICE-LEVEL AGREEMENTS
Give teams their own
compute clusters.
ELIMINATE IDLE RESOURCES
By right-sizing de-coupled
compute and storage.
BUY 10s OF PBS INSTEAD OF 100S
Share data sets across clusters
instead of duplicating them.
INCREASE AGILITY
With spin-up/spin-
down clusters.

7. INSERT DESIGNATOR, IF NEEDED
Red Hat data analytics infrastructure solution
Multi-tenant workload isolation with shared data context
BATCH JOBS
(SLOW)
STREAMING
ANALYTICS
INTERACTIVE
ANALYTICS
OTHER
ANALYTICS
BATCH JOBS
(FAST)
DYNAMIC compute resources and
clusters able to meet different SLAs
UNIFIED single object storage
solution feeding analytics jobs
ELASTIC provisioning and release
of compute resources required by
various analytics jobs

8. BENEFITS - AGILITY AND $$$
● Faster answers through elastic provisioning via OSP on shared data sets
● Fewer roadblocks for empowered users in self-service data labs / clusters
● Private/public cloud versatility with S3A interface
● Reduced cost and risk from not duplicating and maintaining data sets
● CapEx relief by scaling storage independent from compute

9. HOW IT WORKS

10. INSERT DESIGNATOR, IF NEEDED10
GENERATION - I : ANALYTICS
MONOLITHIC HADOOP STACKS
Analytics vendors provide
single-purpose infrastructure
Analytics vendors provide
analytics software
ANALYTICS +
INFRASTRUCTURE
ANALYTICS +
INFRASTRUCTURE
ANALYTICS +
INFRASTRUCTURE

11. INSERT DESIGNATOR, IF NEEDED11
GENERATION - II : ANALYTICS
ELASTIC COMPUTE AND SHARED STORAGE CLOUDS
Analytics vendors provide
analytics software
Red Hat provides
cloud infrastructure software
Provisioned Compute Pool
via OpenStack and OpenShift platforms
Shared Datasets
on Red Hat Ceph Storage

12. INSERT DESIGNATOR, IF NEEDED12
MULTIPLE ANALYTIC CLUSTERS
SHARING DATA
INGEST ETL
INTERACTIVE
QUERY
BATCH QUERY
& JOINS
ELASTIC COMPUTE RESOURCE POOL
Kafka
compute instances
Hive/Map Reduce
compute instances
Presto
compute instances
Spark
compute instances
SHARED DATA LAKE
Platinum SLA
Gold SLA
Silver SLA
Bronze SLA

13. INSERT DESIGNATOR, IF NEEDED13
ANALYTIC WORKLOADS JOINING THE INFRA
storage silo
bare metal silo virtualization infra
shared storage SAN
Red Hat private cloud infra
Red Hat private cloud object store
The rest of an
enterprise’s apps
The rest of an
enterprise’s apps
VMs VMs today -> containers tomorrow

14. MULTI TENANT WORKLOAD ISOLATION
With Shared Data Context
HDFS TMP
HADOOP
RED HAT CEPH STORAGE
COMPUTE
STORAGE
COMPUTE
STORAGE
COMPUTE
STORAGE
WORKER
HADOOP CLUSTER 1
OPENSTACK
VM
OPENSHIFT
CONTAINER
2
3
HDFS TMP
SPARK
HDFS TMP
SPARK/
PRESTO
HDFS TMP
S3A S3A
BAREMETAL
RHEL
S3A/S3

15. INSERT DESIGNATOR, IF NEEDED15
COMMON ARCHITECTURAL MODEL -
PUBLIC OR PRIVATE CLOUD
PUBLIC CLOUD (AWS) PRIVATE CLOUD (RHT)
AWS EC2
PROVISIONING
RED HAT®
OPENSTACK PLATFORM
PROVISIONING
AWS S3
SHARED DATASETS
RED HAT®
CEPH S3
SHARED DATASETS
Hadoop
Presto
Spark Hadoop
Presto
Spark

16. INSERT DESIGNATOR, IF NEEDED16
FEATURES AND BENEFITS
MULTIPLE ANALYTIC CLUSTERS
• Enable teams to meet their individual SLAs without
competing for resources.
SHARED DATA SETS
• Eliminate duplicate storage costs for multiple HDFS cluster silos.
• Eliminate OpEx costs and complexity for maintaining multiple copies
of datasets for multiple HDFS cluster silos.
FAST PROVISIONING OF ANALYTIC CLUSTERS
• Unlocks Agility
• Enables Speed to Capability

17. ADVANCE ANALYTICS on CEPH

18. INSERT DESIGNATOR, IF NEEDED18
MODERN BIG DATA ANALYTICS PIPELINE
Simplified Example
DATA
GENERATION
INGEST DATA
SCIENCE
MACHINE
LEARNING
STREAM
PROCESSING
TRANSFORM,
MERGE,
JOIN
DATA
ANALYTICS

19. INSERT DESIGNATOR, IF NEEDED19
MODERN BIG DATA ANALYTICS PIPELINE
KEY TERMINOLOGY
DATA
GENERATION
INGEST DATA
SCIENCE
MACHINE
LEARNING
STREAM
PROCESSING
TRANSFORM,
MERGE, JOIN
DATA
ANALYTICS
• Sensors
• Click-stream
• Transactions
• Call-detail records
• NiFi
• Kafka • Presto
• Impala
• SparkSQL
• TensorFlow
• Kafka • Hadoop
• Spark
• Spark
• Hadoop

20. INSERT DESIGNATOR, IF NEEDED20
TESTED WITH CEPH OBJECT STORE
DATA
GENERATION
INGEST DATA
SCIENCE
MACHINE
LEARNING
STREAM
PROCESSING
TRANSFORM,
MERGE, JOIN
DATA
ANALYTICS
• TPC-DS data sets
(structured)
• logsynth
(semi-structured)
• bulk load
• MapReduce • Impala
• Presto
• (not tested)
• SparkSQL
• Hive/MapReduce
• SparkSQL
• Hive/MapReduce
• (not tested)

21. INSERT DESIGNATOR, IF NEEDED21
TYPICAL SHARED DATA LAKE
PROJECT STAGES
IDENTIFY
• Potential fit?
QUALIFY
• 1-2 day workshop
• ID questions needing evidence
• Prioritize questions by value
• Design POC architecture
POC OR PILOT
• Answer questions
• Empirical results
• RHT Solution Engineering
• RHT Consulting
DEPLOYMENT
• Phased roll-out
• Red Hat Consulting

22. SUMMARY AND NEXT STEPS

23. INSERT DESIGNATOR, IF NEEDED23
KEY TAKEAWAYS
MISSED SLAs
Large Spark/Hadoop
shops suffering from missed
SLAs due to cluster congestion.
EXCESSIVE CAPEX AND OPEX
due to multi-cluster
solutions without shared data.
Do you do big data analytics on-premises?
Do you have multi-PB data sets?
Do you have multiple Spark/Hadoop clusters?
Do these Spark/Hadoop clusters need to
share data sets?
Do you also have non Spark/Hadoop tools that
need access to these data sets?
PROBLEMS HOW YOU KNOW IT’S YOU

24. INSERT DESIGNATOR, IF NEEDED RED HAT CONFIDENTIAL
ONE CUSTOMER’S UNSOLICITED TESTIMONY
“We managed to deliver tremendous value to our organization”:
● Releasing lock on data: moving the HDFS to an open access object store and opening
the data process to more processes and analysis.
● Releasing lock on compute: now we’re able to spin up and decommission compute
power according to customer needs and utilize cloud benefits (including GPU incorporation
in zero time and effort), without worrying about the data.
● Releasing lock on innovation: we can now allow anyone to try and build something new
without the fear of messing things up (data or cluster wise). We’ve built an environment that
can tolerant mistakes at all levels (process and data), and by doing so, our developers can be
much more daring.“

25. INSERT DESIGNATOR, IF NEEDED RED HAT CONFIDENTIAL
CUSTOMER SATISFACTION
“I’m delighted to announce that its been a few weeks since we’ve launched our Cloudoop*
offering to our customers, and it’s a huge success. The responses from our customers are
very, very positive, and I’m quoting “Big big like!!!”
This shift from the traditional approach is revolutionizing the way we consume and process
our data.”
---- Head of Cloud Infrastructure, government agency
(*Cloudoop is their Spark-as-a-service offering with an S3 backend, Spark by Cloudera and an S3 by Ceph)

26. INSERT DESIGNATOR, IF NEEDED
RESOURCES
Summary-level blogs:
● Breaking down data silos with Red
Hat infrastructure
● Why would companies do this?
● Will mainstream analytics jobs run
directly against a Ceph object store?
● How much slower will they run than
natively on HDFS?
Architect-level blogs:
● What about locality?
● Anatomy of the S3A filesystem client
● To the cloud!
● Storing tables in Ceph object storage
● Comparing with HDFS—TestDFSIO
● Comparing with remote HDFS—Hive
Testbench (SparkSQL)
● Comparing with local HDFS—Hive
Testbench (SparkSQL)
● Comparing with remote HDFS—Hive
Testbench (Impala)
● AI and machine learning workloads

27. 27
SOCIAL MEDIA OPTIONS

28. THANK YOU