Hadoop is about so much more than batch processing. With the recent release of Hadoop 2, there have been significant changes to how a Hadoop cluster uses resources. YARN, the new resource management component, allows for a more efficient mix of workloads across hardware resources, and enables new applications and new processing paradigms such as stream-processing. This talk will discuss the new design and components of Hadoop 2, and examples of Modern Data Architectures that leverage Hadoop for maximum business efficiency.
Scale 12 x Efficient Multi-tenant Hadoop 2 Workloads with Yarn
1. Hadoop
2:
Efficient
mul3-‐tenant
workloads
that
enable
the
Modern
Data
Architecture
SCALE
12X,
Los
Angeles
February
23,
2014
David
Kaiser
@ddkaiser
linkedin.com/in/dkaiser
facebook.com/dkaiser
dkaiser@cdk.com
dkaiser@hortonworks.com
2. Who Am I?
20+
years
experience
with
Linux
3
years
experience
with
Hadoop
Career
experiences:
• Data
Warehousing
• Geospa3al
Analy3cs
• Open-‐source
Solu3ons
and
Architecture
Employed
at
Hortonworks
as
a
Senior
Solu3ons
Engineer
David
Kaiser
@ddkaiser
linkedin.com/in/dkaiser
facebook.com/dkaiser
dkaiser@cdk.com
dkaiser@hortonworks.com
3. Hadoop 2: Efficient multi-tenant workloads
that enable the Modern Data Architecture
• Abstract:
– Hadoop
is
about
so
much
more
than
batch
processing.
With
the
recent
release
of
Hadoop
2,
there
have
been
significant
changes
to
how
a
Hadoop
cluster
uses
resources.
– YARN,
the
new
resource
management
component,
allows
for
a
more
efficient
mix
of
workloads
across
hardware
resources,
and
enables
new
applica3ons
and
new
processing
paradigms
such
as
stream-‐processing.
– This
talk
will
discuss
the
new
design
and
components
of
Hadoop
2,
and
provide
examples
of
Modern
Data
Architectures
that
leverage
Hadoop
2.
14. Hadoop + Linux
Provides a 100% Open-Source framework for efficient
scalable data processing on commodity hardware
Hadoop
–
The
Open-‐source
Data
Opera3ng
System
Linux
–
The
Open-‐source
Opera3ng
System
Commodity
Hardware
15. Hadoop Fundamentals
• Hadoop is a single system, across multiple Linux systems
• Two basic capabilities of Hadoop
– Reliable,
Redundant
and
Distributed
Storage
– Distributed
Computa3on
• Storage: Hadoop Distributed File System (HDFS)
– Replicated,
distributed
filesystem
– Blocks
wriZen
to
underlying
filesystem
on
mul3ple
nodes
• Computation
– Resource
management
– Frameworks
to
divide
workloads
across
collec3on
of
resources
– Hadoop
V1:
MapReduce
framework
only
– Hadoop
V2:
MapReduce,
Tez,
Spark,
others…
17. Hadoop 1 Computation
• MapReduce Framework
– Combined
both
Resource
Management
and
Applica3on
Logic
in
the
same
code
• Limitations
– Resource
alloca3on
units
(slots)
fixed
per
cluster
– Difficult
to
use
a
cluster
for
differing
or
simultaneous
workloads
18. The 1st Generation of Hadoop: Batch
HADOOP
1.0
Built
for
Web-‐Scale
Batch
Apps
Single
App
Single
App
INTERACTIVE
ONLINE
Single
App
Single
App
Single
App
BATCH
BATCH
BATCH
HDFS
HDFS
HDFS
• All
other
usage
paZerns
must
leverage
that
same
infrastructure
• Forces
the
crea3on
of
silos
for
managing
mixed
workloads
20. MapReduce Classic: Limitations
• Scalability
– Maximum
Cluster
size
–
4,000
nodes
– Maximum
concurrent
tasks
–
40,000
– Coarse
synchroniza3on
in
JobTracker
• Availability
– Failure
kills
all
queued
and
running
jobs
• Hard partition of resources into map and reduce slots
– Low
resource
u3liza3on
• Lacks support for alternate paradigms and services
– Itera3ve
applica3ons
implemented
using
MapReduce
are
10x
slower
Page 20
21. Hadoop 1: Poor Utilization of Cluster Resources
Hadoop
1
JobTracker
and
TaskTracker
used
fixed-‐sized
“slots”
for
resource
alloca3on
Map
tasks
are
wai3ng
for
the
slots
which
are
NOT
currently
used
by
reduce
tasks
Hard-‐Coded
values.
Task
tracker
must
be
restarted
aker
a
change
22. Hadoop 2: Moving Past MapReduce
Single
Use
System
Mul/
Purpose
Pla5orm
Batch
Apps
Batch,
Interac/ve,
Online,
Streaming,
…
HADOOP
1.0
HADOOP
2.0
MapReduce
Others
(data
processing)
MapReduce
YARN
(cluster
resource
management
&
data
processing)
(cluster
resource
management)
HDFS
HDFS2
(redundant,
reliable
storage)
(redundant,
highly-‐available
&
reliable
storage)
Page
22
23. Apache Tez as the new Primitive
MapReduce
as
Base
Apache
Tez
as
Base
HADOOP
1.0
HADOOP
2.0
Batch
MapReduce
Pig
(data
flow)
Hive
Others
(sql)
(cascading)
MapReduce
Data
Flow
Pig
SQL
Hive
Others
Real
Time
Stream
Processing
Storm
(cascading)
Tez
(execu3on
engine)
HBase,
Accumulo
??
(HOYA)
(con3nuous
execu3on)
YARN
(cluster
resource
management
&
data
processing)
(cluster
resource
management)
HDFS
HDFS2
(redundant,
reliable
storage)
Online
Data
Processing
(redundant,
reliable
storage)
24. Tez – Execution Performance
• Performance gains over Map Reduce
– Eliminate
replicated
write
barrier
between
successive
computa3ons.
– Eliminate
job
launch
overhead
of
workflow
jobs.
– Eliminate
extra
stage
of
map
reads
in
every
workflow
job.
– Eliminate
queue
and
resource
conten3on
suffered
by
workflow
jobs
that
are
started
aker
a
predecessor
job
completes.
Pig/Hive
-‐
MR
Pig/Hive
-‐
Tez
Page
24
25. YARN: Taking Hadoop Beyond Batch
Store ALL DATA in one place…
Interact with that data in MULTIPLE WAYS
with Predictable Performance and Quality of Service
ApplicaSons
Run
NaSvely
in
Hadoop
BATCH
INTERACTIVE
(MapReduce)
(Tez)
ONLINE
(HBase)
STREAMING
(Storm,
S4,…)
GRAPH
(Giraph)
IN-‐MEMORY
(Spark)
HPC
MPI
(OpenMPI)
OTHER
(Search)
(Weave…)
YARN
(Cluster
Resource
Management)
HDFS2
(Redundant,
Reliable
Storage)
Page 25
26. YARN Overview
• Goals:
– Reduce
the
responsibili3es
of
the
JobTracker
– Separate
the
resource
management
du3es
away
from
the
job
coordina3on
du3es
– Allow
mul3ple
simultaneous
jobs
– Enables
different
style
and
sized
workloads
in
one
cluster
• Design:
– A
separate
Resource
Manager
– 1
Global
Resource
Scheduler
for
the
en3re
cluster
– Each
worker
(slave)
node
runs
a
Node
Manager,
manages
life-‐cycle
of
containers
– JobTracker
is
now
called
Applica3on
Master
– Each
Applica3on
has
1
Applica3on
Master
– Manages
applica3on
scheduling
and
task
execu3on
28. Capacity Sharing: Concepts
• Application
– Applica3on
is
a
temporal
job
or
a
service
submiZed
to
YARN
– Examples
– Map
Reduce
Job
(job)
– Storm
topology
(service)
• Container
– Basic
unit
of
alloca3on
– Fine-‐grained
resource
alloca3on
across
mul3ple
resource
types
(memory,
cpu,
disk,
network,
etc.)
– container_0
=
2GB
– container_1
=
1GB
– Replaces
fixed
map/reduce
slots
(from
Hadoop
1.x)
28
29. YARN – Resource Allocation & Usage!
• ResourceRequest!
– Fine-‐grained
resource
ask
to
the
ResourceManager
– Ask
for
a
specific
amount
of
resources
(memory,
cpu
etc.)
on
a
specific
machine
or
rack
– Use
special
value
of
*
for
resource
name
for
any
machine
ResourceRequest!
priority!
resourceName!
capability!
numContainers!
priority!
capability!
!
0!
!
<2gb, 1 core>!
resourceName! numContainers!
<4gb, 1 core>!
1!
rack0!
1!
*!
1!
host01!
1!
*!
1!
Page
29
30. CGroup
• Linux Kernel capability to limit, account and isolate resources
– CPU
:
Controlling
the
prioriza3on
of
processes
in
the
group.
Think
of
it
as
a
more
advanced
nice
level
– Memory
:
Allow
for
setng
limits
on
RAM
and
swap
usage
– Disk
I/O
– Network
• YARN currently support, CPU / Memory
31. List of YARN Apps
• MapReduce (of course)
• Apache Tez
– Apache
Hive
– Apache
Pig
• Apache Hama - Iterative, Bulk Synchronous Parallel (BSP) engine
• Apache Giraph - Iterative, BSP-based Graph Analysis engine
• HBase on YARN (HOYA)
• Apache Storm – Real-time stream processing
• Apache Spark – Advanced DAG execution engine that supports cyclic data
flow and in-memory computing
• Apache S4 – Real-time processing
• Open MPI – Open source Message Passing Interface for HPC
http://wiki.apache.org/hadoop/PoweredByYarn
32. The YARN Book
• “Coming Soon”
• Expected by 2nd Quarter 2014
• Complete coverage of YARN
33. Modern Data Architecture
• Effective use of data – especially BIG Data – is enhanced when data is
co-located, enabling discovery and mining of unanticipated patterns.
• A “Data Lake” is the growing body of all data
– Encompassing
more
than
a
single
warehouse
– Data
can
con3nuously
stream
in
to
and
out
of
the
lake
34. Multi-Tenancy Requirements
Multi-Tenancy in one shared cluster
• Multiple Business Units
• Multiple Applications
Requirements
• Shared Processing Capacity
• Shared Storage Capacity
• Data Access Security
Page
34
35. Multi-Tenancy: Capabilities
• Group and User:
– Use
of
Linux
and
HDFS
permissions
to
separate
files
and
directories
to
create
tenant
boundaries
–
can
be
integrated
with
LDAP
(or
AD)
• Security
– Used
to
enforce
tenant
boundaries
–
can
be
integrated
with
Kerberos
• Capacity:
– Storage
quota
setup
to
manage
consump3on
– Capacity
resource
scheduler
queues
to
balance
shared
processing
resources
between
tenants
–
Use
ACLs
to
define
tenants
Page
35
36. FUNCTION
Capacity
Sharing
FUNCTION
Capacity
Enforcement
FUNCTION
The Capacity Scheduler
Admin-‐
istraSon
•
Queues
with
priori3es
•
ACLs
for
job
submit
permissions
• Max
capacity
per
queue
• User
limits
within
queue
• Monitoring
+
Management
Admin
ACLs
• Capacity-‐Scheduler.xml
Page 36
37. Roadmap: Capacity Scheduling
Feature
DescripSon
CS
Pre-‐emp3on
• Enhance
SLA
support
• Re-‐claim
capacity
from
tasks
in
queue
that
have
been
over-‐scheduled
Queue
Hierarchy
• Granular
configura3on
of
queues
• Provide
constraints
across
a
set
of
queues
Node
Labels
• Schedule
tasks
on
specific
cluster
nodes
• Account
for
op3mized
hardware
Container
Isola3on
• Stronger
isola3on
of
resources
for
each
container,
incorpora3ng
CPU
CPU
Scheduling
• Schedule
and
share
CPU
core
capacity
across
tasks
37
38. Capacity Scheduler by example
Total
Cluster
capacity
• 20
slots
• 11
Mappers
• 9
Reducers
Queue
:
ProducSon
• Guarantee
70%
resources
• 14
slots
–
8M
/
6R
• Max
100%
Queue
:
Dev
• Guarantee
10%
resources
• 2
slots
–
1M
/
1R
• Max
50%
Queue
:
Default
• Guarantee
20%
resources
• 4
slots
–
2M
/
2R
• Max
80%
39. Hierarchical queues
root
Dev
10%
Eng
20%
Default
20%
Test
80%
Produc3on
70%
DevOps
10%
Reserved
20%
Prod
70%
P0
70%
P1
30%
39
40. CS: Example Queue Configuration
• Default: 10 users | Ad-hoc BI Query jobs etc. | General User SLAs
• Dev: 4 users | Ad-hoc Data Science Only (Pig+Mahout) | Lower SLAs
• Applications: 2 users | Batch ETL and Report Generation jobs | Production SLAs
Yarn.scheduler.capacity.root.default
Capacity
ACLs
Min:
0.10
|
Max:
0.20
|
User
Limit:
0.8
‘Users’
group
Yarn.scheduler.capacity.root.dev
Capacity
ACLs
Min:
0.10
|
Max:
0.10
|
User
Limit:
0.5
‘Engineering’
group
Yarn.scheduler.capacity.root.producSon
Capacity
ACLs
Min:
0.20
|
Max:
0.70
|
User
Limit:
1.0
‘Applica3ons’
group
40
43. Capacity Scheduler by example
• Job 1 : Launch in production queue
– Require
100
slots
– Get
14
slots
at
a
3me
Cluster
resources
Produc3on
Development
Default
Idle
44. Capacity Scheduler by example
• Job 1 : Running in Production queue
– Using
14
slots
• Job 2 : Schedule in Development queue
– Require
50
slots
– Get
4
slots
at
a
3me
Cluster
resources
Produc3on
Development
Default
Idle
45. Capacity Scheduler by example
• Job 1 : Running in Production queue
– 98
complete,
only
2
slots
in
use
un3l
finish
• Job 2 : Schedule in Development queue
– Require
50
slots
– S3ll
only
getng
4
slots
at
a
3me
Cluster
resources
Produc3on
Development
Default
Idle
46. Summary
• YARN is the logical extension of Apache Hadoop
– Complements
HDFS,
the
data
reservoir
• Resource Management for the Enterprise Data Lake
– Shared,
secure,
mul3-‐tenant
Hadoop
Allows for all processing in Hadoop
BATCH
INTERACTIVE
(MapReduce)
(Tez)
ONLINE
(HBase)
STREAMING
(Storm,
S4,…)
GRAPH
(Giraph)
IN-‐MEMORY
(Spark)
HPC
MPI
(OpenMPI)
OTHER
(Search)
(Weave…)
YARN
(Cluster
Resource
Management)
HDFS2
(Redundant,
Reliable
Storage)
Page
46
47. Your Fastest On-ramp to Enterprise Hadoop™!
hZp://hortonworks.com/products/hortonworks-‐sandbox/
The
Sandbox
lets
you
experience
Apache
Hadoop
from
the
convenience
of
your
own
laptop
–
no
data
center,
no
cloud
and
no
internet
connec3on
needed!
The
Hortonworks
Sandbox
is:
• A
free
download:
hZp://hortonworks.com/products/hortonworks-‐sandbox/
• A
complete,
self
contained
virtual
machine
with
Apache
Hadoop
pre-‐configured
• A
personal,
portable
and
standalone
Hadoop
environment
• A
set
of
hands-‐on,
step-‐by-‐step
tutorials
that
allow
you
to
learn
and
explore
Hadoop
Page
47
48. Ques3ons?
David
Kaiser
@ddkaiser
linkedin.com/in/dkaiser
facebook.com/dkaiser
dkaiser@cdk.com
dkaiser@hortonworks.com