7. SPARK : LES CONCEPTS
¢ Application
— Main application
¢ Job
— Roundtrip Driver -> Cluster
¢ Stage
— Shuffle Boundary
¢ Task
— Thread working on a single RDD partition
¢ Partition
— RDD are split into partitions
— Partition is unit of work for each task
¢ Executor
— System process
Application
Job
Stage
Task
1
n
n
n
1
1
Driver code
Spark Action
Shuffle Boundary
One task / partition
RDD
Partition
27. REDUCE OBJECT SIZE
¢ Kryo Serialization
— Reduce space – up to 10X
— Perform gain – 2X to 3X
1. Require Spark to use Kryo serialization
2. Optimize class naming => reduce object size
3. Force all serialized classes to register
Not applicable to the Dataset API that use Encoders instead
30. SPARK MEMORY MANAGEMENT– SPARK 1.6.X
Shuffle fraction
Memory reserved to Spark (300Mb)
Memory assigned to user data
1 - spark.memory.fraction
Memory assigned to spark data
spark.memory.fraction = 0.75
spark.memory.storageFraction = 0.5
16Go
11.78Go
3.92Go
31. SPARK MEMORY MANAGEMENT – SPARK 1.6.X
¢ Storage Fraction
— Host cached RDDs
— Host broadcast variables
— Data in this fraction are subject to eviction
¢ Shuffling Fraction
— Hold Intermediate Data
— Spill to disk when full
— Data this fraction cannot be evicted
by other threads
Memory reserved to Spark (300Mb)
Memory assigned to user data
1 - spark.memory.fraction
Memory assigned to spark data
spark.memory.fraction = 0.75
spark.memory.storageFraction = 0.5
16Go
11.78Go
3.92Go
32. SPARK MEMORY MANAGEMENT – SPARK 1.6.X
¢ Shuffle & Storage fractions may borrow
memory from each other under certain
conditions:
— The shuffling fraction cannot extend beyond its
defined size if storage fraction uses all its
memory
— Storage fraction cannot evict data from the
shuffling fraction even if it expanded beyond its
defined size.
Memory reserved to Spark (300Mb)
Memory assigned to user data
1 - spark.memory.fraction
Memory assigned to spark data
spark.memory.fraction = 0.75
spark.memory.storageFraction = 0.5
16Go
11.78Go
3.92Go
36. SHUFFLE MANAGER
¢ spark.shuffle.manager= hash
— Perform better when the number of mapper/reducer is small (generation of M * R files)
¢ spark.shuffle.manager= sort
— Perform better for an important number of mapper/reducer
¢ Best of both world
— spark.shuffle.sort.bypassMergeThreshold
¢ spark.shuffle.manager= tungsten-sort ???
— Similar to sort with the following benefits :
¢ Off-Heap allocation
¢ Work directly on binary object serialization (much smaller than JVM objects) – aka memcopy J
¢ Use 8 bytes / record pour les sort => take advantage on L* cache
39. COARSE GRAINED MODE
L Only one executor / node
L No control over the
number of executors 8 cores
16g
8 cores
16g
8 cores
16g
8 cores
16g
16 cores
16g
16 cores
16g
32 cores / 64Go 32 cores / 32Go
40. FINE GRAINED MODE
L Only one executor / node
L No guaranty of resource
availability
L No control over the
number of executors
Spark Driver
MesosSchedulerBackend
Mesos Master
Mesos Worker
CoarseGrainedExecutorBackend
Mesos Worker
CoarseGrainedExecutorBackend
Mesos Worker
CoarseGrainedExecutorBackend
Mesos Worker
MesosExecutorBackend
Executor
Task
…
Executor
Task
Executor
Task
Executor
Task
spark.mesos.mesosExecutor.cores (1)
41. SOLUTION 1 : USE MESOS ROLES
¢ Limit the number of cores / node on each mesos worker
L Must be configured for each Spark application
• Assign a Mesos role to your Spark job
8 cores
16g
8 cores
16g
8 cores
16g
8 cores
16g
42. SOLUTION 2 : DYNAMIC ALLOCATION (COARSED GRAINED MODE ONLY)
¢ Principles
— Dynamically add/remove Spark executors
¢ Requires a dedicated process to move shuffled data (spawned on each Mesos
worker through Marathon)