This presentation was at Budapest Spark Meetup. http://www.meetup.com/Budapest-Spark-Meetup/events/229462749/ The video from the presenation can be found here: http://ustre.am/1uMAT Let's Spark together - Beginners Edition! We will code our first Spark program together and I will explain the process step by step. We will dig deep what happens when we run our program, and how to set up your Spark environment at home. ​​Bio: ​​Mate is Co-founder and CTO of enbrite.ly, a Budapest based startup with the vision to create the next generation decision supporting system in online advertising that covers the market needs of the future. Mate has many years of experience with Big Data architectures, data analytics pipelines, operation of infrastructures and growing organizations by focusing on culture.​​Beside enbrite.ly, he is Chief Architect at Dmlab, a leading data analytics company providing innovative data products and services. Mate​teaches Big Data analytics​​ ​at Budapest University of Technology and Economics​ and ​runs courses for companies. Speaker of local and international conferences and meetups.

1. Apache Spark
Mate Gulyas

2. CTO & Co-Founder
GULYÁS MÁTÉ
@gulyasm

3. Getting Started

4. Spark Core
Spark SQL
Spark Streaming
MLlib
GraphX
Cluster Managers
UNIFIED STACK
Spark Core
Spark SQL
Spark Streaming
MLlib
GraphX
Cluster Managers

5. RDD API
Dataframe API
Dataset API
UNIFIED STACK
Spark Core
RDD API
Dataframe API
Dataset API

6. ❏ Scala
❏ Java
❏ Python
❏ R
WHICH LANGUAGE TO SPARK ON?

7. SPARK INSTALL

8. DRIVER
SPARKCONTEXT

9. DRIVER PROGRAM
Your main function. This is what you write.
Launches parallel operations on the cluster. The
driver access Spark through SparkContext.
You access the computing cluster via SparkContext
Via SparkContext you can create RDDs.

10. ❏ INTERACTIVE
❏ STANDALONE
A “SPARK SOFTWARE”

11. Resilient Distributed
Dataset (RDD)
THE MAIN ATTRACTION

12. RDD

13. ❏ TRANSFORMATION
❏ ACTION
OPERATIONS ON RDD

14. CREATES ANOTHER RDD
TRANSFORMATION

15. CALCULATE VALUE AND RETURN IT
TO THE DRIVER PROGRAM
ACTION

16. LAZY EVALUATION

17. INTERACTIVE

18. ❏ The code: github.com/gulyasm/bigdata
❏ Databricks site: spark.apache.org
❏ User mailing list
❏ Spark books
MATERIALS

19. MATE GULYAS
gulyasm@enbrite.ly
@gulyasm
@enbritely
THANK YOU!

20. TRANSFORMATIONS
ACTIONS
LAZY EVALUATION

21. LIFECYCLE OF A SPARK PROGRAM
1. READ DATA FROM EXTERNAL SOURCE
2. CREATE LAZY EVALUATED
TRANSFORMATIONS
3. CACHE ANY INTERMEDIATE RDD TO REUSE
4. KICK IT OFF BY CALLING SOME ACTION

22. PARTITIONS

23. RDD INTERNALS
RDD INTERFACE
➔ set of PARTITIONS
➔ list of DEPENDENCIES on PARENT RDDs
➔ functions to COMPUTE a partition given parents
➔ preferred LOCATIONS (optional)
➔ PARTITIONER for K/V pairs (optional)

24. MULTIPLE RDDs
/**
* :: DeveloperApi ::
* Implemented by subclasses to compute a given partition.
*/
@DeveloperApi
def compute(split: Partition, context: TaskContext): Iterator[T]
/** Implemented by subclasses to return the set of partitions in this RDD. */
protected def getPartitions: Array[Partition]
/** Implemented by subclasses to return how this RDD depends on parent RDDs.
*/
protected def getDependencies : Seq[Dependency[_]] = deps
/** Optionally overridden by subclasses to specify placement preferences. */
protected def getPreferredLocations (split: Partition): Seq[String] = Nil
/** Optionally overridden by subclasses to specify how they are partitioned.
*/
@transient val partitioner: Option[Partitioner] = None

25. INTERNALS

26. THE IMPORTANT PART
❏ HOW EXECUTION WORKS
❏ TERMINOLOGY
❏ WHAT SHOULD WE CARE ABOUT

27. PIPELINING
❏ Parallel to CPU pipelining
❏ More steps at a time
❏ Recap: computation kicks of when an
action is called due to lazy evaluation

28. PIPELINING
text = sc.textFile("twit1.txt")
words = nonempty.flatMap(lambda x: x.split(" "))
fwords = words.filter(lambda x: len(x) > 0)
ones = fwords.map(lambda x: (x, 1))
result = ones.reduceByKey(lambda l,r: r+l)
result.collect()

29. PIPELINING
text = sc.textFile( )
words = nonempty.flatMap( )
fwords = words.filter( )
ones = fwords.map( )
result = ones.reduceByKey( )
result.collect()

30. PIPELINING
sc.textFile( )
.flatMap( )
.filter( )
.map( )
.reduceByKey( )

31. PIPELINING
sc.textFile().flatMap().filter().map().reduceByKey()

32. RDD RDD RDD RDD RDD
textFile(
) flatMap() filter() map() reduceByKey()
text resultwords fwords ones
PIPELINING

33. PIPELINING
def runJob[T, U](
rdd: RDD[T],
partitions: Seq[Int],
func: (Iterator[T]) => U)
) : Array[U]

34. RDD RDD RDD RDD RDD
textFile(
) flatMap() filter() map() reduceByKey()
text resultwords fwords ones
collect()
PIPELINING

35. JOB
❏ Basically an action
❏ An action creates a job
❏ A whole computation with all
dependencies

36. RDD RDD RDD RDD RDD
textFile(
) flatMap() filter() map() reduceByKey()
text resultwords fwords ones
collect()
Job

37. STAGE
❏ Unit of execution
❏ Named after the last transformation
(the one runJob was called on)
❏ Transformations pipelined together into
stages
❏ Stage boundary usually means shuffling

38. RDD RDD RDD RDD RDD
textFile(
) flatMap() filter() map() reduceByKey()
text resultwords fwords ones
collect()
Job
Stage 1 Stage 2

39. STAGE
❏ Unit of execution
❏ Named after the last transformation
(the one runJob was called on)
❏ Transformations pipelined together into
stages
❏ Stage boundary usually means shuffling

40. RDD RDD RDD RDD RDD
textFile(
) flatMap() filter() map() reduceByKey()
text resultwords fwords ones
collect()
Job
Stage 1 Stage 2
PT1
PT2
PT1
PT2
PT1
PT2
PT1
PT2
PT1
PT1
Shuffle

41. Repartitioning
text = sc.textFile("twit1.txt")
words = nonempty.flatMap(lambda x: x.split(" "))
fwords = words.filter(lambda x: len(x) > 1)
ones = fwords.map(lambda x: (x, 1))
rp = ones.repartition(6)
result = rp.reduceByKey(lambda l,r: r+l)
result.collect()

42. TaskSet
THE PROCESS
RDD Objects DAG Scheduler Task Scheduler Executor
RDD
RDD RDD
RDD
RDD
sc.textFile.
map()
.groupBy()
.filter()
Build DAG of
operators
T
T
T
T
T
T
T
T
T
S
S
S
S
- Split DAG into
stages of tasks
- Each stage when
ready = ALL
dependent task are
finished
DAG Task
Task
Scheduler
- Launches tasks
- Retry failed tasks
Executor
Block manager
Task threads
Task threads
Task threads
- Store and serve
blocks
- Executes tasks

43. MATE GULYAS
gulyasm@enbrite.ly
@gulyasm
@enbritely
THANK YOU!