Spark RDD Spark Streaming

1. TechLabs by
A la découverte de Machine Learning, de Redis et de Spark

2. TechLabs by
2
Maturin BADO
@mccstanmbg
github.com/mccstan
SPARK

3. Spark : Introduction

4. Outline
❏ Data processing today
❏ Spark, hadoop, MapReduce
❏ Spark ecosystem
❏ Spark basics

5. Data processing today
Data intensive application
Definition :
“We call an application data-intensive if data is its primary challenge—the
quantity of data, the complexity of data, or the speed at which it is changing—as
opposed to compute-intensive, where CPU cycles are the bottleneck.”
Martin Klepmann

6. Data processing today
Today apps needs :
❏ Store data (databases)
❏ Caches
❏ search data (search index)
❏ Asynchronously message handling (stream processing)
❏ batch processing

7. Spark, hadoop, MapReduce

8. Spark, hadoop, MapReduce
Spark : main differences with Map Reduce
❏ Spark load most of the dataset in memory
❏ Implement cache mechanisms which reduce read from disk
❏ Is much faster than MapReduce : Job scheduling
❏ Does not implement any data distribution technology but
can run on top of hadoop clusters (HDFS )

9. Spark ecosystem : open source

10. Spark ecosystem : features

11. Spark ecosystem : deployment

12. Spark basics : RDD
RDD : Resilient Distributed data
❏ Primary spark abstraction
❏ Fault tolerant collection of elements
❏ Partitioned and Immutable
❏ Two types operations
❏ Lazy Transformation

13. Spark basics : An execution flow

14. Spark Streaming

15. Outline
❏ Why In-stream processing ?
❏ Runtime and Programming Model
❏ Spark Streaming : Overview
❏ Benefits of Discretized Stream Processing
❏ Processing flow
❏ Transform operations
❏ Window operations

16. Why In-stream processing ?

17. Why In-stream processing ?

18. Runtime and Programming Model
Native Streaming

19. Runtime and Programming Model
Micro-batch Streaming

20. Spark Streaming : Overview

21. Benefits of Discretized Stream Processing
Dynamic load balancing

22. Benefits of Discretized Stream Processing
Fast failure and straggler recovery

23. Benefits of Discretized Stream Processing
❏ Unification of batch, streaming and interactive analytics
❏ Advanced analytics like machine learning and interactive SQL
❏ Streaming + SQL and DataFrames
❏ Streaming + MLlib

24. Spark Streaming : Processing flow

25. Spark Streaming : DStreams
Discretized Streams (DStreams) :
❏ The basic spark streaming abstraction
❏ A continuous series of RDDs

26. Spark Streaming : Transformations
Transform Operations : Any operation applied on a DStream translates
to operations on the underlying RDDs

27. Spark Streaming : Transformations
Window Operations :

28. Spark Streaming : Time abstractions
Batch interval
Sliding interval
Window size

29. Spark Streaming : Time abstractions
Batch interval
Window size
Sliding interval

30. Spark Streaming : Some examples
❏ Wordcount
❏ stateless operation, counting words for every batch
❏ Basic Error count
❏ stateless operation, using a filter : contains(“ERROR”)
❏ Cumulative Error count
❏ Stateful operation, errors from the beginning of the processing
❏ Windowed Errors counts
❏ Stateful operation, errors from the sliding window of time

31. The git repo
https://github.com/SoatGroup/spark-streaming-java-examples
https://github.com/SoatGroup/spark-streaming-python