The Big Data industry emerged in response to the unprecedented sizes of data sets collected by Internet companies and the particular needs they had to store and use that data. Today, the need to process that data more quickly is morphing Big Data architectures into Fast Data architectures. This session discusses the forces driving this trend and the most popular tools that have emerged to address particular design challenges: Spark - For sophisticated processing of data streams, as well as traditional batch-mode processing. Kafka - For durable and scalable ingestion and distribution of data streams. Cassandra - For scalable, flexible persistence. Reactive Platform: Lagom, Akka, and Play - For integration of other components and building microservices. Mesos - For cluster resource management. --- About the presenter: Dean Wampler, Ph.D. is the Architect for Big Data Products and Services and a member of the office of the CTO at Lightbend. He is designing the product strategy and technical architecture for Lightbend's Spark on Mesos products and emerging streaming tools built around Spark and Lightbend’s ConductR and Akka products. Dean has written books on Scala, Functional Programming, and Hive for O'Reilly. He speaks at and co-organizes many industry conferences. He also organizes several Chicago-area user groups and contributes to many open-source projects, including Apache Spark. Dean has a Ph.D. in Physics from the University of Washington.

1. Modernizing Infrastructures for Fast Data
Spark, Kafka, Cassandra, Reactive Platform and Mesos
by Dean Wampler, Ph.D. (@deanwampler)

2. Outline
•Reactive Enterprise Architectures: The Lightbend Perspective
•Big Data and the Emergence of Apache Spark
•An Architecture for Fast Data
2

3. Reactive Enterprise Applications:
The Lightbend Perspective

4. Reactive Manifesto

5. The Lightbend Reactive Platform

7. 7
Online Services
IoT
Retail
Education
Technology
Social
Media
Finance
7

8. Big Data and the Emergence of
Apache Spark

9. Distributed compute frameworks: MapReduce
9
• Distribution computation over that data.

10. Hadoop
YARN
HDFS
MR job #1
MR job #2
Flume Sqoop
DBs
Slave Node
DiskDiskDiskDiskDisk
Node Mgr
Data Node
Master
Resource
Manager
Name Node

11. Hadoop Strengths
• Lowest CapEx system for Big Data.
• Excellent for ingesting and integrating diverse datasets.
• Flexible: from classic analytics (aggregations and data warehousing) to
machine learning.
11

12. Hadoop Weaknesses
• Complex administration.
• YARN can’t manage all distributed services.
• MapReduce:
•Has poor performance.
•A difficult programming model.
•Doesn’t support stream processing.
12

13. Why Apache Spark?

14. YARN
HDFS
MR job #1
MR job #2
Flume Sqoop
DBs
Slave Node
DiskDiskDiskDiskDisk
Node Mgr
Data Node
Master
Resource
Manager
Name Node
Spark job #1
Spark job #2
Hadoop 2013:
Embrace Spark

15. Spark vs. MapReduce Performance
15
100x better for
many algorithms.

16. Spark: Major Performance Improvements
16
Sort 100TB

17. One of the Fastest Growing OS Projects
17

18. Modules
18
Spark Streaming
(~Real Time)
MLlib
(Machine Learning)
SQL/DataFrames
(Structured Data)
GraphX
(Graphs)
Spark RDD
(Core)

19. The Core - Resilient Distributed Datasets
19
Spark Streaming
(~Real Time)
MLlib
(Machine Learning)
SQL/DataFrames
(Structured Data)
GraphX
(Graphs)
Spark RDD
(Core)
Cluster
Node Node Node Node
RDD
Partition 1
RDD
Partition 2
RDD
Partition 3
RDD
Partition 4

20. “Inverted Index” in Spark
20
sparkContext.textFile("/path/to/input")
.map { line =>
val array = line.split(",", 2)
(array(0), array(1))
}.flatMap {
case (id, contents) => toWords(contents).map(w=>((w,id),1))
}.reduceByKey {
(count1, count2) => count1 + count2
}.map {
case ((word, path), n) => (word, (path, n))}
.groupByKey
.map {
case (word, list) => (word, sortByCount(list))
}.saveAsTextFile("/path/to/output")
reduceByKey
flatMap
textFile
map
map
groupByKey
map
saveAsTextFile

21. SQL queries and a “DataFrame” DSL
21
Spark Streaming
(~Real Time)
MLlib
(Machine Learning)
SQL/DataFrames
(Structured Data)
GraphX
(Graphs)
Spark RDD
(Core)
• For data with a fixed schema...
• Write SQL queries (currently a subset of HiveQL).
• Use equivalent Python-inspired DataFrame API.

22. Use SQL or the Idiomatic DataFrame API
22
# SQL:
sqlContext.sql("""
SELECT state, age, COUNT(*) AS cnt
FROM people
GROUP BY state, age
ORDER BY cnt DESC, state ASC, age ASC
""")
// DataFrame (Scala):
people.state($"state", $"age")
.groupBy($"state", $"age").count()
.orderBy($"count".desc, $"state".asc, $"age".asc)

23. Spark Streaming: “Mini-batch” Processing
23
Spark Streaming
(~Real Time)
MLlib
(Machine Learning)
SQL/DataFrames
(Structured Data)
GraphX
(Graphs)
Spark RDD
(Core)
DStream
RDD #2
Event
Event
Event
Event
Event
…
Windows
(2 batches)
t0
RDD #1
Event
Event
RDD #3
Event
Event
Event
t1 =
t0 + ∆
t2 =
t0 + 2∆
t3 =
t0 + 3∆

24. Streaming Inverted Index
24
val kafkaBrokers = "host1:port1,host2:port2,..."
val kafkaTopics = Set("topic1", "topic2", ...)
val sparkConf = new SparkConf().setAppName("...")
val ssc = new StreamingContext(sparkConf, Seconds(2))
// Create direct kafka stream with kafkaBrokers and kafkaTopics
val kafkaParams = Map[String, String]("metadata.broker.list" -> kafkaBrokers)
val messages =
KafkaUtils.createDirectStream[String,String,StringDecoder,StringDecoder](
ssc, kafkaParams, kafkaTopics)
messages.flatMap {case (topic,text) => toWords(text).map(w=>((w,topic),1L))}
.reduceByKey (_ + _)
.map {case ((word, topic), n) => (word, (path, n))}
.groupByKey
.map {case (word, list) => (word, sortByCount(list))}

25. 25
val ssc = new StreamingContext(sparkConf, Seconds(2))
// Create direct kafka stream with kafkaBrokers and kafkaTopics
val kafkaParams = Map[String, String]("metadata.broker.list" -> kafkaBrokers)
val messages =
KafkaUtils.createDirectStream[String,String,StringDecoder,StringDecoder](
ssc, kafkaParams, kafkaTopics)
messages.flatMap {case (topic,text) => toWords(text).map(w=>((w,topic),1L))}
.reduceByKey (_ + _)
.map {case ((word, topic), n) => (word, (path, n))}
.groupByKey
.map {case (word, list) => (word, sortByCount(list))}
.saveAsTextFiles("/path/to/output")
ssc.start()
ssc.awaitTermination()

26. An Architecture
for Fast Data

27. •Update a search engine in real time as web page or
documents change.
•Train a SPAM filter with every email.
•Detect anomalies as they happen through processing of
logs and monitoring data.
Fast as in Streaming. Why?

28. Mesos, YARN
on
Bare Metal, Cloud
HDFS, S3, CFSv2SQL/NoSQL
Core
Streaming SQL
MLlib GraphX
Fast Data Architecture
HTTP/
REST
Internet
ReacHve
Services
Logs and
Other Files
Actors
Cluster …Persist
Akka Streams
Web Services

29. Mesos, YARN
on
Bare Metal, Cloud
HDFS, S3, CFSv2SQL/NoSQL
Core
Streaming SQL
MLlib GraphX
Fast Data Architecture
HTTP/
REST
Internet
ReacHve
Services
Logs and
Other Files
Actors
Cluster …Persist
Akka Streams
Web Services
Core of Spark, Kafka,
and Cassandra

30. Mesos, YARN
on
Bare Metal, Cloud
HDFS, S3, CFSv2SQL/NoSQL
Core
Streaming SQL
MLlib GraphX
Fast Data Architecture
HTTP/
REST
Internet
ReacHve
Services
Logs and
Other Files
Actors
Cluster …Persist
Akka Streams
Web Services
“SMACK”
Stack

31. Mesos, YARN
on
Bare Metal, Cloud
HDFS, S3, CFSv2SQL/NoSQL
Core
Streaming SQL
MLlib GraphX
Fast Data Architecture
HTTP/
REST
Internet
ReacHve
Services
Logs and
Other Files
Actors
Cluster …Persist
Akka Streams
Web Services
Data
Sources

32. Mesos, YARN
on
Bare Metal, Cloud
HDFS, S3, CFSv2SQL/NoSQL
Core
Streaming SQL
MLlib GraphX
Fast Data Architecture
HTTP/
REST
Internet
ReacHve
Services
Logs and
Other Files
Actors
Cluster …Persist
Akka Streams
Web Services
Event
Event
Event
Event
Event
Event
Producer Consumer
bounded queue
feedbackfeedback
Reactive Streams

33. Mesos, YARN
on
Bare Metal, Cloud
HDFS, S3, CFSv2SQL/NoSQL
Core
Streaming SQL
MLlib GraphX
Fast Data Architecture
HTTP/
REST
Internet
ReacHve
Services
Logs and
Other Files
Actors
Cluster …Persist
Akka Streams
Web Services
Lightbend Reactive
Platform

34. Mesos, YARN
on
Bare Metal, Cloud
HDFS, S3, CFSv2SQL/NoSQL
Core
Streaming SQL
MLlib GraphX
Fast Data Architecture
HTTP/
REST
Internet
ReacHve
Services
Logs and
Other Files
Actors
Cluster …Persist
Akka Streams
Web Services
Kafka for Stream
Storage

35. Mesos, YARN
on
Bare Metal, Cloud
HDFS, S3, CFSv2SQL/NoSQL
Core
Streaming SQL
MLlib GraphX
Fast Data Architecture
HTTP/
REST
Internet
ReacHve
Services
Logs and
Other Files
Actors
Cluster …Persist
Akka Streams
Web Services
Service 1
Log &
Other Files
Internet
Services
Service 2
Service 3
Services
Services
N * M links ConsumersProducers

36. Mesos, YARN
on
Bare Metal, Cloud
HDFS, S3, CFSv2SQL/NoSQL
Core
Streaming SQL
MLlib GraphX
Fast Data Architecture
HTTP/
REST
Internet
ReacHve
Services
Logs and
Other Files
Actors
Cluster …Persist
Akka Streams
Web Services
Service 1
Log &
Other Files
Internet
Services
Service 2
Service 3
Services
Services
N + M links ConsumersProducers

37. Mesos, YARN
on
Bare Metal, Cloud
HDFS, S3, CFSv2SQL/NoSQL
Core
Streaming SQL
MLlib GraphX
Fast Data Architecture
HTTP/
REST
Internet
ReacHve
Services
Logs and
Other Files
Actors
Cluster …Persist
Akka Streams
Web Services
Minibatch
Processing

38. Mesos, YARN
on
Bare Metal, Cloud
HDFS, S3, CFSv2SQL/NoSQL
Core
Streaming SQL
MLlib GraphX
Fast Data Architecture
HTTP/
REST
Internet
ReacHve
Services
Logs and
Other Files
Actors
Cluster …Persist
Akka Streams
Web Services
Short and Long-
term Storage

39. Mesos, YARN
on
Bare Metal, Cloud
HDFS, S3, CFSv2SQL/NoSQL
Core
Streaming SQL
MLlib GraphX
Fast Data Architecture
HTTP/
REST
Internet
ReacHve
Services
Logs and
Other Files
Actors
Cluster …Persist
Akka Streams
Web Services
Infrastructure

40. •Next Steps
•Learn - Fast Data: Big Data Evolved
•Watch - Using Spark, Kafka, Cassandra and Akka on
Mesos for Real-Time Personalization
•Review - Spark success stories by Lightbend clients