This document discusses lessons learned from developing and managing massive (300TB+) Apache Spark pipelines in production. It covers using Spark for its performance, testability, and modularity benefits. It also discusses managing large scale through automation, simple interfaces, planning for growth by persisting data to HDFS, using efficient serialization and data structures, and testing on sampled data to address data skew issues.

1. LESSONS LEARNED DEVELOPING AND MANAGING MASSIVE
(300TB+) APACHE SPARK PIPELINES IN PRODUCTION
Brandon Carl

3. MARCH 15, 2016
"SEE THE MOMENTS YOU CARE ABOUT FIRST"

5. MACHINE LEARNING

6. MACHINE LEARNING LIFECYCLE
Training
Examples
Machine
Learning
Model
Make
Predictions
Measure
Outcomes
Ranking
Events
Client
Events

7. WHY SPARK?
• Performance
• Testability
• Modularity
• Serialized Logging

8. SERIALIZED LOGGING
{
"id": 123,
"scores": {
"modelA": 0.2345,
"modelB": 0.0012
},
"features": {
1001: 0.9934,
1002: 0.1923
}
}

9. SERIALIZED LOGGING
struct Candidate {
1: i64 id;
2: map<string, double> scores;
3: map<i64, double> features;
}
new Candidate()
.setId(id)
.setScores(scores)
.setFeatures(features)

10. CHANGES OVER TIME

11. CHANGES OVER TIME
• RDD
• Dataset
• Training Data Joiner

12. TRAINING DATA JOINER
class MyTrainingDataJoiner(spark: SparkSession) extends TrainingDataJoiner {
val labels: Map[String, LabelFunction] = ???
}
case class Output(id: Long, label_value: Double)

13. MANAGING MASSIVE SCALE

14. MANAGING MASSIVE SCALE - PEOPLE

15. AUTOMATE EVERYTHING

16. SIMPLE INTERFACE

17. SIMPLE INTERFACE
RankingEvent
.read('input_table', '2017-10-25')
.filter(...)
.map(...)
.write('output_table', '2017-10-25')

18. MANAGING MASSIVE SCALE - DATA

19. PLAN FOR GROWTH

20. PERSIST TO HDFS

21. PERSIST TO HDFS
Source
Data
Map/Filter
Join Output
Source
Data
Map/Filter

22. PERSIST TO HDFS
Source
Data
Map/Filter
Temporary
Table
Source
Data
Map/Filter
Temporary
Table
Join Output

23. KRYO SERIALIZATION

24. KRYO SERIALIZATION
new SparkConf()
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.set("spark.kryo.registrationRequired", "true")
.registerKryoClasses(Array(classOf[...], ...))

25. BIG-O MATTERS

26. BIG-O MATTERS
final def withName(s: String): Value =
values
.find(_.toString == s)
.getOrElse(throw new NoSuchElementException(...))

27. BIG-O MATTERS
final def withName(s: String): Value =
values
.find(_.toString == s)
.getOrElse(throw new NoSuchElementException(...))

28. DATA STRUCTURES MATTER

29. DATA STRUCTURES MATTER
• AnyRefMap
• IntMap
• LongMap
• fastutil (http://fastutil.di.unimi.it)

30. DATA SKEW MATTERS

31. TEST ON SAMPLED DATA