Scala as the Big data language

1. Scala
The language for Big
Data
Tzach Zohar @ Kenshoo, March/2016

2. Who am I
System Architect @ Kenshoo
Java backend for 10 years
Working with Scala + Spark for 2 years
https://www.linkedin.com/in/tzachzohar

3. Who’s Kenshoo
10-year Tel-Aviv based startup
500+ employees
Industry Leader in Digital Marketing
Heavy data shop
http://kenshoo.com/

4. And who are you?

5. Agenda
NOT the usual Scala pitch

6. Scala - Short Intro

7. Scala
Created by Martin Odersky and his research group in EPFL, 2003
Open Source
Runs on the JVM, Seamless Java Interoperability
Strongly Typed
Object Oriented
Functional

8. Functional Programming
From Wikipedia:
“[…] functional programming […] treats
computation as the evaluation of mathematical
functions and avoids changing-state and
mutable data”

9. Functional Languages
Functions are “first-class citizens”, i.e. values
Higher-Order Functions
Minimize side effects
Minimize mutability

10. Example: Imperative to Functional
Java / Imperative:
private static class Person {
String firstName;
String lastName;
}
private List<Person> firstNFamilies(int n, List<Person> persons) {
final List<String> familiesSoFar = new LinkedList<>();
final List<Person> result = new LinkedList<>();
for (Person p : persons) {
if (familiesSoFar.contains(p.lastName)) {
result.add(p);
} else if (familiesSoFar.size() < n) {
familiesSoFar.add(p.lastName);
result.add(p);
}
}
return result;
}
Scala / Functional:
case class Person(firstName: String, lastName: String)
def firstNFamilies(n: Int, persons: List[Person]): List[Person] = {
val firstFamilies = persons.map(p => p.lastName).distinct.take(n)
persons.filter(p => firstFamilies.contains(p.lastName))
}

11. Hey, won’t this look
rather similar with
Java8’s Streams +
Lambdas?

12. Scala + Big Data

13. What can a language
do for Big Data?

14. Language “requirements”
Open Source
Strongly Typed
Java/JVM Friendly
Interactive
Performant
Abstracts “machinery” away

15. Open Source
Strongly Typed
Java/JVM Friendly
Interactive
Performant
Abstracts “machinery”
Java Interoperability
class DirectParquetOutputCommitter(outputPath: Path, context: TaskAttemptContext)
extends ParquetOutputCommitter(outputPath, context) { … }
Java class from org.apache.parquet:parquet-hadoop
Scala class from org.apache.spark:spark-core_2.10

16. Open Source
Strongly Typed
Java/JVM Friendly
Interactive
Performant
Abstracts “machinery”
Scala is Interactive
Scala has a built-in REPL, extensible by Scala-based tools

17. Open Source
Strongly Typed
Java/JVM Friendly
Interactive
Performant
Abstracts “machinery”
Performant
Benchmarking languages is hard and suffers from bias
Most benchmarks show Scala is at least on-par with Java,
e.g. Google’s benchmark:
Nonsense!
No Way!
RAGE!!11

18. Does it matter?

19. Open Source
Strongly Typed
Java/JVM Friendly
Interactive
Performant
Abstracts “machinery”
Performant
Ability to scale out is more significant than per-CPU performance
http://vmturbo.com/wp-content/uploads/2015/05/ScaleUpScaleOut_sm-min.jpg

20. Abstracts “machinery”
away - What?

21. Open Source
Strongly Typed
Java/JVM Friendly
Interactive
Performant
Abstracts “machinery”
Think about MapReduce
Hadoop’s Mapper and Reducer - code what to do, not:
How
Where
In what order
How to handle failures
Leaves these concerns for the framework to figure out

22. Open Source
Strongly Typed
Java/JVM Friendly
Interactive
Performant
Abstracts “machinery”
Think about MapReduce
Hadoop’s Java API imitates Functional Programming:
Mapper and Reducer are Functions
Executed by “Higher Order Functions”
No Side Effects / Mutability

23. Abstracts “machinery”
away - Why?

24. Open Source
Strongly Typed
Java/JVM Friendly
Interactive
Performant
Abstracts “machinery”
Functional makes concurrency easy
val numbers = 1 to 100000
val result = numbers.map(slowF)

25. Open Source
Strongly Typed
Java/JVM Friendly
Interactive
Performant
Abstracts “machinery”
Functional makes concurrency easy
val numbers = 1 to 100000
val result = numbers.par.map(slowF)
Parallelizes next manipulations over available CPUs

26. Open Source
Strongly Typed
Java/JVM Friendly
Interactive
Performant
Abstracts “machinery”
Functional makes distribution easy
val numbers = 1 to 100000
val result = sparkContext.parallelize(numbers).map(slowF)
Parallelizes next manipulations over scalable cluster, by
creating a Spark RDD - a Resilient Distributed Dataset

27. “Spark RDDs are
the ultimate Scala
collections"
- Martin Odersky
photo: http://www.swissict-award.ch/fileadmin/award/Pressebilder/Martin_Odersky_Scala.jpg

28. Open Source
Strongly Typed
Java/JVM Friendly
Interactive
Performant
Abstracts “machinery”
Functional makes Resiliency easy
Pure functions are idempotent, which allows retriability
Map
Map
MapMap Map (retry)

29. What if we always
coded this way?

30. A functional language
means just that

31. Language “requirements”
Open Source
Strongly Typed
Java/JVM Friendly
Interactive
Performant
Abstracts “machinery” away

32. But “Scala is hard!”

34. It’s really not that scary...
From
Manuel Bernhardt's "Debunking Some Myths About Scala And Its Environment":
“I need to become a mathematician and know all about Monads before I can get
started”
“I can throw all of my object-orientation knowledge out of the window”
“There is no good IDE support for Scala”

35. Thank You!