Scala and Spark: Coevolving Ecosystems for Big Data
•
4 gostaram•1,339 visualizações
Presented at Datapalloza Seattle, Feb 10-11, 2016
Recomendados
Mais conteúdo relacionado
Destaque
Destaque (19)
Mais de John Nestor
Último
Último (20)
Scala and Spark: Coevolving Ecosystems for Big Data
- 1. Scala and Spark: Coevolving Ecosystems for Big Data John Nestor 47 Degrees Datapalooza Seattle February 10-11, 2016 www.47deg.com 147deg.com
- 2. 47deg.com Outline • Scala • Spark • Scala Impact on Spark • Spark Impact on Scala • Summary • Questions 2
- 3. Scala 3
- 4. 47deg.com Scala History • Scala created by Martin Odersky, EFPL Switzerland • wrote javac, the compiler for Java • designed Java generics • 2004 Scala announced • 2006 Scala 2.0 • 2011 Typesafe started, Scala 2.9 • 2012 Scala 2.10 • 2014 Scala 2.11 • 2015 Scala 2.12-RC1 • 2016 Scala now 12 years old and quite mature 4
- 5. 47deg.com Why Scala? • Open Source • Strong typing • Concise elegant syntax • Runs on JVM (Java Virtual Machine) • Supports both object-oriented and functional • Small simple programs (REPL) through very large multi-server systems • Easy to cleanly extend with new libraries and DSL’s • Ideal for parallel and distributed systems 5
- 6. 47deg.com Scala: Strong Typing and Concise Syntax • Strong typing like Java. • Compile time checks • Better modularity via strongly typed interfaces • Easier maintenance: types make code easier to understand • Concise syntax like Python. • Type inference. Compiler infers most types that had to be explicit in Java. • Powerful syntax that avoid much of the boilerplate of Java code (see next slide). • Best of both worlds: safety of strong typing with conciseness (like Python). 6
- 7. 47deg.com Scala Case Class • Java version class User { private String name; private Int age; public User(String name, Int age) { this.name = name; this.age = age; } public getAge() { return age; } public setAge(Int age) { this.age = age;} } User joe = new User(“Joe”, 30); • Scala version case class User(name:String, var age:Int) val joe = User(“Joe”, 30) 7
- 8. 47deg.com Key Scala Features • Immutable Collections • Seq, Set, Map • Functional Programming • (i:Int) => i + 1 • Functions can be parameters and results • For collections: map, filter, flatMap, groupBy, … • Case Classes • case class Person(name:String, age:Int) • Define Domain Specific Languages (DSLs) • Akka, SBT, Spray, ScalaTest and now Spark 8
- 9. 47deg.com View Sample Scala Code • Word Count in Scala 9
- 10. Spark 10
- 11. 47deg.com Spark History • 2009 Mesos developed at UC Berkeley AmpLab • 2009 Matei Zaharia starts Spark as a test case for Mesos • 2012 Spark 0.5 • 2014 Spark 1.0, Top Level Apache Project • 2014 Databricks started • 2016 Spark 1.6 11
- 12. 47deg.com Why Spark? • Support for not only batch but also (near) real-time • Fast - keeps data in memory as much as possible • Often 10X to 100X Hadoop speed • A clean easy-to-use API • A richer set of functional operations than just map and reduce • A foundation for a wide set of integrated data applications • Can recover from failures - recompute or (optional) replication • Scalable for very large data sets and reduced time 12
- 13. 47deg.com Spark Components • Spark Core • Scalable multi-node cluster • Failure detection and recovery • RDDs, Dataframes and functional operations • MLLib - for machine learning • linear regression, SVMs, clustering, collaborative filtering, dimension reduction • more on the way! • GraphX - for graph computation • Streaming - for near real-time 13
- 14. 47deg.com Spark RDDs • RDD[T] - resilient distributed data set • typed • immutable • ordered • can be processed in parallel • lazy evaluation - permits more global optimizations • Rich set of functional operations ( a small sample) • map, flatMap, reduce, … • filter, groupBy, sortBy, take, drop, … 14
- 15. 47deg.com Spark Data Frames • Similar to SQL tables • can be transformed using SQL • Focus of much of the work on Spark performance optimization • Unlike RDD’s, optimized knows about fields • Dynamically typed • Not a natural fit for Scala (more on this later) 15
- 16. 47deg.com View Sample Spark Code - RDDs • Word Count for Spark written Scala 16
- 17. 47deg.com View Sample Spark Code - Data Frames • Tweet language count using Data Frames written in Scala 17
- 18. Scala Impact on Spark 18
- 19. 47deg.com Spark Impact on Scala • Written in Scala • Scala is primary API • Must use Scala to extend • Source code as documentation: Scala code • Design from Scala • functional programming • immutable collections • Spark is a Scala DSL 19
- 20. 47deg.com Spark Application Language Choices - 1 • 71% Scala, 58% Python • Code length similar • Scala faster for RDDs (no need to move serialized data between JVM and Python) • Scala is generally faster • Scala provides strong typing • Compile time checks • Code is easier to understand • Types help when writing and maintaining code 20
- 21. 47deg.com Spark Application Language Choices- 2 • In addition to Scala and Python also supports Java and R • If you want to build scalable high performance production code based on Spark • R by itself is too specialized • Python is too slow • Java is tedious to write and maintain • Scala is “just right” 21
- 22. 47deg.com Transformations: Parallel versus Serial • Scala has operations that traverse sequence elements in serial order: foldLeft, scanLeft • Spark RDDs can only process sequence elements in parallel • Enables full use of multiple works with multiple cores • Serial operations like foldLeft would be slow compared to current parallel operations • But there are cases where foldLeft or scanLeft is really needed • Example: time series where early event can affect later event • Example: Sherlock, word count by story • So should Spark add foldLeft or similar operators? 22
- 23. 47deg.com Sample Spark Internal Code • Lets examine the internal code in Spark • Spark Context • RDD 23
- 24. Spark Impact on Scala 24
- 25. 47deg.com Pair RDD’s for Scala 25 Scala Sequence Key Value Unique Set Map Duplicate Seq ??? Odersky Spark Sequence Key Value Unique Duplicate RDD Pair RDD .distinct
- 26. 47deg.com Lazy Evaluation • Strict evaluation (Scala) • Each transformation is evaluated as it is seen • Easy to understand and debug • Lazy evaluation (Spark) • Transformations are collected into a linage graph • Evaluated only when a final action is applied • Allows cross transformation optimization 26
- 27. 47deg.com Lazy Evaluation in Scala • Scala has Stream (a kind of sequence) • Elements are evaluated in order as needed • Allows infinite collections • But no cross transform optimization • Lazy Spark evaluation does all elements at once • Enables parallelism • Scala may add lazy versions of its collections that are more like Spark (Odersky) 27
- 28. 47deg.com © Copyright 2015 47 Degrees Spark Clusters - Serialization 28 Your Spark App Spark Context Cluster Manager Worker Node Executer Task Task Spark Driver Worker Node Executer Task Task ... Static Code Jar File Dynamic Code Closures
- 29. 47deg.com Dynamic Code Serialization • Depends on values known at run-time • Often a function passed to Spark transformations such as map, filter, and flatMap • Sent from driver to workers • Serialized on driver to byte array • Deserialized on each work 29
- 30. 47deg.com Serialization of Closures • Often include more than needed (or expected) • Demo • Scala may become smarter about including less in closure serialization (Odersky) 30
- 31. 47deg.com Datasets • DataFrames are becoming the focus of Spark optimization • DataFrames are dynamically typed • Scala API would be nicer if there was static typing • Datasets (new experimental in Spark 1.6) attempt to solve this • Sample Code • It would be nice if DataBricks and Typesafe could work together to produce a better solution 31
- 32. Scala and Spark in Seattle 32
- 33. 47deg.com Scala and Spark in Seattle • Seattle Meetups • Scala at the Sea Meetup (over 1000 members) http://www.meetup.com/Seattle-Scala-User-Group/ • Seattle Spark Meetup (over 1400 members) http://www.meetup.com/Seattle-Spark-Meetup/ • 47 DegreesTraining (Seattle and Worldwide) http://www.47deg.com/events#training • Typesafe Scala Training: Scala, Akka • Spark Training: Programming Spark with Scala • UW Scala Professional Certificate Program http://www.pce.uw.edu/certificates/scala-functional-reactive-programming.html 33
- 34. Summary 34
- 35. 47deg.com Summary • Scala and Spark are great technologies for big data applications • Both are functional • Both have immutable Data • Both can be used to build very large systems • Typesafe and Databricks evolving relationship • Databricks is a consumer of Typesafe components • Typesafe now supports Spark in addition to the Scala compiler and other Scala components • They are working together to evolve toward ever more coordinated and integrated ecosystem for big data • Martin Odersky - Spark — The Ultimate Scala Collections 35
- 36. Questions 36