舊金山 Spark Summit 2016 西遊記分享

1. Spark Summit 2016 @San Francisco
Stana He

2. • gitter
https://gitter.im/hubertfc/SparkHsinchu
• Gitter app https://gitter.im/apps
• meetup
https://www.meetup.com/Apache-Spark-Hsinchu/

3. Who am I ?
• Stana He
• Is-land

4. Agenda
• Something about Apache Spark
• Enterprise use case

5. What is Apache Spark?
• Open source cluster computing framework.
• Developed at the UC, Berkeley's AMPLab.
• Donated to the Apache Software Foundation.

6. Benefits of Apache Spark
• Speed
- 100x faster than Hadoop for large scale
data processing.
• Ease of Use
- Easy-to-use APIs.
• Unified Engine
- Packaged with higher-level
libraries,including streaming data,SQL
queries,machine learning and graph
processing.

7. What’s New in 2.0 ?
• Structured API improvements
- SQL, DataFrames, Datasets
• Structured Streaming
• MLlib model export
• MLlib R bindings
• SQL 2003 support
• Scala 2.12 support

8. What’s New in 2.0 ?
• Whole-stage code generation
- Fuse across multiple operators
• Optimized input / output
- Apache Parquet + built-in cache
reference:http://www.slideshare.net/databricks/spark-summit-san-francisco-2016-matei-zaharia-keynote-apache-spark-20

9. Enterprise use case

11. Winning the game with Spark!

12. Unfortunately, it doesn’t!

13. reference:http://www.slideshare.net/SparkSummit/video-games-at-scale-improving-the-gaming-experience-with-apache-spark

14. Players and Data
• 67+ million
monthly active players
• 500+ billion
data points per day
• 26 petabytes
data collected since beta

15. What does Spark do ?
• Spark SQL
Data exploration and reporting
• Spark Streaming
Network performance
• Spark MLlib
Recommendation system

16. Spark SQL
-Data exploration and reporting

17. Performance
reference:http://www.slideshare.net/SparkSummit/video-games-at-scale-improving-the-gaming-experience-with-apache-spark

18. Spark Streaming
-Network performance

19. Build network
Riot Direct
reference:http://www.slideshare.net/SparkSummit/video-games-at-scale-improving-the-gaming-experience-with-apache-spark

20. Normal Network Model
reference:http://www.slideshare.net/SparkSummit/video-games-at-scale-improving-the-gaming-experience-with-apache-spark

21. Detect model
reference:http://www.slideshare.net/SparkSummit/video-games-at-scale-improving-the-gaming-experience-with-apache-spark

22. Another detect model
reference:http://www.slideshare.net/SparkSummit/video-games-at-scale-improving-the-gaming-experience-with-apache-spark

23. Model Building/
Evaluation
HIVE
(stores aggregated data)
Kafka
Consume/
Aggregate
Alerts
Spark
Elasticsearch
Dashboards
reference:http://www.slideshare.net/SparkSummit/video-games-at-scale-improving-the-gaming-experience-with-apache-spark

24. Spark MLlib
-Recommendation system

25. reference:http://www.slideshare.net/SparkSummit/video-games-at-scale-improving-the-gaming-experience-with-apache-spark

26. reference:http://www.slideshare.net/SparkSummit/video-games-at-scale-improving-the-gaming-experience-with-apache-spark

27. Modeling/
Evaluation
HIVE
Explore/
Feature
engineering
Recommendation
Game Server
Data
Feature
SparkSQL MLlib
Spark
reference:http://www.slideshare.net/SparkSummit/video-games-at-scale-improving-the-gaming-experience-with-apache-spark

28. Q & A

29. Spark Summit
• https://spark-summit.org/2016/
• Slides and video

30. Spark Cookbook-
• Ch1. Getting Started with Apache Spark (Chunhung Huang) (4 )
• Ch2. Developing Applications with Spark ( )
• Spark RDD (Allen )
• Ch3. External Data Sources ( ) (8 )
• Ch4. Spark SQL ( )
• Ch5. Spark Streaming ( )
• Ch6. Getting Started with Machine Learning Using MLlib ( )
• Ch7. Supervised Learning with MLlib - Regression (Dean Du)
• Ch8. Supervised Learning with MLlib - Classification ( )
• Ch9. Unsupervised Learning with MLlib (Vito)
• Ch10.Recommender System (Leorick)
• Ch11.Graph Processing using GraphX ( )
• Ch12.Optimizations and Performance Tuning ( )