Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.

1. Apache Spark
Stream Programming and Distributed Data Processing
Habib Ahmed Bhutto
Senior Software Engineer
iConnect360

2. Outline
• What’s Spark
• Why Spark
• Fundamental concepts
• Cluster Deployment
• Spark Streaming
• Application Development
• Deployment
• Application Monitoring
• Debugging

3. What’s Spark
• Fast and speedy
• General (purpose) engine
• For large-scale data processing
• In memory processing
• Built at AMPLab, University of California,
Berkeley as sub-project of Hadoop
• Now it’s Apache’s

4. Why Spark
• Speed
• Ease of use
• Generality
• Runs everywhere
(Hadoop, Mesos, standalone or in cloud)
• Fault Tolerance
• Integration
• Deployment

5. Fundamental Concepts
• What exactly it does
Hadoop execution flow
Spark execution flow

6. Fundamental Concepts
• How exactly it does

7. Fundamental Concepts
• Resilient Distributed Dataset (RDD)
– Abstraction
– Immutable
– Partitioned collection
– Operated on in parallel
• RDD Operations
– Actions
– Transformations
• Spark Context

8. Fundamental Concepts
• Driver Program
• Cluster Manager
• Worker Node
• Executer
• Job
• Stage
• Task
• Application Jar
• Deploy Mode

9. Cluster Deployment
• Standalone
• Amazon EC2
• Apache Mesos
• Hadoop Yarn

10. Cluster Deployment
• Master page to monitor your cluster
– http://<server-url>:8080

11. Spark Streaming
• How it works

12. Spark Streaming
• How it works internally

13. Spark Streaming

14. Spark Streaming
• Discretised Streams
– Abstraction
– Continuous Stream
– Input data/ processed data
– Series of RDDs

15. Spark Streaming
• Any operation applied on a DStream translates
to operations on the underlying RDDs

16. Spark Streaming
• Window Operations
• Output Operations
• DataFrame and SQL Operations
– DataFrame is abstraction that can act as
distributed SQL query engine.

17. Application Development
• Spark-Shell
– Code in Scala with instant execution

18. Application Development
• Self-Contained Applications
– Dependencies /Linking Libraries

19. Application Development
• Self-Contained Applications
– A simple app

20. Application Development
• Self-Contained Applications
– Packaging
– Don’t forget app dependencies

21. Deployment
• That’s how you deploy

22. Application Monitoring
• monitor your app
– http://<driver-node>:4040

23. Application Monitor
• History Server
– Enable and Start History Server http://<server-url>:18080

24. Application Monitor
• History Server
– Enable and Start History Server http://<server-url>:18080

25. Debugging
• Remote debugging
– Enable Remote debugging
– Must be running on local[*]

26. Running on Yarn
• Why to run on Yarn?
– Cluster resources
– Schedulers
– Security

27. Running on Yarn
• Standalone

28. Running on Yarn
• Yarn Architecture
– Resource Manager
– Node Manager
– Application Master
– Container

29. Running on Yarn
• Yarn Client Mode

30. Running on Yarn
• Yarn Cluster Mode

31. Running on Yarn
• Standalone vs Spark on Yarn

32. References
[1] Apache Spark official site http://spark.apache.org/
[2] Introduction to Spark http://www.slideshare.net/rahuldausa/introduction-to-apache-spark-
39638645
[3] Running Spark on Yarn http://badrit.com/blog/2015/2/29/running-spark-on-
yarn#.VnEQub9eeaq
[4] Debugging Apache Spark Jobs http://danosipov.com/?p=779
[5] Habib’s brain

33. A Big Thank You
Spark it up
You got questions?