Structured Streaming in Spark
•2 gostaram•934 visualizações
Denunciar
Compartilhar
Baixar para ler offline
Apache Spark has been gaining steam, with rapidity, both in the headlines and in real-world adoption. Spark was developed in 2009, and open sourced in 2010. Since then, it has grown to become one of the largest open source communities in big data with over 200 contributors from more than 50 organizations. This open source analytics engine stands out for its ability to process large volumes of data significantly faster than contemporaries such as MapReduce, primarily owing to in-memory storage of data on its own processing framework. That being said, one of the top real-world industry use cases for Apache Spark is its ability to process ‘streaming data‘.
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados(20)
Similar a Structured Streaming in Spark
Similar a Structured Streaming in Spark(20)
Mais de Digital Vidya
Mais de Digital Vidya(20)
Último
Último(20)
Structured Streaming in Spark
- 2. Structured Streaming in Spark Vikram Agrawal Qubole
- 3. About Me ● Pursued Computer Science and Engineering from IIT Delhi ● Co-founded a web conferencing solution company before joining Qubole ● In last 5 years at Qubole, I wore multiple hats and worked across stacks to provide big-data solutions over cloud ● Currently leading the Streaming Team At Qubole
- 4. Who should watch this? ● Big Data Engineer (DevOps, Architect, Software, Engineer, Admin) ● Data Platform Manager ● Big Data Enthusiast (Consultant, Executive, Data User, Analyst)
- 5. How is streaming used in production? ● Identifying sessions based on user behavior from real time activity streams ● Anomaly and fraud detection: running ML predictions on data streaming in to keep the model updated continuously as new data comes in ● Time-based window aggregations: using window functions to do associative aggregations and run real time stats
- 8. Streaming Paradigm ● Stream In Stream out ○ Low Latency - How Low? ○ Complexity of Analytics ○ Volume - How high? ● Stream In Batch out ○ No Tight Latency Constraint ○ Higher Ingestion Rate ○ Aggregation/Data or Schema Transformation/Data Enrichment ○ Downstream ETL Operation
- 9. Why use Spark Streaming ● No ultra low Latency requirement ○ Processing time of few secs is acceptable ● Scalable and Mature Processing engine ● Higher Level API abstraction ○ Ease of Code Reuse from Batch jobs ○ Simple and Modular ● Vibrant Community ○ Active Development on new features
- 11. Structured Streaming - under the hood ● Abstractions of Repeated Queries ○ Data Streams as unbounded Table ○ Streaming query is a batch- like operation on this table
- 12. Structured Streaming - under the hood ● Query Planning & Execution ○ In Batch Execution, Planner creates code & memory optimized execution plan ○ For Streaming Query, Planner convert streaming Logical plans to a series of incremental execution plan to process next chunk of data DataFrame Logical Plan Planner Execution Plan Planner Incremental Execution 1 Incremental Execution 2 Incremental Execution 3
- 13. Programming Paradigm Start with Spark Session Specify Data Source, schema and other options (create input df) Write your incremental query to generate output Specify Data Sink and other options to export your data Val S= SparkSession.builder.appName("kafka streaming Example").getOrCreate() val ds = S.readStream.format("kafka") .option("kafka.bootstrap.servers", brokers) option("subscribe", topics).load().selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)").as[(String, String) val c= ds.groupBy("value").count() c.writeStream.queryName("aggregates").format(" memory").outputMode("complete").start()
- 14. Productionizing Streaming Application ● Monitoring ○ Throughput ○ Latency ○ Time Lag ● Fault Tolerance ○ Checkpointing ○ Exactly Once or At Least Once
- 15. Q&A