Alexander Kolb – Flink. Yet another Streaming Framework?
•
2 likes•8,163 views
The document evaluates and compares several streaming frameworks, including SQLStream, Pulsar, SPQR, Apache Spark, and Apache Flink. It assesses the frameworks based on usability, functionality, architecture, support, and non-functional requirements. For each framework, it provides information on architectural diagrams, window aggregation examples, and scores the frameworks in various categories. It concludes that Apache Spark and Apache Flink received the highest overall scores based on the evaluation.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Alexander Kolb – Flink. Yet another Streaming Framework?
Similar to Alexander Kolb – Flink. Yet another Streaming Framework? (20)
More from Flink Forward
More from Flink Forward (20)
Recently uploaded
Recently uploaded (20)
Alexander Kolb – Flink. Yet another Streaming Framework?
- 1. Alexander Kolb, Otto Group BI Hamburg, Germany, 2015 Flink, Yet another Streaming Framework?
- 2. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Alexander Kolb otto group BI @lofifnc
- 3. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Introduction
- 5. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Approach 5 Rating based on: - Research - Hands-on
- 6. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Use-case
- 7. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Use-case 7
- 8. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Frameworks
- 9. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Frameworks 9 SQLStream Pulsar SPQR Apache Spark Apache Flink
- 10. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 SQLStream
- 11. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 SQLStream 11 Architecture source: sqlstream.com
- 12. SQLStream
- 13. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 SQLStream 13 Window Aggregation 1 SELECT STREAM 2 B.pagetype, 3 B.ecid, 4 productid, 5 SUM(QUANTITY) AS "views" 6 FROM VIEWS AS B 7 GROUP BY FLOOR((B.ROWTIME - TIMESTAMP '1970-01-01 00:00:00') 8 MINUTE / 5 to MINUTE), 9 PRODUCTID, PAGETYPE, ECID;
- 14. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Pulsar
- 15. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Pulsar 15 Architecture source: github.com/pulsarIO
- 16. Pulsar
- 17. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Pulsar 17 Window Aggregation 1 create context MCContext start @now end after 60 seconds; 2 3 context MCContext 4 insert into ViewAgg select count(*) as views, prid 5 from PageView group by prid output snapshot when terminated;
- 18. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 SPQR
- 20. SPQR
- 21. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 SPQR 21 Window Aggregation 1 select productid, ecid, sum(quantity) 2 from views.win:time_batch(5 min) 3 group by productid, ecid
- 22. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Apache Spark
- 23. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Apache Spark 23 Architecture source: spark.apache.org
- 24. ApacheSpark
- 25. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Apache Spark 25 Aggregation 1 val aggViews = views.reduceByKeyAndWindow({ 2 case ((pageType, ecid, sum, price),(_,_,quant,_)) => 3 (pageType, ecid, sum + quant, price) 4 }, Minutes(5), Minutes(5))
- 26. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Apache Flink 26 Architecture source: spark.apache.org source: flink.apache.org
- 27. ApacheFlink
- 28. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Apache Flink 28 Window Aggregation 1 val aggViews = input.window(Time.of(5, TimeUnit.MINUTES)) 2 .groupBy(“productId”).sum(“quantity”);
- 29. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Result Evaluation
- 30. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Summary 30 Use-case Topic Unit Framework Pulsar.io SQLStre am SPQR Flink Spark Time for building the stream hours 40 35+ (POC) 8+ (POC) 13 4 Time for adding missing connector hours 3 8 1 3 0.5 Points 3.14 2.06 3.44 4.16 4.45
- 31. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 List of Rating Aspects 31 DSL/DDL/UI for creating Pipelines / Required know-how to define new Pipelines / Project documentation /Workflow / Testing Workflows / hot deploying / redeploying of pipelines / dynamic topology changes / Monitoring / Deployment / Dashboard for data visualization / Ease of defining udf's / Merge / Sum / Count / Min/max/avg / Aggregate / Transform / Parsing (xml/json/csv) / Group-by / Join / Ease of defining new connectors / Kafka / WebSocket / JDBC / JMS / HDFS / File / Effort for cluster deployment / Configuration effort / Supports YARN / Supports Mesos / Scalability / Resilience /Predefined communication framework / Dependencies / Flexibility / Expandability / Buffering/Pressure handling / Partitioning/Parallelism / Strategy for Partitioning/ Parallelism? / Ordering / Guarantees / State-Management / Fault tolerance / Licensing model / Professional support available / Community Activity / License / Maturity / Manageable code-base / Community Size
- 32. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Summary 32 Topic Framework weight SQLStream Pulsar.io SPQR Spark Flink Usability 2.6 3 2.2 3.6 3.9 15 Flexibility 2.5 4 3 1.5 1.5 8 User-Interface 3.3 1.8 1 3 3.3 6 Operators 4.8 4.3 4.3 3.9 4.7 10 Connectors 4.7 1.9 1.9 2.4 2.6 6 Deployment 2.4 3.2 2.8 3.8 3.7 10 Architecture/ Concepts 2 3.2 3.3 4 4 12 Functional Requirements 2 3.2 2.4 4 4 14 Costs 0 5 5 5 5 5 Service/ Support 2.5 0.5 1 4.5 3.5 8 Project 1.8 3.3 2.5 4.3 3.8 6 Sum 2.58 3.11 2.74 3.74 3.80 100
- 34. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 Final Scores 34 SQLStream Pulsar.io SPQR Spark Flink Evaluation 2.58 3.11 2.74 3.74 3.8 Use-case 2.06 3.14 3.44 4.45 4.16
- 35. Evaluation Streaming Frameworks Alexander Kolb, Otto Group BI, Hamburg, Germany, 2015 ottogroup.comWE ARE HIRING!