Stream vs Batch Processing: Divide, Distribute and Conquer Data
•
3 gostaram•539 visualizações
This document discusses stream processing versus batch processing. It notes that stream processing involves data that is in motion and processed in real-time using streaming platforms and directed acyclic graphs. Batch processing involves data at rest that is processed through queries on the full dataset. The document also discusses challenges of stream processing like out-of-order and late data, and how windowing can provide a finite view of infinite data streams.
Recomendados
Recomendados
Mais conteúdo relacionado
Semelhante a Stream vs Batch Processing: Divide, Distribute and Conquer Data
Semelhante a Stream vs Batch Processing: Divide, Distribute and Conquer Data (20)
Mais de Viktor Gamov
Mais de Viktor Gamov (14)
Último
Último (17)
Stream vs Batch Processing: Divide, Distribute and Conquer Data
- 1. @gamussa @confluentinc @thephillyjug Divide, Distribute and Conquer: Stream v. Batch
- 3. Who am I?
- 6. Solutions Architect Developer Advocate @gamussa in internetz Who am I?
- 7. Solutions Architect Developer Advocate @gamussa in internetz Hey you, yes, you, go follow me in twitter © Who am I?
- 9. @gamussa @confluentinc @thephillyjug BATCH PROCESSING Data at rest
- 10. @gamussa @confluentinc @thephillyjug Data and Queries Origin and processing
- 14. @gamussa @confluentinc @thephillyjug ✓ … inherently immutable Data… ✓ … time-based
- 16. @gamussa @confluentinc @thephillyjug Processing is a query
- 17. @gamussa @confluentinc @thephillyjug Processing is a query Function on full data set
- 18. @gamussa @confluentinc @thephillyjug Processing is a query Function on full data set Projection
- 19. @gamussa @confluentinc @thephillyjug Processing is a query Function on full data set Projection Aggregations
- 20. @gamussa @confluentinc @thephillyjug Processing is a query Function on full data set Projection Aggregations Joins
- 21. SELECT user_vote, count(*) FROM AccessLog WHERE event_date BETWEEN"04/07/2017" AND "04/07/2017" GROUP BY user_vote;
- 22. SELECT user_vote, count(*) FROM AccessLog WHERE event_date BETWEEN "04/7/2017" AND "04/08/2017" GROUP BY user_vote;
- 23. SELECT user_vote, count(*) FROM AccessLog WHERE event_date BETWEEN"04/07/2017" AND "04/08/2007" GROUP BY user_vote;
- 24. @gamussa @confluentinc @thephillyjug Lambda architecture origins http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html
- 26. @gamussa @confluentinc @thephillyjug Lambda Architecture
- 28. @gamussa @confluentinc @thephillyjug TFW Trying to explain modern big data landscape
- 29. @gamussa @confluentinc @thephillyjug Precomputed Results http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html
- 30. @gamussa @confluentinc @thephillyjug Batch Process http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html
- 31. @gamussa @confluentinc @thephillyjug STREAM PROCESSING Data is motion
- 32. @gamussa @confluentinc @thephillyjug Streaming Platform
- 33. @gamussa @confluentinc @thephillyjug Streaming Platform
- 34. @gamussa @confluentinc @thephillyjug Directed Acyclic Graph
- 37. @gamussa @confluentinc @thephillyjug Interesting cases Before You Go
- 38. I FOUND YOUR LACK OF FAULT TOLERANCE DISTURBING
- 39. Data is too important to store it in one computer
- 44. @gamussa @confluentinc @thephillyjug How to process «infinite» data?
- 46. @gamussa @confluentinc @thephillyjug Time model Different use cases time semantics
- 47. @gamussa @confluentinc @thephillyjug Time model Different use cases time semantics Majority of use cases require event- time semantics
- 48. @gamussa @confluentinc @thephillyjug Time model Different use cases time semantics Majority of use cases require event- time semantics Other use cases may require processing-time or special variants like ingestion-time
- 53. @gamussa @confluentinc @thephillyjug Windowing Windowing is an operation that groups events
- 55. @gamussa @confluentinc @thephillyjug Windowing Input data, where colors represent different users events Rectangles denote different event-time windows processing-time event-time windowing alice bob dave
- 56. @gamussa @confluentinc @thephillyjug Windowing Windowing is an operation that groups events Most commonly needed: time windows, session windows Examples: ✗Real-time monitoring: 5-minute averages ✗Reader behavior on a website: user browsing sessions
- 58. @gamussa @confluentinc @thephillyjug Out-of-order and late data Is very common in practice, not a rare corner case ✗Related to time model discussion
- 59. @gamussa @confluentinc @thephillyjug Out-of-order and late data
- 60. @gamussa @confluentinc @thephillyjug Out-of-order and late data Users with mobile phones enter airplane, lose Internet connectivity
- 61. @gamussa @confluentinc @thephillyjug Out-of-order and late data Users with mobile phones enter airplane, lose Internet connectivity Emails are being written during the 10h flight
- 62. @gamussa @confluentinc @thephillyjug Out-of-order and late data Users with mobile phones enter airplane, lose Internet connectivity Emails are being written during the 10h flight Internet connectivity is restored, phones will send queued emails now
- 63. @gamussa @confluentinc @thephillyjug Stream Processing: results
- 64. @gamussa @confluentinc @thephillyjug Stream Processing: results • Yes, it’s possible to get computation results in real time
- 65. @gamussa @confluentinc @thephillyjug Stream Processing: results • Yes, it’s possible to get computation results in real time • Windows – finite view of infinite data • Based on temporal characteristics of the evet
- 66. @gamussa @confluentinc @thephillyjug Stream Processing: results • Yes, it’s possible to get computation results in real time • Windows – finite view of infinite data • Based on temporal characteristics of the evet • Late event processing • You choose how long to wait