A Benchmark Suite for Distributed Stream Processing Systems
•
1 gostou•338 visualizações
This document presents a benchmark suite for evaluating distributed stream processing systems. It discusses the need to process large volumes of real-time data and examines existing stream processing benchmarks. The proposed benchmark suite contains several real-world applications, defines metrics like latency and throughput, and provides results comparing the performance of Storm and Spark on the applications. The conclusion is that the benchmark can help compare different stream processing platforms for specific application types. Future work may include additional applications and platforms.
Recomendados
Mais conteúdo relacionado
Semelhante a A Benchmark Suite for Distributed Stream Processing Systems
Semelhante a A Benchmark Suite for Distributed Stream Processing Systems (20)
Mais de Maycon Viana Bordin
Mais de Maycon Viana Bordin (20)
Último
Último (20)
A Benchmark Suite for Distributed Stream Processing Systems
- 1. A Benchmark Suite for Distributed Stream Processing Systems Maycon Viana Bordin Claudio Geyer Advisor April, 2017 1
- 2. 2
- 3. HUGE amounts of data are being generated in real-time 3
- 4. 4
- 5. 500M tweets are sent per day 5
- 6. 6
- 7. 4.75B shares 4.5B likes 420M status updates 300M photos EVERY DAY. 7
- 8. 8
- 9. They need to process… 9
- 10. They need to process… large volumes of data 10
- 11. They need to process… large volumes of data in real-time 11
- 12. They need to process… large volumes of data in real-time continuously 12
- 13. They need to process… large volumes of data in real-time continuously producing actionable information 13
- 14. 14
- 16. 16
- 17. 17
- 18. B 18
- 19. B 19
- 20. 20
- 21. Data Stream 21
- 22. B 22
- 23. B 23
- 24. B 1234567 24
- 25. Data from the stream source may or may not be structured 25
- 26. The amount of data is usually unbounded in size 26
- 27. The input rate is variable and typically unpredictable 27
- 28. There are many platforms on the market 28
- 29. Problem: How to know which platform is better for an specific type of application? 29
- 30. Problem: Current stream processing benchmarks are composed mostly of synthetic applications. 30
- 31. Problem: Benchmarks for other Big Data platforms use more real world applications, e.g. BigDataBench and HiBench. 31
- 32. Goals: 32
- 34. 34
- 36. Linear Road Benchmark [Ara04] • • • 36
- 43. Comparison 43
- 44. 44
- 46. 46
- 47. 47
- 48. 48
- 49. 49
- 50. API • • • • 50
- 51. Metrics • • • 51
- 54. 54
- 55. 55
- 56. 56
- 57. • • • • 57
- 58. 58
- 59. 59
- 60. 60
- 61. 61
- 62. 62
- 63. 63
- 64. 64
- 65. 65
- 66. 66
- 68. 68
- 69. 69
- 70. 𝐿𝑎𝑡𝑒𝑛𝑐𝑦 = 𝑇𝑒𝑛𝑑 − 𝑇𝑒𝑛𝑑 70
- 71. 71
- 72. 72
- 73. 73
- 74. 𝑇ℎ𝑟𝑜𝑢𝑔ℎ𝑝𝑢𝑡 = 𝑁𝑢𝑚. 𝑃𝑟𝑜𝑐𝑒𝑠𝑠𝑒𝑑 𝑇𝑢𝑝𝑙𝑒𝑠 𝑅𝑢𝑛𝑡𝑖𝑚𝑒 74
- 75. 75
- 76. 76
- 77. 77
- 78. Comparison with the other Benchmarks 78
- 79. 79
- 80. 80
- 82. 82
- 83. • • • • 83
- 84. • • • 84
- 85. 85
- 86. 86
- 87. 87
- 88. n1_x1_x5_x6_x3 n1_x1_x2_x1_x4_x2 n1_x4_x2_x2 n1_x2_x5_x6_x3 n1_x2_x2_x1_x4_x2 n4_x2_x2_x2 n1_x3_x5_x6_x3 n1_x4_x2_x1_x4_x2 n4_x8_x2_x2 n2_x1_x5_x6_x3 n1_x8_x2_x1_x4_x2 n2_x2_x5_x6_x3 n2_x1_x2_x1_x4_x2 n2_x3_x5_x6_x3 n2_x2_x2_x1_x4_x2 n4_x1_x5_x6_x3 n2_x4_x2_x1_x4_x2 n4_x2_x5_x6_x3 n2_x8_x2_x1_x4_x2 n4_x3_x5_x6_x3 n4_x1_x2_x1_x4_x2 n8_x1_x5_x6_x3 n4_x2_x2_x1_x4_x2 n8_x2_x5_x6_x3 n4_x4_x2_x1_x4_x2 n8_x3_x5_x6_x3 n4_x8_x2_x1_x4_x2 n8_x1_x2_x1_x4_x2 n8_x2_x2_x1_x4_x2 n8_x4_x2_x1_x4_x2 n8_x8_x2_x1_x4_x2 88
- 90. 90
- 91. 91
- 92. 92
- 93. 93
- 96. 96
- 97. 97
- 98. 98
- 99. 99
- 101. 101
- 102. 102
- 103. 103
- 106. 106
- 107. 107
- 108. 108
- 109. 109
- 111. 111
- 112. 112
- 113. 113
- 115. 115
- 116. 116
- 117. 117
- 118. 118
- 119. 119
- 120. 120
- 121. 121
- 122. Conclusion 122
- 123. • • • 123
- 124. • • 124
- 125. Future Work 125
- 126. • • • • 126
- 127. • • Publications 127
- 128. A Benchmark Suite for Distributed Stream Processing Systems Maycon Viana Bordin Claudio Geyer Advisor April, 2017 128
- 129. 129
- 130. 130
- 131. 131
- 132. 132
- 133. 133
- 134. 134