The speaker dongeforever(RocketMQ Committer/PMC Member) explained the biggest challenges of BigData and shown how Kafka and RocketMQ handle them.

1. Performance and
BigData dongeforever@apache.org

2. dongeforever
Apache RocketMQ PMC/Committer.
Interested in algorithms and
performance.
Now spending more time in open
source, including MQ and BigData

3. Content Map
❖ Challenge of Big Data Stream
❖ Kafka Overview
❖ Batch Through
❖ Compression Through
❖ Structural Compression
❖ BigData Eco-system

4. Challenge of Big Data Stream

5. Challenge of Big Data Stream
❖ High Throughput —— Million TPS
❖ IO BandWidth —— Network & Disk
❖ Storage Cost

6. Kafka Overview
❖ Open sourced in early 2011, graduate from the Apache
Incubator on 23 October 2012
❖ Log Aggregation, Messaging, Stream Processing, Event
Sourcing
❖ Widely used in BigData processing, integrate with
Storm, Spark, Flink, Samza, Hadoop, Flume, etc.

7. Kafka Domain Model

8. Kafka Store Structure

9. Content Map
❖ Challenge of Big Data Stream
❖ Kafka Overview
❖ Batch Through
❖ Compression Through
❖ Structural Compression
❖ BigData Eco-system

10. Batch Through

11. Kafka Producer TPS vs Msg Size

12. Proxy Producer TPS vs Msg Size

13. RocketMQ Batch
❖ send(Collection<Message> msgs)
❖ Atomic
❖ Get 150w TPS
❖ https://rocketmq.apache.org/docs/batch-example/

14. Content Map
❖ Challenge of Big Data Stream
❖ Kafka Overview
❖ Batch Through
❖ Compression Through
❖ Structural Compression
❖ BigData Eco-system

15. Compression Through
• Clients handle the Compress/DeCompress
• Only Need One Operation in Broker

16. Compression Through
• For log collection, about 5~10 compression ratio

17. Content Map
❖ Challenge of Big Data Stream
❖ Kafka Overview
❖ Batch Through
❖ Compression Through
❖ Structural Compression
❖ BigData Eco-system

18. Structural Compression

19. Structural Compression

20. Structural Compression
❖ Assume each msg has N bytes, each batch has B msgs
❖ Size of 0.10: (34 + N) * B
❖ Size of 0.11: 61 + (7 + N) * B
❖ For N <= 100, save storage upper to 20%~50%

21. Content Map
❖ Challenge of Big Data Stream
❖ Kafka Overview
❖ Batch Through
❖ Compression Through
❖ Structural Compression
❖ BigData Eco-system

22. External ecosystem

23. External ecosystem
https://github.com/apache/rocketmq-externals