This document discusses challenges with writing streaming data from Kafka to Parquet files stored in HDFS. It evaluates several approaches: 1) windowing, which works but uses too much memory; 2) bucketing data into time-based files, which works for failures but requires modifying Flink; 3) closing files at checkpoints, which was later supported in Flink; and 4) hourly batch jobs, which avoids streaming complexity but limits real-time use. The conclusion is that streaming solutions are not trivial and it may be better to use a database or different tool instead of files for this use case. Supporting both real-time and batch processing is challenging.

1. Journeys from Kafka to
Parquet
DataWorks Summit, Barcelona 2019
21/03/2019

2. Largest e-commerce
platform in NL and BE

3. Gábor Hermann
• data engineer @ bol.com
• Previously at research institute
• MTA SZTAKI, Budapest
• Now measuring, recommendations

4. This talk
• The problem: sinking data from stream to files
• Solution 1. windowing
• Solution 2. bucketing sink
• Solution 3. closing files at checkpoints
• Solution 4. daily/hourly batch job
• Conclusion

5. Problem
click data stream
(Kafka)

6. Problem
click data stream
(Kafka)
10,000 event/sec

7. Problem
HDFS
click data stream
(Kafka)
10,000 event/sec
(immutable)
files

8. Problem
HDFS
click data stream
(Kafka)
10,000 event/sec
(immutable)
files
?

9. Batch vs streaming

10. Batch vs streaming
batch

11. Batch vs streaming
batch streaming

12. Problem

13. Problem
Expectation

14. Problem
Expectation Reality

15. Requirements

16. Requirements
• Scalable solution
• 10,000 message/sec

17. Requirements
• Scalable solution
• 10,000 message/sec
• Exactly-once
• Data consumers should not deduplicate
!"DISTINCT

18. Requirements
• Scalable solution
• 10,000 message/sec
• Exactly-once
• Data consumers should not deduplicate
• Files in event-time
• Consumers should not worry about late events
!"late events

19. Requirements
• Scalable solution
• 10,000 message/sec
• Exactly-once
• Data consumers should not deduplicate
• Files in event-time
• Consumers should not worry about late events
• Columnar format (Parquet)
• Optimize for reading, not for writing
!"
slow loading

20. Columnar format (Parquet)
CUSTOMER ID VISITED
PRODUCT
PRODUCT
CATEGORY
45182584 370000004333 Books
45182584 300000053536 Games
11538222 857334358658 Electronics
79245368 370000004333 Books
11538222 370000004333 Books
11538222 942000033234 Electronics
78438133 370000004333 Books

21. Columnar format (Parquet)
CUSTOMER ID VISITED
PRODUCT
PRODUCT
CATEGORY
45182584 370000004333 Books
45182584 300000053536 Games
11538222 857334358658 Electronics
79245368 370000004333 Books
11538222 370000004333 Books
11538222 942000033234 Electronics
78438133 370000004333 Books
Row-oriented

22. Columnar format (Parquet)
CUSTOMER ID VISITED
PRODUCT
PRODUCT
CATEGORY
45182584 370000004333 Books
45182584 300000053536 Games
11538222 857334358658 Electronics
79245368 370000004333 Books
11538222 370000004333 Books
11538222 942000033234 Electronics
78438133 370000004333 Books
Row-oriented

23. Columnar format (Parquet)
CUSTOMER ID VISITED
PRODUCT
PRODUCT
CATEGORY
45182584 370000004333 Books
45182584 300000053536 Games
11538222 857334358658 Electronics
79245368 370000004333 Books
11538222 370000004333 Books
11538222 942000033234 Electronics
78438133 370000004333 Books
Row-oriented Column-oriented
CUSTOMER ID VISITED
PRODUCT
PRODUCT
CATEGORY
45182584 370000004333 Books
45182584 300000053536 Games
11538222 857334358658 Electronics
79245368 370000004333 Books
11538222 370000004333 Books
11538222 942000033234 Electronics
78438133 370000004333 Books

24. Columnar format (Parquet)
CUSTOMER ID VISITED
PRODUCT
PRODUCT
CATEGORY
45182584 370000004333 Books
45182584 300000053536 Games
11538222 857334358658 Electronics
79245368 370000004333 Books
11538222 370000004333 Books
11538222 942000033234 Electronics
78438133 370000004333 Books
CUSTOMER ID VISITED
PRODUCT
PRODUCT
CATEGORY
45182584 370000004333 Books
45182584 300000053536 Games
11538222 857334358658 Electronics
79245368 370000004333 Books
11538222 370000004333 Books
11538222 942000033234 Electronics
78438133 370000004333 Books
Row-oriented Column-oriented

25. Columnar format (Parquet)
CUSTOMER ID VISITED
PRODUCT
PRODUCT
CATEGORY
45182584 370000004333 Books
45182584 300000053536 Games
11538222 857334358658 Electronics
79245368 370000004333 Books
11538222 370000004333 Books
11538222 942000033234 Electronics
78438133 370000004333 Books
Row-oriented
!"slow loading

26. Columnar format (Parquet)
CUSTOMER ID VISITED
PRODUCT
PRODUCT
CATEGORY
45182584 370000004333 Books
45182584 300000053536 Games
11538222 857334358658 Electronics
79245368 370000004333 Books
11538222 370000004333 Books
11538222 942000033234 Electronics
78438133 370000004333 Books
CUSTOMER ID VISITED
PRODUCT
PRODUCT
CATEGORY
45182584 370000004333 Books
45182584 300000053536 Games
11538222 857334358658 Electronics
79245368 370000004333 Books
11538222 370000004333 Books
11538222 942000033234 Electronics
78438133 370000004333 Books
Row-oriented Column-oriented

27. Columnar format (Parquet)
CUSTOMER ID VISITED
PRODUCT
PRODUCT
CATEGORY
45182584 370000004333 Books
45182584 300000053536 Games
11538222 857334358658 Electronics
79245368 370000004333 Books
11538222 370000004333 Books
11538222 942000033234 Electronics
78438133 370000004333 Books
CUSTOMER ID VISITED
PRODUCT
PRODUCT
CATEGORY
45182584 370000004333 Books
45182584 300000053536 Games
11538222 857334358658 Electronics
79245368 370000004333 Books
11538222 370000004333 Books
11538222 942000033234 Electronics
78438133 370000004333 Books
Row-oriented Column-oriented

28. Columnar format (Parquet)
CUSTOMER ID VISITED
PRODUCT
PRODUCT
CATEGORY
45182584 370000004333 Books
45182584 300000053536 Games
11538222 857334358658 Electronics
79245368 370000004333 Books
11538222 370000004333 Books
11538222 942000033234 Electronics
78438133 370000004333 Books
Column-oriented
!"❤
fast loading

29. Requirements
• Scalable solution
• 10 000 message/sec
• Exactly-once
• Data consumers should not deduplicate
• Files in event-time
• Consumers should not worry about late events
• Columnar format (Parquet)
• Optimize for reading, not for writing

30. Requirements
• Scalable solution
• 10 000 message/sec
• Exactly-once
• Data consumers should not deduplicate
• Files in event-time
• Consumers should not worry about late events
• Columnar format (Parquet)
• Optimize for reading, not for writing
Apache Flink?

31. Requirements
• Scalable solution ✅
• 10 000 message/sec
• Exactly-once ✅
• Data consumers should not deduplicate
• Files in event-time ✅
• Consumers should not worry about late events
• Columnar format (Parquet) ✅
• Optimize for reading, not for writing
Apache Flink?

32. Problem
HDFS
click data stream
(Kafka)
10,000 event/sec
(immutable)
files
?

33. Problem
HDFS
click data stream
(Kafka)
10,000 event/sec
(immutable)
files

34. Solution 1.
windowing

35. Windowing
Kafka Flink HDFS
■ 17-00.parquet18:00-18:59
19:00-19:59

36. ■ 17-00.parquet18:00-18:59
19:00-19:59
Windowing
Kafka Flink HDFS

37. ■ 17-00.parquet
■ 18-00.parquet
19:00-19:59
18:00-18:59
Windowing
Kafka Flink HDFS

38. Handling failures?

39. Handling failures?
• Out-of-the-box!

40. Handling failures?
• Out-of-the-box!
• But…

41. Handling failures?
• Out-of-the-box!
• But…
• Too much memory

42. Handling failures?
• Out-of-the-box!
• But…
• Too much memory
Kafka Flink HDFS
■ 16-00.parquet18:00-19:59

43. Handling failures?
• Out-of-the-box!
• But…
• Too much memory
Kafka Flink HDFS
■ 16-00.parquet18:00-19:59

44. Handling failures?
• Out-of-the-box!
• But…
• Too much memory
Kafka Flink HDFS
■ 16-00.parquet18:00-19:59
OUT OF MEMORY

45. Solution 2.
bucketing sink

46. Bucketing sink
• Writes data to file “buckets” based on time

47. Bucketing sink
• Writes data to file “buckets” based on time
Kafka Flink HDFS
■ 17-00.parquet
■ 18-00.parquet
■ 19-00.parquet
18:00-18:59
19:00-19:59

48. Handling failures?
Kafka Flink HDFS
■ 17-00.parquet
■ 18-00.parquet
■ 19-00.parquet
18:00-18:59
19:00-19:59
Step 1

49. Handling failures?
Kafka Flink HDFS
■ 17-00.parquet
■ 18-00.parquet
■ 19-00.parquet
18:00-18:59
19:00-19:59
Step 2 (checkpoint)

50. ■ 17-00.parquet
■ 18-00.parquet
■ 19-00.parquet
18:00-18:59
19:00-19:59
Step 3 (failure)
Handling failures?
Kafka Flink HDFS

51. ■ 17-00.parquet
■ 18-00.parquet
■ 19-00.parquet
18:00-18:59
19:00-19:59
Step 3 (failure)
Handling failures?
Kafka Flink HDFS

52. ■ 17-00.parquet
■ 18-00.parquet
■ 19-00.parquet
18:00-18:59
19:00-19:59
Step 4 (recovery)
Handling failures?
Kafka Flink HDFS

53. But…
• Truncating is not supported on HDFS
• Already fixed (Flink 1.6.0): copy instead of truncate
• Cannot flush Parquet files

54. Solution 3.
closing files at checkpoint

55. ■ 17-00
■ 18-00/
■ part-0.parquet
■ 19-00/
■ part-0.parquet
18:00-18:59
19:00-19:59
Step 1
Closing files at checkpoints
Kafka Flink HDFS

56. Closing files at checkpoints
Kafka Flink HDFS
18:00-18:59
19:00-19:59
■ 17-00
■ 18-00/
■ part-0.parquet
■ part-1.parquet
■ 19-00/
■ part-0.parquet
■ part-1.parquet
Step 2 (checkpoint)

57. 18:00-18:59
19:00-19:59
■ 17-00
■ 18-00/
■ part-0.parquet
■ part-1.parquet
■ 19-00/
■ part-0.parquet
■ part-1.parquet
Step 3 (failure)
Closing files at checkpoints
Kafka Flink HDFS

58. 18:00-18:59
19:00-19:59
■ 17-00
■ 18-00/
■ part-0.parquet
■ part-1.parquet
■ 19-00/
■ part-0.parquet
■ part-1.parquet
Step 3 (failure)
Closing files at checkpoints
Kafka Flink HDFS

59. 18:00-18:59
19:00-19:59
■ 17-00
■ 18-00/
■ part-0.parquet
■ part-1.parquet
■ 19-00/
■ part-0.parquet
■ part-1.parquet
Step 4 (recovery)
Closing files at checkpoints
Kafka Flink HDFS

60. But…
• We need to change Flink bucketing sink code

61. But…
• We need to change Flink bucketing sink code
• Was also fixed in 1.6.0: StreamingFileSink can close files on checkpoints
• Kudos to Flink community!

62. But…
• We need to change Flink bucketing sink code
• Was also fixed in 1.6.0: StreamingFileSink can close files on checkpoints
• Kudos to Flink community!
• A lot of files
• Small files on HDFS is bad

63. Streaming fault-tolerance
WHY YOU SO COMPLICATED?

64. Solution 4.
daily/hourly batch job

65. ■ 17-00.parquet
■ 18-00.parquet
18:00-18:59
Hourly batch job
Kafka Flink HDFS

66. ■ 17-00.parquet
■ 18-00.parquet
18:00-18:59
Hourly batch job
Kafka Flink HDFS

67. Hourly batch job
Kafka Flink HDFS
■ 17-00.parquet
■ 18-00.parquet
18:00-18:59

68. ■ 17-00.parquet
■ 18-00.parquet
■ 19-00.parquet
19:00-19:59
Hourly batch job
Kafka Flink HDFS

69. 19:00-19:59 ■ 17-00.parquet
■ 18-00.parquet
■ 19-00.parquet
Hourly batch job
Kafka Flink HDFS

70. 19:00-19:59 ■ 17-00.parquet
■ 18-00.parquet
■ 19-00.parquet
Hourly batch job
Kafka Flink HDFS

71. Sticking to batch processing…

72. It’s not perfect
• Reprocessing data

73. It’s not perfect
• Reprocessing data
• Does not support “semi-real-time”

74. It’s not perfect
• Reprocessing data
• Does not support “semi-real-time”
• Optimization
• Kafka time-based indexing

75. Conclusion

76. Proper solution?
• Use a database instead of files, or
• Use a different tool (e.g. Kafka Streams), or
• Write small files and merge them in the end, or
• Skip late events
• e.g. accept 5 minutes late, but not 12 hours

77. Support real-time?
• Kappa-architecture
• Streaming-only
• Lambda-architecture
• Batch system + streaming system
• Late events in daily batches + 5-minute files dropping late events

78. Streaming is not trivial

79. Streaming is not trivial (yet)

80. Streaming is not trivial (yet)
• Keep it simple!
• Understand the system!

81. ?Gábor Hermann
ghermann@bol.com
@GbrHrmnn
Blogpost
https://tinyurl.com/kafka-to-parquet