I will present the recent additions to Kafka to achieve exactly-once semantics (0.11.0) within its Streams API for stream processing use cases. This is achieved by leveraging the underlying idempotent and transactional client features. The main focus will be the specific semantics that Kafka distributed transactions enable in Streams and the underlying mechanics to let Streams scale efficiently.

1. Exactly-once Stream Processing
with Kafka Streams
Guozhang Wang
Kafka Summit SF, Aug. 28, 2017

2. Outline
• Stream processing with Kafka
• Exactly-once for stream processing
• How Kafka Streams enabled exactly-once
2

3. 3
Stream Processing with Kafka
Process
State
Ads Clicks
Ads Displays
Billing Updates
Fraud Suspects
Your App

4. 4
Stream Processing with Kafka
Process
State
Ads Clicks
Ads Displays
Billing Updates
Fraud Suspects
ack
ack
commit
Your App

5. 5
Stream Processing: Do it Yourself
while (isRunning) {
// read some messages from Kafka
inputMessages = consumer.poll();
// do some processing…
// send output messages back to Kafka, wait for ack
producer.send(outputMessages).get();
// commit offsets for processed messages
consumer.commit(..);
}

6. 6
• Ordering
• Partitioning &
Scalability
• Fault tolerance
• State Management
• Time, Window &
Out-of-order Data
• Re-processing
DIY Stream Processing is Hard

7. 7
API,
coding
“Full stack”
evaluation
Operations, debugging,
…

8. 8
Exactly-Once
An application property for stream processing,
.. that for each received record,
.. its process results will be reflected exactly once,
.. even under failures

9. 9
Error Scenario #1: Duplicate
Writes
Process
State
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack
Streams App

10. 10
Error Scenario #1: Duplicate
Writes
Process
State
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack
producer config: retries = N (default = 0)
Streams App

11. 11
Error Scenario #2: Re-process
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
commit
ack
ack
State
Process
Streams App

12. 12
Error Scenario #2: Re-process
State
Process
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
State
Streams App

13. 13
Process
State
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
Life before 0.11: At-least-once +
Dedup

14. 14
Process
State
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack
ack
Life before 0.11: At-least-once +
Dedup

15. 15
Process
State
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack
ack
commit
Life before 0.11: At-least-once +
Dedup

16. 16
2
2
3
3
4
4
Life before 0.11: At-least-once +
Dedup

17. 17
Exactly-once, the Kafka Way!(0.11+)

18. 18
• Building blocks to achieve exactly-once
• Idempotence: de-duped sends in order per partition
• Transactions: atomic multiple-sends across topic partitions
• Kafka Streams: enable exactly-once in a single
knob
Exactly-once, the Kafka Way!(0.11+)

19. Kafka Streams (0.10+)
• New client library beyond producer and
consumer
• Powerful yet easy-to-use
• Event time, stateful processing
• Out-of-order handling
• Highly scalable, distributed, fault tolerant
19

20. 20
Anywhere, anytime
Ok. Ok. Ok. Ok.

21. 21
Anywhere, anytime
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-streams</artifactId>
<version>0.11.0.0</version>
</dependency>

22. 22
Simple is Beautiful

23. Kafka Streams DSL
23
public static void main(String[] args) {
// specify the processing topology by first reading in a stream from a topic
KStream<String, String> words = builder.stream(”topic1”);
// count the words in this stream as an aggregated table
KTable<String, Long> counts = words.groupBy(..).count(”Counts”);
// write the result table to a new topic
counts.to(”topic2”);
// create a streams client and start running it
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
}

24. Kafka Streams DSL
24
public static void main(String[] args) {
// specify the processing topology by first reading in a stream from a topic
KStream<String, String> words = builder.stream(”topic1”);
// count the words in this stream as an aggregated table
KTable<String, Long> counts = words.groupBy(..).count(”Counts”);
// write the result table to a new topic
counts.to(”topic2”);
// create a streams client and start running it
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
}

25. Kafka Streams DSL
25
public static void main(String[] args) {
// specify the processing topology by first reading in a stream from a topic
KStream<String, String> words = builder.stream(”topic1”);
// count the words in this stream as an aggregated table
KTable<String, Long> counts = words.groupBy(..).count(”Counts”);
// write the result table to a new topic
counts.to(”topic2”);
// create a streams client and start running it
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
}

26. Kafka Streams DSL
26
public static void main(String[] args) {
// specify the processing topology by first reading in a stream from a topic
KStream<String, String> words = builder.stream(”topic1”);
// count the words in this stream as an aggregated table
KTable<String, Long> counts = words.groupBy(..).count(”Counts”);
// write the result table to a new topic
counts.to(”topic2”);
// create a streams client and start running it
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
}

27. Processor Topology
27
State
KStream<..> stream1 = builder.stream(”topic1”);
KStream<..> stream2 = builder.stream(”topic2”);
KStream<..> joined = stream1.leftJoin(stream2, ...);
KTable<..> aggregated = joined.groupBy(…).count(”store”);
aggregated.to(”topic3”);

28. Processor
28
State
KStream<..> stream1 = builder.stream(”topic1”);
KStream<..> stream2 = builder.stream(”topic2”);
KStream<..> joined = stream1.leftJoin(stream2, ...);
KTable<..> aggregated = joined.groupBy(…).count(”store”);
aggregated.to(”topic3”);

29. Stream
29
State
KStream<..> stream1 = builder.stream(”topic1”);
KStream<..> stream2 = builder.stream(”topic2”);
KStream<..> joined = stream1.leftJoin(stream2, ...);
KTable<..> aggregated = joined.groupBy(…).count(”store”);
aggregated.to(”topic3”);

30. State Store
30
State
KStream<..> stream1 = builder.stream(”topic1”);
KStream<..> stream2 = builder.stream(”topic2”);
KStream<..> joined = stream1.leftJoin(stream2, ...);
KTable<..> aggregated = joined.groupBy(…).count(”store”);
aggregated.to(”topic3”);

31. Kafka Streams DSL
31
public static void main(String[] args) {
// specify the processing topology by first reading in a stream from a topic
KStream<String, String> words = builder.stream(”topic1”);
// count the words in this stream as an aggregated table
KTable<String, Long> counts = words.groupBy(..).count(”Counts”);
// write the result table to a new topic
counts.to(”topic2”);
// create a streams client and start running it
KafkaStreams streams = new KafkaStreams(builder, config);
streams.start();
}

32. Processor
Topology
32Kafka Streams Kafka
State

33. 33
Kafka Topic B Kafka Topic A
P1
P2
P1
P2
Processing in Kafka
Streams

34. 34
Kafka Topic B Kafka Topic A
Processor Topology
P1
P2
P1
P2
Processing in Kafka
Streams

35. 35
Kafka Topic AKafka Topic B
Processing in Kafka
Streams

36. MyApp.2MyApp.1
Kafka Topic B
Task2Task1
36
Kafka Topic A
State State
Processing in Kafka
Streams

37. MyApp.2MyApp.1
Kafka Topic B
Task2Task1
37
Kafka Topic A
State State
Processing in Kafka
Streams
Kafka Changelog Topic

38. 38
Process
State
Kafka Topic C
Kafka Topic D
ack
ack
Kafka Topic A
Kafka Topic B
commit
Exactly-once with Kafka
• Acked produce to sink topics
• Offset commit for source
topics
• State update on processor

39. 39
• Acked produce to sink topics
• Offset commit for source
topics
• State update on processor
Exactly-Once with Kafka

40. 40
• Acked produce to sink topics
• Offset commit for source
topics
• State update on processor
All or Nothing
Exactly-Once with Kafka

41. 41
Exactly-Once with Kafka Streams
(0.11+)
• Acked produce to sink topics
• Offset commit for source
topics
• State update on processor

42. 42
Exactly-Once with Kafka Streams
(0.11+)
• Acked produce to sink topics
• A batch of records sent to the offset topic
• State update on processor

43. 43
• Acked produce to sink topics
• A batch of records sent to the offset topic
Exactly-Once with Kafka Streams
(0.11+)
• A batch of records sent to changelog
topics

44. 44
Exactly-Once with Kafka Streams
(0.11+)
• A batch of records sent to sink topics
• A batch of records sent to the offset topic
• A batch of records sent to changelog
topics

45. 45
Exactly-Once with Kafka Streams
(0.11+)
All or Nothing
• A batch of records sent to sink topics
• A batch of records sent to the offset topic
• A batch of records sent to changelog
topics

46. 46Kafka Streams
Kafka Input Topics
Kafka Changelog Topic
Kafka Output Topic
try {
producer.beginTxn();
} catch (KafkaException e) {
}
Kafka Streams Exactly-Once
State

47. 47Kafka Streams
Kafka Input Topics
Kafka Changelog Topic
Kafka Output Topic
Kafka Streams Exactly-Once
State
try {
producer.beginTxn();
recs = consumer.poll();
for (Record rec <- recs) {
// process ..
}
} catch (KafkaException e) {
}

48. Kafka Streams Exactly-
Once
48Kafka Streams
Kafka Input Topics
Kafka Changelog Topic
Kafka Output Topic
try {
producer.beginTxn();
recs = consumer.poll();
for (Record rec <- recs) {
// process ..
}
} catch (KafkaException e) {
}
State

49. Kafka Streams Exactly-
Once
49Kafka Streams
Kafka Input Topics
State
Kafka Changelog Topic
Kafka Output Topic
try {
producer.beginTxn();
recs = consumer.poll();
for (Record rec <- recs) {
// process ..
producer.send(“output”, ..);
producer.send(“changelog”, ..);
}
} catch (KafkaException e) {
}

50. Kafka Streams Exactly-
Once
50Kafka Streams
Kafka Input Topics
State
Kafka Changelog Topic
Kafka Output Topic
try {
producer.beginTxn();
recs = consumer.poll();
for (Record rec <- recs) {
// process ..
producer.send(“output”, ..);
producer.send(“changelog”, ..);
producer.sendOffsets(“input”, ..);
}
} catch (KafkaException e) {
}

51. Kafka Streams Exactly-
Once
51Kafka Streams
Kafka Input Topics
State
Kafka Changelog Topic
Kafka Output Topic
try {
producer.beginTxn();
recs = consumer.poll();
for (Record rec <- recs) {
// process ..
producer.send(“output”, ..);
producer.send(“changelog”, ..);
producer.sendOffsets(“input”, ..);
}
producer.commitTxn();
} catch (KafkaException e) {
producer.abortTxn();
}

52. 52
StateProces
s
StateProces
s
StateProces
s
Exactly-Once with Failures
Kafka
Kafka Streams
Kafka Changelog
Kafka

53. 53
StateProces
s
StateProces
s
StateProces
s
Exactly-Once with Failures
Kafka
Kafka Streams
Kafka Changelog
Kafka

54. 54
StateProces
s
StateProces
s
Proces
s
Kafka
Kafka Streams
Kafka Changelog
Kafka
Exactly-Once with Failures
State

55. 55
StateProces
s
StateProces
s
StateProces
s
Kafka
Kafka Streams
Kafka Changelog
Kafka
Exactly-Once with Failures

56. 56
StateProces
s
StateProces
s
StateProces
s
Kafka
Kafka Streams
Kafka Changelog
Kafka
Exactly-Once with Failures

57. 57
Exactly-Once with Failures
config: processing.mode = exactly-once
(default = at-least-once)
[KIP-98, KIP-
129]

58. 58
API,
coding
“Full stack”
evaluation
Operations, debugging,
…
Simple is Beautiful

59. 59
Life is Good with Exactly-Once, but..

60. 60
What if not all my data is in Kafka?

61. 61

62. 62
Connectors
• 60+ since first
release (0.9+)
• 20+ from
& partners
(exactly-once coming)

63. 63
Connect
End-to-End Exactly-Once
Connect
Connect
Connect
Connect
Connect
Connect
Streams
Streams
Streams
Streams
Streams
Exactly-Once Zone
Wild Wild West
Connect

64. Take-aways
• Exactly-once: important property for stream
processing
64

65. Take-aways
• Exactly-once: important property for stream
processing
• Kafka Streams: exactly-once processing made
easy
65

66. Take-aways
66
THANKS!
Guozhang Wang | guozhang@confluent.io | @guozhangwang
Additional Resources: http://www.confluent.io/resources
• Exactly-once: important property for stream
processing
• Kafka Streams: exactly-once processing made
easy

67. 67

68. 68

69. 69
BACKUP SLIDES

70. Exactly-Once Performance
• 3 Brokers, 3 Streams on AWS m3.xlarge
• max.inflight.request = 1, commit.interval = 100ms
70
record size (bytes) at-least-once exactly-once
100 100% 68%
500 100% 75%
1024 100% 82%
config: processing.mode = exactly-once (default = at-least-once)

71. 71
Exactly-Once does NOT mean..
• Two Generals problem can now be solved
• .. or FLP result is proved wrong
• .. or TCP at transport level is “perfect”
• .. or you can get distributed consensus in any
settings

72. 72
Producer
Kafka Topic Cack
Idempotent Producer
pid = 1
pid = 1
seq = 28
pid = 1
seq = 28

73. 73
Producer
Kafka Topic Cack (dup)
Idempotent Producer
pid = 1
pid = 1
seq = 28
pid = 1
seq = 28
config: enable.idempotence = true

74. 74
We are Hiring!