Is Exactly Once Delivery a pipe dream? Recent versions of Kafka have claimed they have made it a reality. In this talk I will go over the different message delivery guarantees and the protocols that implement them. I will focus on Kafka's transaction based messaging protocol, It’s trade-offs and shortcomings. And how it can power Wix's event-driven Infra for its highly distributed environment. Is Exactly Once Delivery a pipe dream? Recent versions of Kafka have claimed they have made it a reality. In this talk I will go over the basic theory of messaging in distributed systems, the different message delivery guarantees and the protocols that implement them. I will focus on exactly once delivery guarantees and the way Kafka implements it with transaction based messaging protocol between the producer and the consumer. Including a discussion of the latency/throughput trade-offs, resource utilisation and its overall shortcomings. Finally, I will show how it has helped power Wix's event-driven data-streaming and data-storage Infrastructure we hope to open source soon. This infrastructure greatly simplifies building and maintaining services in a highly distributed production environment. It lets developers focus on business logic instead of keep making sure their code is idempotent and fault-tolerant.

1. natans@wix.com twitter @NSilnitsky linkedin/natansilnitsky github.com/natansil
Exactly Once Delivery
is a harsh mistress
Natan Silnitsky
Backend Infra Developer, Wix.com

2. As it happens,
we have a distributed system at Wix.
~1400 micro-services
@NSilnitsky

3. As it happens,
we have a distributed system at Wix.
Edit MySite
~1400 micro-services
~850M messages every day
@NSilnitsky

4. PurchaseCompleted
Classic ecommerce flow
UpdateInventory(ItemN)
updates
UpdateInventory(Item1)
UpdateInventory(Item2)
@NSilnitsky

5. Classic ecommerce flow - hard to achieve
PurchaseCompleted
UpdateInventory(ItemN)
updates
UpdateInventory(Item1)
UpdateInventory(Item2)
😳 But... how do we update all items exactly once? on failures/restarts too...

6. PurchaseCompleted
updates
UpdateInventory(Item1)
UpdateInventory(Item2)
Classic ecommerce flow - hard to achieve
Inventory:
Item1 9 → 8
Item2 5 → 4
@NSilnitsky

7. PurchaseCompleted
updates
UpdateInventory(Item1)
UpdateInventory(Item2)
Classic ecommerce flow - hard to achieve
Inventory:
Item1 9 → 7
Item2 5 → 3
@NSilnitsky

8. Achieving Exactly-Once delivery in
distributed systems is NOT easy.

9. Message delivery over the network
is unreliable.
@NSilnitsky

10. Message delivery over the network
is unreliable.
Message
Consumer
Message
Producer
Broker
@NSilnitsky

11. Kafka Broker
Topic-1
0 1 2 3 45
0 1 2 3 45
0 1 2 3 45
0 1 2 3 45
0 1 2 3 4
Topic-2
0 1 2 3 45
0 1 2 3 45
0 1 2 3 45
0 1 2 3 45
0 1 2 3 4
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
(append-only logs)
Partitions
@NSilnitsky

12. Kafka
Consumer
Kafka
Producer
Kafka Broker
Topic-1
0 1 2 3 45
0 1 2 3 45
0 1 2 3 45
0 1 2 3 45
0 1 2 3 4
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
0 1 2 3 4 5
@NSilnitsky

13. The options for
message delivery
Why Exactly-Once
is difficult
Kafka’s solution for
Exactly-Once
delivery

14. Kafka Broker
Kafka
Producer
0 1 2 3 4 5
😕 Services need to address message duplicates
at-most-onceat-least-once exactly-once
Producer retries on
every failure

15. Kafka Broker
0 1 2 3 4 5
😕 Services need to handle messages exactly once
consumerRecords = Consumer.poll
process(consumerRecords)
consumer.commit
at-most-onceat-least-once exactly-once
Consumer retries
on every failure
Kafka
Consumer

16. Kafka Broker
0 1 2 3 4 5
😕 Messages may be lost
Consumer commits
before processing
consumerRecords = Consumer.poll
Consumer.commit
process(consumerRecords)
at-most-onceat-least-once exactly-once
Kafka
Consumer

17. Kafka Broker
0 1 2 3 4 5
Kafka
Consumer
😕 Really really really hard to do
Messages are read and processed
exactly once by the consumer
?
at-least-once exactly-onceat-most-once

18. The options for
message delivery
Why Exactly-Once
is difficult
Kafka’s solution for
Exactly-Once
delivery

19. The “Two Generals Problem"
thought experiment
Alice Bob

20. Tomorrow
7 AM
The “Two Generals Problem"
thought experiment
Alice Bob

21. The “Two Generals Problem"
thought experiment
Alice Bob

22. The “Two Generals Problem"
thought experiment
Alice Bob
Tomorrow
7 AM

23. OK!
Tomorrow
7 AM
The “Two Generals Problem"
thought experiment
Alice Bob

24. ?
The “Two Generals Problem"
thought experiment
Alice Bob
OK!

25. The “Two Generals Problem"
thought experiment
AliceService BobService

26. The options for
message delivery
Why Exactly-Once
is difficult
Kafka’s solution for
Exactly-Once
delivery

27. Kafka Broker
Processor Observer
Consumer
-Producer
Consumer
Topic A Topic B
@NSilnitsky

28. Kafka Broker
Processor Observer
Consumer
-Producer
Consumer
Topic A Topic B
@NSilnitsky

29. Kafka Broker
Kafka
Idempotent
Producer
0 1 2 3 4 5
0 1
1
Attaches offset to
message
Enable.idempotence = true
@NSilnitsky

30. Kafka Broker
Kafka
Idempotent
Producer
0 1 2 3 4 5
0 1
1
Attaches offset to
message
Enable.idempotence = true
duplicate!
@NSilnitsky

31. Kafka Broker
Topic A Topic B
Processor Observer
Consumer
-Producer
Consumer
@NSilnitsky

32. Kafka Broker
consumer.poll
Attaches offset to message +
marks transaction
ABCDEF
Processor Observer
AB
01
Topic A Topic B
@NSilnitsky

33. Kafka Broker
consumer.poll
producer.beginTransaction
Producer.send
Attaches offset to message +
marks transaction
ABCDEF
Processor Observer
ABC
01
Topic A Topic B
@NSilnitsky

34. Kafka Broker
consumer.poll
producer.beginTransaction
Producer.send
producer.sendOffsets
Attaches offset to message +
marks transaction
ABCDEF
Processor Observer
ABC
012
Topic A Topic B
@NSilnitsky

35. Kafka Broker
consumer.poll
producer.beginTransaction
Producer.send
producer.sendOffsets
producer.commitTransaction
Attaches offset to message +
marks transaction
ABCDEF
Processor Observer
ABC
012
Topic A Topic B
@NSilnitsky

36. Kafka Broker
consumer.poll
producer.beginTransaction
Producer.send
producer.sendOffsets
Attaches offset to message +
marks transaction
ABCDEF
Processor Observer
ABC
01
Topic A Topic B
@NSilnitsky

37. Kafka Broker
The bigger the batch the longer the latency.
Processor Observer
@NSilnitsky

38. Kafka Broker
Processor Observer
No end-to-end exactly-once guarantees
out of the box.
@NSilnitsky

39. In reality, this is more complex
than how I describe it.
Out of scope:
▪ Two-phase-commit with transaction coordinator
▪ Transaction log
▪ Additional “fencing” data
▪ 1 processor-producer - 1 partition
@NSilnitsky

40. Purchase
Completed
UpdateInventory(Item1)
Exactly Once (Classic ecommerce) flow at Wix
UpdateInventory(Item2)
UpdateInventory(ItemN)
ABCDEF
1
iNB
1
I3I8
I1I2I5I5
01
updates
@NSilnitsky

41. Exactly-Once delivery is...
like the holy grail of message delivery over the network.
It’s a tough nut to crack.

42. Exactly-Once delivery is...
complex to implement, to use, and it requires fine-tuning.

43. Exactly-Once delivery is...
crucial for achieving atomic actions in distributed systems.

44. Thank You
natans@wix.com twitter @NSilnitsky linkedin/natansilnitsky github.com/natansil

45. Resources
EoS in Kafka talk by Jason Gustafson
Exactly-once Semantics are Possible by Neha Narkhede
Transactions in Apache Kafka
Revisiting Exactly One Semantics by Jason Gustafson, Confluent
Proposal to improve EOS Produce scalability
How akka works with exactly once message delivery by Hugh McKee

46. Slides & More
slideshare.net/NatanSilnitsky
medium.com/@natansil
twitter.com/NSilnitsky