Transaction preview of Apache Pulsar

1. Penghui Li
Apache Pulsar PMC Member
Transaction Support in Pulsar
Yong Zhang
Apache Pulsar Contributor

2. What is Apache Pulsar?

3. Pub/Sub Messaging

4. “Flexible Pub/Sub messaging
backed by durable log/stream storage”

6. 2012: Pulsar idea started at Yahoo!
5 years on production, 100+ applications, 10+ data centers
2016/09 Yahoo open sourced Pulsar
2017/06 Yahoo donated Pulsar to ASF
2018/09 Pulsar graduated as a Top-Level project
2018/09 InfoWorld Best Open Source Project

7. Pulsar Community

8. Pulsar Community

9. • At-most once
• At-least once
• Exactly once
Messaging Semantics

10. • At-most once
• At-least once
• Exactly once
Messaging Semantics
Before 1.20.0-incubating

11. • At-most once
• At-least once
• Exactly once
Messaging Semantics
PIP-6: Guaranteed Message Deduplication

12. Revisit Existing Semantics

13. Pulsar’s Existing Semantics
Log
BrokerProducer
send(m1)

14. Pulsar’s Existing Semantics
Log
BrokerProducer
append(m1)

15. Pulsar’s Existing Semantics
Log
BrokerProducer
m1

16. Pulsar’s Existing Semantics
Log
BrokerProducer
ack(m1)
m1

17. Pulsar’s Existing Semantics
Log
BrokerProducer
ack(m1)
m1

18. Pulsar’s Existing Semantics
Log
BrokerProducer
send(m2)
m1

19. Pulsar’s Existing Semantics
Log
BrokerProducer
append(m2)
m1
m2

20. Pulsar’s Existing Semantics
Log
BrokerProducer
m1
m2
ack(m2)

21. Pulsar’s Existing Semantics
Log
BrokerProducer
m1
m2
ack(m2)
What do we do now?

22. At Least Once
Log
BrokerProducer
m1
m2
send(m2)

23. At Least Once
Log
BrokerProducer
m1
m2
append(m2)
m2

24. At Least Once
Log
BrokerProducer
m1
m2
append(m2)
m2
Duplicates !!

25. • Broker can fail
• The request from Producer to Broker can fail
• Producer or Consumer can fail
Why the duplicates are introduced?

26. I want exactly-once

27. • Producer: Idempotent Producer
• Broker: Guaranteed Message Deduplication (PIP-6)
• Consumer: Reader + Checkpoints (Flink / Spark)
Message Deduplication

28. • Producer Name - Identify who is producing the messages
• Sequence ID - Identify the message
• Producer Name + Sequence ID: The unique identifier for a
message
Idempotent Producer

29. • Broker maintains a map between Producer Name and Last-
Produced-Sequence-ID
• Broker accepts messages if the sequence id of a new
message is larger than its last produced sequence id
• Broker treats messages whose sequence id are smaller
• Broker keeps the map in a de-duplication cursor (stored in
bookkeeper)
Guaranteed Message Deduplication

30. Exactly Once
Log
BrokerProducer
send(1, m1)

31. Exactly Once
Log
BrokerProducer
append(1, m1)
1,m1

32. Exactly Once
Log
BrokerProducer
append(2, m2)
1,m1
2,m2

33. Exactly Once
Log
BrokerProducer
1,m1
2,m2
ack(2, m2)
What do we do now?

34. Exactly Once
Log
BrokerProducer
1,m1
2,m2
send(2, m2)

35. Exactly Once
Log
BrokerProducer
1,m1
2,m2
append(2, m2)

36. Exactly Once
Log
BrokerProducer
1,m1
2,m2
append(2, m2)
Duplicate detected

37. Exactly Once
Log
BrokerProducer
1,m1
2,m2
ack(2, m2)

38. • `bin/pulsar-admin set-deduplication -e tenant/namespace`
• Set producer name when creating a Producer
• Specify increasing sequence id when producing messages
Enable Exactly Once

39. • It only works when producing messages to one partition
• It only works for producing one message
• There is no atomicity when producing multiple messages to
one partition or many partitions
• Consumers are required to store the MessageId along with
its state and seek back to the MessageId when restoring
the state
Limitations

40. Introducing Transactions

41. PulsarCash

42. PulsarCash
Transfer $10
Alice Bob

43. • Transfer Topic : record the transfer requests
• Cash Transfer Function: perform the cash transfer action
• BalanceUpdate Topic: record the balance-update requests
PulsarCash, powered by Apache Pulsar

44. PulsarCash
Cash Transfer
Function
Balance
user:alice, debit($10)
balance
update
balance
update
user:bob, credit($10)
(100,0,0): transfer($10, alice -> bob)
Transfer Topic

45. Ack Transfer
Cash Transfer
function
Balance
user:alice, debit($10)
balance
update
balance
update
user:bob, credit($10)
(100,0,0): transfer($10, alice -> bob)
Ack: (100,0,0)

46. Reprocessed Transfer!
Cash Transfer
function
Balance
user:alice, debit($10)
balance
update
balance
update
user:bob, credit($10)
(100,0,0): transfer($10, alice -> bob)
Ack: (100,0,0)

47. Lost Money!
Cash Transfer
function
Balance
user:alice, debit($10)
balance
update
balance
update
user:bob, credit($10)
(100,0,0): transfer($10, alice -> bob)
Ack: (100,0,0)

48. Pulsar Transaction Explained

49. • Atomic writes across multiple partitions
• Atomic acknowledges across multiple subscriptions
• All the actions made within one transaction either all
succeed or all fail
• Consumers are *ONLY* allowed to read committed
messages
Transaction Semantics

50. Message<String> message = inputConsumer.receive();
CompletableFuture<MessageId> sendFuture1 =
producer1.newMessage().value(“output-message-1”).sendAsync();
CompletableFuture<MessageId> sendFuture2 =
producer2.newMessage().value(“output-message-2”).sendAsync();
inputConsumer.acknowledgeAsync(message.getMessageId());
Without Transaction API

51. Broker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Data Log Data Log
Pulsar Client
Input
Consumer
Producer 1 Producer 2
0) Receive Message
1) Produce Messages
2) Ack Messages

52. Message<String> message = inputConsumer.receive();
Transaction txn = client.newTransaction().withTransactionTimeout(…).build().get();
CompletableFuture<MessageId> sendFuture1 =
producer1.newMessage(txn).value(“output-message-1”).sendAsync();
CompletableFuture<MessageId> sendFuture2 =
producer2.newMessage(txn).value(“output-message-2”).sendAsync();
inputConsumer.acknowledgeAsync(message.getMessageId(), txn);
txn.commit().get();
MessageId msgId1 = sendFuture1.get();
MessageId msgId2 = sendFuture2.get();
Transaction API

53. CoordinatorBroker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Transaction Log
Data Log
Txn Buffer
Data Log
Txn Buffer
Pulsar Client
Input
Consumer
Producer 1 Producer 2

54. CoordinatorBroker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Transaction Log
Data Log
Txn Buffer
Data Log
Txn Buffer
Pulsar Client
Input
Consumer
Producer 1 Producer 2

55. • TC: transaction manager, coordinating committing and
aborting transactions
• In-Memory + Transaction Log
• Transaction Log is powered by a partitioned Pulsar topic
• `pulsar/system/__transaction_coordinator_log`
• Locating a TC is locating a partition of the transaction log
topic
Transaction Coordinator (TC)

56. CoordinatorBroker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Transaction Log
Data Log
Txn Buffer
Data Log
Txn Buffer
Pulsar Client
Input
Consumer
Producer 1 Producer 2

57. • TB: store and index transaction data per topic partition
• TB is implemented using another ML (managed-ledger) as
TB log
• Messages are appended to into TB log
• Transaction Index is maintained in memory and
snapshotted to ledgers
• Transaction Index can be replayed from TB log
Transaction Buffer (TB)

58. CoordinatorBroker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Transaction Log
Data Log
Txn Buffer
Data Log
Txn Buffer
Pulsar Client
Input
Consumer
Producer 1 Producer 2

59. • Introduce ACK_PENDING state
• Add response for acknowledgement, aka ack-on-ack
• Ack state is updated to cursor ledger
• Ack state can be replayed from cursor ledger
Transactional Subscription State

60. Transaction Execution Flow

61. Message<String> message = inputConsumer.receive();
Transaction txn = client.newTransaction().withTransactionTimeout(…).build().get();
CompletableFuture<MessageId> sendFuture1 =
producer1.newMessage(txn).value(“output-message-1”).sendAsync();
CompletableFuture<MessageId> sendFuture2 =
producer2.newMessage(txn).value(“output-message-2”).sendAsync();
inputConsumer.acknowledgeAsync(message.getMessageId(), txn);
txn.commit().get();
MessageId msgId1 = sendFuture1.get();
MessageId msgId2 = sendFuture2.get();
Transaction API - New Transaction

62. CoordinatorBroker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Transaction Log
Data Log
Txn Buffer
Data Log
Txn Buffer
Pulsar Client
Input
Consumer
Producer 1 Producer 2
Txn
New Txn
1. New Txn
Tx1

63. Message<String> message = inputConsumer.receive();
Transaction txn = client.newTransaction().withTransactionTimeout(…).build().get();
CompletableFuture<MessageId> sendFuture1 =
producer1.newMessage(txn).value(“output-message-1”).sendAsync();
CompletableFuture<MessageId> sendFuture2 =
producer2.newMessage(txn).value(“output-message-2”).sendAsync();
inputConsumer.acknowledgeAsync(message.getMessageId(), txn);
txn.commit().get();
MessageId msgId1 = sendFuture1.get();
MessageId msgId2 = sendFuture2.get();
Transaction API - Produce Messages

64. CoordinatorBroker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Transaction Log
Data Log
Txn Buffer
Data Log
Txn Buffer
Pulsar Client
Input
Consumer
Producer 1 Producer 2
Txn
New Txn
2.0 Add Produced Topics
To Txn
Tx1
Tx1: add [T1, T2] Tx1: M1 Tx1: M2
2.1 Produced Messages
To Topics with Txn

65. Message<String> message = inputConsumer.receive();
Transaction txn = client.newTransaction().withTransactionTimeout(…).build().get();
CompletableFuture<MessageId> sendFuture1 =
producer1.newMessage(txn).value(“output-message-1”).sendAsync();
CompletableFuture<MessageId> sendFuture2 =
producer2.newMessage(txn).value(“output-message-2”).sendAsync();
inputConsumer.acknowledgeAsync(message.getMessageId(), txn);
txn.commit().get();
MessageId msgId1 = sendFuture1.get();
MessageId msgId2 = sendFuture2.get();
Transaction API - Acknowledges

66. CoordinatorBroker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Transaction Log
Data Log
Txn Buffer
Data Log
Txn Buffer
Pulsar Client
Input
Consumer
Producer 1 Producer 2
Txn
New Txn
3.0 Add Acked Subscriptions To Txn
Tx1
Tx1: add [T1, T2] Tx1: M1 Tx1: M2
3.0 Ack messages with Txn
Tx1: ACK (M0)
Tx1: add [S0]

67. Message<String> message = inputConsumer.receive();
Transaction txn = client.newTransaction().withTransactionTimeout(…).build().get();
CompletableFuture<MessageId> sendFuture1 =
producer1.newMessage(txn).value(“output-message-1”).sendAsync();
CompletableFuture<MessageId> sendFuture2 =
producer2.newMessage(txn).value(“output-message-2”).sendAsync();
inputConsumer.acknowledgeAsync(message.getMessageId(), txn);
txn.commit().get();
MessageId msgId1 = sendFuture1.get();
MessageId msgId2 = sendFuture2.get();
Transaction API - Commit

68. CoordinatorBroker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Transaction Log
Data Log
Txn Buffer
Data Log
Txn Buffer
Pulsar Client
Input
Consumer
Producer 1 Producer 2
Txn
New Txn
4.0 Commit Txn
Tx1
Tx1: add [T1, T2] Tx1: M1 Tx1: M2
Tx1: ACK (M0)
Tx1: add [S0]
4.0 Committing Txn
Tx1: Committing

69. CoordinatorBroker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Transaction Log
Data Log
Txn Buffer
Data Log
Txn Buffer
Pulsar Client
Input
Consumer
Producer 1 Producer 2
Txn
New Txn
Tx1
Tx1: add [T1, T2] Tx1: M1 Tx1: M2
Tx1: ACK (M0)
Tx1: add [S0]
4.1.0 Commit Txn
On Topics
4.1.1 Commit Txn
On Subscriptions
Tx1 (c) Tx1 (c)
Tx1: Committing
Tx1: Committed
Tx1: Committed

70. CoordinatorBroker-0 Broker-1
InputTopic OutputTopic-1 OutputTopic-2
Cursor
Transaction Log
Data Log
Txn Buffer
Data Log
Txn Buffer
Pulsar Client
Input
Consumer
Producer 1 Producer 2
Txn
New Txn
Tx1
Tx1: add [T1, T2] Tx1: M1 Tx1: M2
Tx1: ACK (M0)
Tx1: add [S0]
Tx1: Committing
Tx1 (c) Tx1 (c)
Tx1: Committed
Tx1: Committed
4.2 Committed Txn

71. inputConsumer.receiveAsync().thenCompose(message -> {
return client.newTransaction().withTransactionTimeout(…).build().thenCompose(txn -> {
producer1.newMessage(txn).value(“output-message-1”).sendAsync();
producer2.newMessage(txn).value(“output-message-2”).sendAsync();
inputConsumer.acknowledgeAsync(message.getMessageId(), txn);
return txn.commit();
});
})
Transaction API - Async Example

72. PulsarCash
Cash Transfer
function
Balance
user:alice, debit($10)
balance
update
balance
update
user:bob, credit($10)
(100,0,0): transfer($10, alice -> bob)Ack: (100,0,0)

73. PulsarCash
Cash Transfer
function
Balance
user:alice, debit($10)
balance
update
balance
update
user:bob, credit($10)
(100,0,0): transfer($10, alice -> bob)Ack: (100,0,0)
Transaction

74. Make Event Streaming
easy, simple, and reliable for everyone
Pulsar Transaction

75. Available to use in Pulsar 2.6.0
When is it available?

76. • Transaction support in other languages (e.g. C++, Go)
• Transaction in Pulsar Functions & Pulsar IO
• Transaction in Kafka-on-Pulsar (KOP)
• Transaction for Flink / Spark job
• Transaction for State storage in Pulsar Functions
• …
Roadmap

77. • Ivan Kelly
• Matteo Merli
• Jia Zhai
• Penghui Li
• Marvin Cai
• Yong Zhang
• … and many other Pulsar users & contributors
Credits

78. Wechat Subscription: ApachePulsar
Mailing Lists
dev@pulsar.apache.org, users@pulsar.apache.org
Slack
https://apache-pulsar.slack.com (#china)
register: https://apache-pulsar.herokuapp.com/
https://github.com/apache/pulsar
https://github.com/apache/bookkeeper

79. Thanks!
Penghui Li Yong Zhang