Apache Pulsar is used for various streaming use cases. There is a strong requirement for storing checkpoints while processing the stream in Pulsar Functions, so that in case of any interruption stream processing engine could go back to the last checkpoint. Pulsar uses Zookeeper not only for leader elections or service discovery like critical use cases but also for storing various metadata which puts unnecessary load on zookeeper which hampers mission critical use of Zookeeper. A durable key value store based off of the Apache Pulsar ecosystem addresses the above mentioned use cases nicely. This talk focuses on taking existing Apache Bookkeeper Table Service/State store implementation and taking it to production. Furthermore this talk also touches upon contributing all the features, bug fixes, tools and other improvements back to open source.

1. Pulsar Virtual Summit North America 2021
Apache BookKeeper State Store:
A Durable Key-Value Store

2. Pulsar Virtual Summit North America 2021
Prashant Kumar
Principal Software Engineer @ Splunk
● Principal Software Developer at Splunk.
● Ex Yahoo, Verizon Media.
● Prior Experience, Key member and
contributor to Sherpa, the geographically
replicated multi tenant key value store @
Yahoo.

3. Pulsar Virtual Summit North America 2021
Agenda
I. Introduction to Apache Bookkeeper Statestore.
II. Why yet another KV store?
III. How does it fit into Pulsar ecosystem?
IV. Intended use case.
V. Brief architecture.
VI. Current state and production worthiness.
VII. Product roadmap and future work.

4. Pulsar Virtual Summit North America 2021
Apache Bookkeeper Statestore
● It’s a Key-Value store.
● It’s durable.
● It’s locally replicated.
● It’s eventually consistent.
● It’s fault tolerant.
● It’s cloud native and k8s based deployment.

5. Pulsar Virtual Summit North America 2021
Argh!. Yet another KV store?

6. Pulsar Virtual Summit North America 2021
Integral part of Apache Pulsar ecosystem
● Uses same Zookeeper deployment that Pulsar uses.
● Uses same Bookkeeper deployment that Pulsar uses.
● Uses same infrastructure for metrics, dashboards etc as
Bookkeeper
● Part of bookkeeper/stream code base.
● Existing client side integration in Apache Pulsar function
service.

7. Pulsar Virtual Summit North America 2021
Primary use cases
● Store and access function state and checkpoints
● A secondary metadata store for Apache Pulsar, away from
Zookeeper
● Other various KV store use cases

8. Pulsar Virtual Summit North America 2021
Data model

9. Pulsar Virtual Summit North America 2021
High level serving architecture (Bird view)

10. Pulsar Virtual Summit North America 2021
High level Datastore architecture

11. Pulsar Virtual Summit North America 2021
Benchmarking
● Benchmarking with YCSB
● Setup
○ YCSB Thread count = 40
○ # k8s pods = 3
○ cpuRequest = 8
○ cpuLimit = 16
○ memoryRequest = 24Gi
○ memoryLimit = 24Gi
● Read output
○ Throughput = 22557.7 Ops/S
○ Average latency = 1.699 ms
○ 99%tile latency = 5.323 ms
● Write output
○ Throughput = 15256.16 Ops/S
○ Average latency = 8.820 ms
○ 99%tile = 27.071 ms

12. Pulsar Virtual Summit North America 2021
Production readiness
● It has already been in production for last few
months
● It’s a k8s based deployment
● Sustained production traffic
○ Read throughput 240 Ops/S
○ Write throughput 90 Ops/S

13. Pulsar Virtual Summit North America 2021
Product roadmap
● Contribute internal changes back to open source
● Storage hardening
○ Pick up the data litter actively and reactively
○ Clear obsolete transaction log
● Operability
○ Improved and granular monitoring and alerting.
● Availability
○ Implementation of replica.
○ Serving read traffic from a replica.
○ Elevation of replica to be primary when primary fails
● Scaling and load balancing
○ Splitting a shard
○ Moving shards across cluster

14. Pulsar Virtual Summit North America 2021
References
● Statestore Repo:
https://github.com/apache/bookkeeper/tree/master/stream
● Distributedlog Repo:
https://github.com/apache/bookkeeper/tree/master/stream/distri
butedlog
● Pulsar Function - Statestore integration:
https://github.com/apache/pulsar/blob/master/pulsar-
functions/worker/src/main/java/org/apache/pulsar/functions/wor
ker/PulsarWorkerService.java#L420