Distributed ID generator in ChatWork

チャットワークのメッセージシステム
を支える
新分散ID発行器の内部
安田裕介/Yusuke Yasuda (@TanUkkii007)
Distributed ID generator in ChatWork

2018/02/27 © ChatWork All rights reserved.
Agenda
● What is our motivation for distributed ID
● How our distributed ID generator works
● Designing distributed ID generator with actor model
● Implementing distributed ID generator with actor model
● Distributed ID generator in the wild

About me
● Yusuke Yasuda / 安田裕介
● @TanUkkii007
● Working for ChatWork for 2 years
● Scala developer

About ChatWork

What is our motivation for
distributed ID

Messaging system architecture overview
You can find more information about our architecture at Kafka summit 2017.
Today’s topic

Motivation
● Migration from MySQL to Kafka/HBase
○ High scalability
○ No single point of failure
● compatibility with existing IDs
○ Time-ordered, sortable integer IDs
● ID space extension
○ 32bit → 64bit
● ID generator itself is required to be scalable and distributed

Snowflake
● Distributed ID generator developed by Twitter
○ https://github.com/twitter/snowflake
● Motivation: migration to Cassandra from MySQL
● Roughly time-ordered, sortable 64bit ID
● > 10k ID/s per process, ~ 2ms response rate
41 bit
timestamp
5 bit
datacenter ID
5 bit
worker ID
12 bit
sequenceNr
… … … …

ZooKeeper
● Developed by Yahoo! Research
○ to simplify distributed system implementation by
supporting common patterns used in distributed systems
● Quorum based coordination
● Total order broadcast
● Filesystem-like API
○ Create, GetData, SetData, Exists, GetChildren, Delete
○ Watching node changes
We use ZooKeeper to coordinate worker IDs in distributed ID generator.

Distributed ID generator system overview
ID worker
ID worker
ID worker
ZooKeeper
ID client
ZooKeeper ensemble
ZooKeeper ZooKeeper
ID generator ID client in Message Write API

How our distributed ID generator
works

ID worker discovery via ZooKeeper
ZooKeeper
ID
client
/id-worker/1
Command: GetChildren
Path: /id-worker/1
Watch: True
Router
Datacenter 1
root node
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
Path: /id-worker/1
Watch: True
Router
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
/id-worker/1/1 akka.tcp://system@11.2.9.14:***
Command: Create
Path: /id-worker/1/1
Data: akka.tcp://system@11.2.9.14:***
mode: Ephemeral
Router
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
Event: NodeChildrenChanged
Path: /id-worker/1
Router
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
Path: /id-worker/1
Watch: True
Router
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
Command: GetData
Watch: False
Router
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
Identify
Router
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
ActorIdentity worker1
Router
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
ID request/response
worker1
Router
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
ID
worker
Path: /id-worker/1
Watch: True
worker1
Router
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
ID
worker
Command: Create
mode: Ephemeral
worker1
Router
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
ID
worker
Path: /id-worker/1
worker1
Router
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
ID
worker
Path: /id-worker/1
Watch: True
worker1
Router
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
ID
worker
Command: GetData
Watch: False
worker1
Router
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
ID
worker
Identify worker1
Router
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
worker1ID
worker
ActorIdentity
worker2
Router
command/event
watch

Client-side failure handling
ZooKeeper
ID
client
/id-worker/1
ID
worker
worker1ID
worker
worker2
Router
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
worker1ID
worker
worker2
Router
Path: /id-worker/1
command/event
watch

ZooKeeper
ID
client
/id-worker/1
ID
worker
worker1ID
worker
Router
Path: /id-worker/1
Watch: True
command/event
watch

Worker ID Consensus process
1. No worker ID duplication invariant
ZooKeeper
/id-worker/1
ID
worker
Path: /id-worker/1
Watch: True
command/event
watch

ZooKeeper
/id-worker/1
ID
worker
ID
worker
Path: /id-worker/1
Watch: True
command/event
watch

ZooKeeper
/id-worker/1
ID
worker
Command: Create
mode: Ephemeral
ID
worker
Failed with
NODEEXISTS
command/event
watch

ZooKeeper
/id-worker/1
ID
worker
ID
worker
akka.tcp://system@11.2.9.16:***/id-worker/1/31
ID
worker ID
worker
Path: /id-worker/1
Watch: True
2. No out of range worker ID creation
command/event
watch

ZooKeeper
/id-worker/1
ID
worker
ID
worker
ID
worker ID
worker
Suspended because no available
worker IDs.
command/event
watch

ZooKeeper
/id-worker/1
ID
worker
ID
worker
ID
worker ID
worker
command/event
watch

ZooKeeper
/id-worker/1
ID
worker
ID
worker
ID
worker ID
worker
Path: /id-worker/1
command/event
watch

ZooKeeper
/id-worker/1
ID
worker
ID
worker
ID
worker ID
worker
Path: /id-worker/1
Watch: True
command/event
watch

ZooKeeper
/id-worker/1
ID
worker
ID
worker
ID
worker ID
worker
Command: Create
mode: Ephemeral
command/event
watch

Automaton notation of ID worker
Create my worker ID
Initial state
Check available
worker IDs
Wait until a worker
ID gets available
Ready to generate
message IDs
Fetch list of worker IDsStart
Create a worker ID node
Retry
Retry
Something happened
to my worker ID
Retry
Worker ID conflict
No available
worker IDs
Retry

Designing distributed ID generator
with actor model

Processes, Modules, Automata, Steps
Process 1
Module 1
Module 2
Module 3
Process 2
Module 1
Module 2
Module 3
Automaton
local interaction
communication

Distributed algorithm abstraction
● Process: the unit of failure, the unit of communication
● Module: building block of processes
○ communication with modules on peer process
○ local interaction with modules on the same process
● Automata: set of states and transitions that regulates
computation steps
● Step: distributed algorithm consists of sequence of steps
○ e.g. receiving a message, sending a message,
executing a local computation
syntactically
identical but
different notion
Reliable and Secure Distributed Programming 2nd ed.

Algorithm abstraction to
concrete implementation mapping
● Process → Actor (top level actor with remote interface)
● Module → Actor
○ communication → message passing to remote actors
○ local interaction → message passing to local actors
● Layers of modules → Actor hierarchy tree
● Automata
○ State → Receive partial functions and its internal states
○ Transition → context.become()
● Step:
○ receiving a message → case clauses of Receive function
○ sending a message → ! or tell() function
○ executing a local computation: arbitrary computation
syntactically
identical

Modules in ID worker process
● IdGenerator: exports
communication interface
● IdWorker: calculates
Snowflake ID
● ZNodeMaster: manages
worker ID ZNode
● ReactiveZookeeper: Actor
based ZooKeeper client
ID client
process
ZooKeeper
process
IdGenerator
IdWorker ZNodeMaster
ReactiveZookeeper
ID worker process

IdWorker
IdGenerator
ReactiveZookeeper
Designing actor hierarchy based on
message flow simplifies its implementation.
See Akka in Action section 4.3.
Mapping layers of modules to actor
hierarchy based on message flow.
ZNodeMaster
IdGenerator
ReactiveZookeeper
ID worker process
message flow
parent child
relationship
message flow

failure handling
IdWorker
IdGenerator
ReactiveZookeeper
ZNodeMaster
ZooKeeper application is only valid within a session.
Instead of handling session timeout everywhere,
locate ReactiveZooKeeper at the top of the hierarchy.
Just let it crash on session timeout.
message flow
parent child
relationship
IdGenerator
ReactiveZookeeper
ID worker process

Implementing distributed ID
generator with actor model

What ZooKeeper application needs
● Asynchronous
● Event driven
● Passive: Don't call us, we'll call you style
● Stateful
● Complicated state machine
● Retry everything
● Crash immediately in fatal situation (session expiration)
● Recover gracefully

ReactiveZooKeeper
Let-it-crash style ZooKeeper client based on Akka actor
https://github.com/TanUkkii007/reactive-zookeeper

Implementing
automata with
Akka
class ZNodeMaster(reactiveZK: ActorRef) extends Actor {
override def receive: Receive = initial
def initial: Receive = {
case Start =>
self ! CheckWorkerIds
context.become(checkingWorkerIds)
}
def checkingWorkerIds: Receive = {
case CheckWorkerIds =>
reactiveZK ! GetChildren("id-worker/1", watch = true)
case ChildrenGot(path, children, _) if children.length <= maxWorkerId =>
val workerId = chooseId(children)
context.become(creatingWorkerId(workerId))
self ! CreateWorkerId
}
def creatingWorkerId(workerId: Int): Receive = {
case CreateWorkerId =>
reactiveZK ! Create(workerIdPath(workerId), address,
Ids.OPEN_ACL_UNSAFE.asScala.toList, CreateMode.EPHEMERAL)
}
}
Create
my
worker ID
Initial
state
Check
available
worker
IDs
Fetch
list of
worker IDsStart
State 1
State 2
State 3
state transition with
context.become
internal state as a closure variable
that is only visible within the state
ZNodeMaster module

Implementing self-loop of automata
Check
available
worker
IDs
Retry
def checkingWorkerIds: Receive = {
case CheckWorkerIds =>
reactiveZK ! GetChildren("id-worker/1", watch = true)
case ChildrenGot(path, children, _) if children.length <= maxWorkerId =>
val workerId = chooseId(children)
context.become(creatingWorkerId(workerId))
self ! CreateWorkerId
case GetChildrenFailure(e, _, _) if e.code() == Code.CONNECTIONLOSS =>
self ! CheckWorkerIds
}
Retry on connection loss by sending the
same message to self.
ZNodeMaster module

Implementing communication interface
def receive: Receive = {
case GetIdWorkerAddress(WorkerId(workerId)) =>
reactiveZK ! GetData(s"/${settings.rootNode}/${settings.dcId}/$workerId", watch = false, ctx = workerId)
case DataGot(path, data, stat, workerId: Long) =>
val idWorkerZNode = deserializeIdWorkerZNode(data)
val workerPath = idWorkerZNode.fullActorPath
val selection = context.actorSelection(workerPath)
selection ! Identify(workerId)
case ActorIdentity(workerId: Long, Some(workerRef)) =>
workers += WorkerId(workerId) -> context.watch(workerRef)
val newRoutee = IdWorkerRoutee(WorkerId(workerId), workerRef)
if (!router.routees.contains(newRoutee))
router = router.addRoutee(newRoutee)
case msg: GenerateId => router.route(IdWorkerGenerateId(msg.requestId.toString), sender())
}
Remote messages.
ID worker client implementation
Configuring RemoteActorRef
Provider let us write the
communication syntactically same
as the local interaction.

Implementing
crash
recovery
class ZooKeeperSessionActor(childProps: Props) extends Actor with WatcherCallback{
val zookeeper = new ZooKeeper("zookeeper:2181", 5000, watchCallback(self))
val zookeeperOperation: ActorRef =
context.actorOf(ZooKeeperOperationActor.props(zookeeper))
val childActor: ActorRef = context.actorOf(childProps, childName))
case ZooKeeperWatchEvent(e) =>
e.getState match {
case Expired => throw ZooKeeperSessionRestartException(None)
case _ =>
}
case cmd: ZKOperations.ZKCommand => zookeeperOperation forward cmd
case other => childActor forward other
}
override def postStop(): Unit = {
zookeeper.close()
super.postStop()
}
}
Throw exception on ZooKeeper
session expiration.
Note that ZooKeeper
application is valid only if its
session is valid.
Just let it crash on session
expiration.
Akka automatically restart
child actors. Stale states of
actors are refreshed on
restart.
ReactiveZooKeeper module

Edge case that breaks ID uniqueness
ZooKeeper
ID worker 1
ID worker 2
ID client 1
delete worker ID 1
session expired
NodeChildren
Changed
my worker ID
is 1
my worker ID
is 1
ID client 2
worker 1’s
ID is 1
worker 2’s
ID is 1
worker 1’s
ID is 1
worker 2’s
ID is 1
NodeChildren
Changed
network
partition
create worker ID 1
session timeout
ID duplication
risk!
network
recovered

Mitigate condition that breaks consensus
inferred by FLP impossibility
case ZooKeeperWatchEvent(e) =>
e.getState match {
case Expired => throw ZooKeeperSessionRestartException(None)
case Disconnected => throw new
ConnectionRecoveryTimeoutException(connectionTimeout)
case _ =>
}
case cmd: ZKOperations.ZKCommand => zookeeperOperation forward cmd
case other => childActor forward other
}
Fail fast on disconnection.
ReactiveZooKeeper module

Distributed ID generator in the wild

ID generation latency and throughput
The stress test used a gatling plugin for Akka Remote protocol.
https://github.com/chatwork/gatling-akka
Single ID worker performance

Effect of
ZooKeeper
node down
ID generation
throughput
The number of ID
workers connected
to the downed
ZooKeeper
Error count of ID
clients
# of affected
workers by
ZooKeeper
down
Effect
1/3 Not observable
2/3 Relatively high error rate
3/3 Throughput decrease
High error rate
We use redundant requests to mitigate effects of ID worker down.

Write API latency (average)
~6ms improvement.
The distributed ID
generator release

Write API latency (100, 95, 50 pt.)
The distributed ID
generator release
95 pt. is not much improved.

ID Generator latency (average)
The distributed ID
generator release
~8ms improvement.

ID Generator latency (100, 95, 50 pt.)
The distributed ID
generator release
Worse 95pt.
Possible fixes:
- use higher throughput for dispatcher
- use pinned dispatcher
- competitive redundant requests with
scatter gather pattern

We are hiring!
https://corp.chatwork.com/ja/recruit/

Distributed ID generator in ChatWork

Recommended

Recommended

More Related Content

Similar to Distributed ID generator in ChatWork

Similar to Distributed ID generator in ChatWork (20)

More from TanUkkii

More from TanUkkii (16)

Recently uploaded

Recently uploaded (20)

Distributed ID generator in ChatWork