Moiz Raja's talk on MD-SAL clustering at ODL meetup in Silicon Valley on April 27, 2015.

1. www.opendaylight.org
odl-mdsal-clustering
Moiz Raja
OpenDaylight Silicon Valley Meetup
04/28/2015

2. www.opendaylight.org
Topics
▪ Components
▪ Requirements
▪ Design
▪ Testing
▪ Monitoring
▪ Challenges
▪ Insights
▪ What’s missing

3. www.opendaylight.org
Components

4. www.opendaylight.org
▪Remote RPC
▪Distributed Data Store
▪Remote Notifications ???

5. www.opendaylight.org
Requirements

6. www.opendaylight.org
▪Location Transparency
▪Drop in replacement for IMDS
▪Persistence
▪Strongly Consistency
▪Data Change Notifications
▪Static configuration
▪Sharding capabilities

7. www.opendaylight.org
Design

8. www.opendaylight.org
Data Broker
Config DataStore Operational DataStore

9. www.opendaylight.org
Data Broker
IMDS IMDS

10. www.opendaylight.org
Data Broker
CDS CDS

11. www.opendaylight.org
org.opendaylight.controller.sal.core.spi.data.DOMStore
DistributedDataStore

12. www.opendaylight.org
Distributed Data
Store
Data Broker Inventory
Shard
Topology
Shard
Default
Shard
Shard
Manager

13. www.opendaylight.org
Distributed Data
Store
Data Broker Inventory
Shard
Transaction Proxy
Transaction Context Shard Transaction
Shard
Manager
modification
1
2
3
4
5
6
txn = db.newReadWriteTransaction();
txn.merge(LogicalDataStoreType.OPERATIONAL, id,
node)

14. www.opendaylight.org
Akka Remoting
Akka
Clustering Akka
Persistence
Raft Actor
Shard Consistency
Persistence
Remote Communication
Discovery
State Holder

15. www.opendaylight.org
Akka
Remoting
Akka
Clustering Akka
Persistence
Raft Actor
Shard – Replica 1
Akka
Remoting
Akka
Clustering Akka
Persistence
Raft Actor
Shard – Replica 2
member-1 (10.194.126.242) member-2 (10.194.126.243)
member-2 is on 10.194.126.243 member-1 is on 10.194.126.242

16. www.opendaylight.org
We are here

17. www.opendaylight.org
What about Availability?
▪ We are available as long as a majority of the replicas are connected
▪ (N / 2) + 1 out of an N node cluster
▪ 2 out of 3 nodes in a 3 node cluster
▪ 3 out of 4 nodes in a 4 node cluster
▪ 3 out of 5 nodes in a 5 node cluster
▪ and so on

18. www.opendaylight.org
Configuration/RAFT determine availability
▪ If inventory is configured to be on,
▪ member-1
▪ member-2
▪ member-3
▪then atleast 2 of those members need to be running for the inventory data cluster to
be available
▪ If inventory in only configured to be on,
▪ member-1
▪then just member-1 needs to be running for the inventory data cluster to be available

19. www.opendaylight.org
Testing
▪ Unit tests (> 80% code coverage)
▪ Integration test (aka car-people test) for testing HA/Failover in a real cluster
▪ dsBenchmark for performance testing
▪ Dummy Datastore for testing replication overhead
▪ Raft test driver for testing the Raft implementation on a single box
▪ Other Performance/Scale tests
▪ BGP using exabgp and some other test scripts
▪ PCEP using pcc-mock
▪ Netconf using the netconf simulator
▪ Cbench for openflow

20. www.opendaylight.org
Monitoring
▪ Mbeans
▪ org.opendaylight.controller
▪DistributedConfigDataStore
▪DistributedOperationalDataStore
▪ org.opendaylight.controller.actor.metric
▪ org.opendaylight.controller.cluster.datastore

21. www.opendaylight.org

22. www.opendaylight.org

23. www.opendaylight.org

24. www.opendaylight.org

25. www.opendaylight.org
Challenges
▪ API
▪ Serialization
▪ Memory
▪ Messaging and Context Switching
▪ Back Pressure
▪ Operations
▪ Transactions
▪ Remoting Latencies
▪ Persistence Latencies

26. www.opendaylight.org
Insights
▪ Bulk transactions work best
▪ Avoid multiple writers
▪ Try to write to only one shard in a transaction

27. www.opendaylight.org
What’s missing?
▪ Remote Notifications
▪ Dynamic addition/removal of servers
▪ Fine grained sharding

28. www.opendaylight.org
The End!