Apache cassandra 4.0 nio Report

1. Apache Cassandra 4.0 NIO MessagingService
Japan Cassandra Community
Cassandra Study
Kazutaka Tomita

2. SELF INTRODUCTION
Kazutaka Tomita (@railute)
• INTHEFOREST Co., Ltd. CEO/CTO
• Consulting for Apache Cassandra and Apache Spark Systems
• Supporting for Apache Cassandra in Japan
• An organizer of Cassandra Summit Tokyo
Specialty
• RDBMS (Oracle,SQLServer,MySQL,PostgreSQL)
• Apache Cassandra
• Apache Spark
• Apache Hadoop with YARN
• And the other NoSQL
• NLP and Text mining for Japanese

3. Background
The internode networking on Apache Cassandra
Changed Points

4. Introduction
Background
PURPOS
E
In Apache Cassandra 4.0, the first major changes
to the network layer have been implemented.
The purpose of this slide is reviewing this change
in order to consider how it will affect Apache
Cassandra.

5. Background
Background
Background for Changed
As the number of nodes in the cluster increases, the
performance impact from the context switch becomes more
apparent.
Value Percentile TotalCount 1/(1-Percentile)
0.000 0.000000000000 5554
1.00
~snip~
1634.303 0.987500000000 1229643
80.00
1688.575 0.989062500000 1231596
91.43
1756.159 0.990625000000 1233523
106.67
~snip~
148176371.711 0.999999237061 1245183
1310720.00
148176371.711 1.000000000000 1245183
Value Percentile TotalCount 1/(1-Percentile)
0.000 0.000000000000 21883 1.00
~snip~
875.519 0.987500000000 3494765 80.00
935.423 0.989062500000 3500258 91.43
1010.175 0.990625000000 3505780 106.67
~snip~
186562641.919 0.999999761581 3538943 4194304.00
186562641.919 1.000000000000 3538943
*From CASSANDRA-8457
75% of messages within 100 microseconds and 99% within 1
millisecond.
50% of messages are sent within 100 microseconds of
each other and 92% are sent within one millisecond.
Before adaptation After adaptation

6. NetworkService
Networking Structure
The internode networking on Apache Cassandra
C* C*
C*
Inbound
Outbound
Gossip
Small
Large
(URGENT)
Client
The Networking Structure on Apache Cassandra

7. Non-Blocking IO
The internode networking on Apache Cassandra
REQUESTS
EVENT
LOOP
SINGLE
THREAD
DELEGATE
ASYNC
THREADS C*

8. Notification
The internode networking on Apache Cassandra
C* C*
C*
Client
The implement of NIO in 4.0 applies only to inter
nodes.
Already applied between client and coordinator nodes when implementing native
protocol The benefit is not available without proper OS configuration
beforehand, such as proper IRQ configuration or jumbo frame
adaptation.

9. Before 3.X
Changed Points
/**
* Listen on the specified port.
*
* @param localEp InetAddress whose port to listen on.
*/
private void listen(InetAddress localEp) throws ConfigurationException
{
for (ServerSocket ss : getServerSockets(localEp))
{
SocketThread th = new SocketThread(ss, "ACCEPT-" + localEp);
th.start();
socketThreads.add(th);
}
}
3.11.10
/**
* Listen on the specified port.
* @param localEp InetAddress whose port to listen on.
*/
public void listen(InetAddress localEp) throws IOException
{
ServerSocketChannel serverChannel = ServerSocketChannel.open();
final ServerSocket ss = serverChannel.socket();
ss.setReuseAddress(true);
ss.bind(new InetSocketAddress(localEp, DatabaseDescriptor.getStoragePort()));
socketThread = new SocketThread(ss, "ACCEPT-" + localEp);
socketThread.start();
listenGate.signalAll();
}
/**
* Listen on the specified port.
* @param localEp InetAddress whose port to listen on.
*/public void listen(InetAddress localEp) throws IOException, ConfigurationException
{
callbacks.reset(); // hack to allow tests to stop/restart MS
for (ServerSocket ss: getServerSocket(localEp))
{
SocketThread th = new SocketThread(ss, "ACCEPT-" + localEp);
th.start();
socketThreads.add(th);
}
listenGate.signalAll();
}
1.0.12
0.6.13
org.apache.cassandra.net.MessagingService

10. After 4.0
Changed Points
public void listen()
{
inboundSockets.open();
}
4.0-beta4
org.apache.cassandra.net.InboundSocket
org.apache.cassandra.net.MessagingService

11. Summary
 The bottom layer has been completely changed, so the impact is significant.
 When used in public clouds including AWS, we should take care in tuning for
servers.