My JeeConf talk regarding creating highload reactive server with netty.

1. Highload reactive server
with Netty

2. Dmitriy Dumanskiy
Blynk, CTO
Java blog : https://habrahabr.ru/users/doom369/topics
DOU : https://dou.ua/users/DOOM/articles/

3. Makers problem
+ = ?

4. Makers problem
● Http/s
● Mqtt
● WebSockets
● Own binary protocol

5. Blynk
10000 req/sec
3 VM * 2 cores, 60$
25% load
10k of local installations

6. Why netty?
Cassandra
Apache Spark
Elasticsearch
Graylog
Neo4j
Vert.x
HornetQInfinispan
Finagle
Async-http-client
Firebase
Akka
CouchbasePlay framework
Redisson

7. Why netty?
~700k servers

8. Why netty?
● Less GC

9. Why netty?
● Less GC
● Optimized for Linux based OS

10. Why netty?
● Less GC
● Optimized for Linux based OS
● High performance buffers

11. Why netty?
● Less GC
● Optimized for Linux based OS
● High performance buffers
● Well defined threading model

12. Why netty?
● Less GC
● Optimized for Linux based OS
● High performance buffers
● Well defined threading model
● HTTP, HTTP/2, SPDY, SCTP, TCP,
UDP, UDT, MQTT, etc

13. When to use?
● Performance is critical

14. When to use?
● Performance is critical
● Own protocol

15. When to use?
● Performance is critical
● Own protocol
● Full control over network
(so_reuseport, tcp_cork,
tcp_fastopen, tcp_nodelay, etc)

16. When to use?
● Performance is critical
● Own protocol
● Full control over network
● Game engines (agario, slither,
minecraft)

17. When to use?
● Performance is critical
● Own protocol
● Full control over network
● Game engines
● <3 reactive

18. Non-Blocking
● Few threads
● No context switching
● No memory consumption

19. Non-Blocking
new Channel
read / write
Selector
Thread
new Channel
read / write
new Channel
read / write

20. java.nio.channels.Selector
Selector selector = Selector.open();
channel.configureBlocking(false);
SelectionKey key = channel.register(selector, SelectionKey.OP_READ);
while(true) {
selector.select();
Set<SelectionKey> selectedKeys = selector.selectedKeys();
Iterator<SelectionKey> keyIterator = selectedKeys.iterator();
while(keyIterator.hasNext()) {
key = keyIterator.next();
if (key.isReadable()) { ... }
}
}

21. Selector selector = Selector.open(); // creating selector
channel.configureBlocking(false);
SelectionKey key = channel.register(selector, SelectionKey.OP_READ);
while(true) {
selector.select();
Set<SelectionKey> selectedKeys = selector.selectedKeys();
Iterator<SelectionKey> keyIterator = selectedKeys.iterator();
while(keyIterator.hasNext()) {
key = keyIterator.next();
if (key.isReadable()) { ... }
}
}

22. Selector selector = Selector.open();
channel.configureBlocking(false);
//registering channel with selector, listening for READ events only
SelectionKey key = channel.register(selector, SelectionKey.OP_READ);
while(true) {
selector.select();
Set<SelectionKey> selectedKeys = selector.selectedKeys();
Iterator<SelectionKey> keyIterator = selectedKeys.iterator();
while(keyIterator.hasNext()) {
key = keyIterator.next();
if (key.isReadable()) { ... }
}
}

23. Selector selector = Selector.open();
channel.configureBlocking(false);
SelectionKey key = channel.register(selector, SelectionKey.OP_READ);
while(true) {
selector.select(); //blocking until we get some READ events
Set<SelectionKey> selectedKeys = selector.selectedKeys();
Iterator<SelectionKey> keyIterator = selectedKeys.iterator();
while(keyIterator.hasNext()) {
key = keyIterator.next();
if (key.isReadable()) { ... }
}
}

24. Selector selector = Selector.open();
channel.configureBlocking(false);
SelectionKey key = channel.register(selector, SelectionKey.OP_READ);
while(true) {
selector.select();
//now we have channels with some data
Set<SelectionKey> selectedKeys = selector.selectedKeys();
Iterator<SelectionKey> keyIterator = selectedKeys.iterator();
while(keyIterator.hasNext()) {
key = keyIterator.next();
if (key.isReadable()) { ... }
}
}

25. Selector selector = Selector.open();
channel.configureBlocking(false);
SelectionKey key = channel.register(selector, SelectionKey.OP_READ);
while(true) {
selector.select();
Set<SelectionKey> selectedKeys = selector.selectedKeys();
Iterator<SelectionKey> keyIterator = selectedKeys.iterator();
while(keyIterator.hasNext()) {
key = keyIterator.next();
//do something with data
if (key.isReadable()) { key.channel() }
}
}

26. Flow
Selector
SelectionKey
Channel
ChannelPipeline

27. Flow
ChannelPipeline
fireEvent()
invokeChannelRead() executor.execute()
invokeChannelRead()

28. Minimal setup
ServerBootstrap b = new ServerBootstrap();
b.group(
new NioEventLoopGroup(1),
new NioEventLoopGroup()
) .channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer() {...});
ChannelFuture f = b.bind(8080).sync();
f.channel().closeFuture().sync();

29. Minimal setup
ServerBootstrap b = new ServerBootstrap();
b.group(
new NioEventLoopGroup(1), //IO thread
new NioEventLoopGroup()
) .channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer() {...});
ChannelFuture f = b.bind(8080).sync();
f.channel().closeFuture().sync();

30. Minimal setup
ServerBootstrap b = new ServerBootstrap();
b.group(
new NioEventLoopGroup(1),
new NioEventLoopGroup() //worker threads
) .channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer() {...});
ChannelFuture f = b.bind(8080).sync();
f.channel().closeFuture().sync();

31. Minimal setup
ServerBootstrap b = new ServerBootstrap();
b.group(
new NioEventLoopGroup(1),
new NioEventLoopGroup() //worker threads
) .channel(NioServerSocketChannel.class)
.childHandler(new ChannelInitializer() {...}); //pipeline init
ChannelFuture f = b.bind(8080).sync();
f.channel().closeFuture().sync();

32. Minimal setup
new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) {
final ChannelPipeline pipeline = ch.pipeline();
pipeline.addLast(new MyLogicHere());
}
};

33. ChannelPipeline

34. ChannelPipeline
● Inbound event ->
ChannelInboundHandler (CIHA)
● Outbound event ->
ChannelOutboundHandler (COHA)

35. ChannelInboundHandler
public interface ChannelInboundHandler extends ChannelHandler {
...
void channelRegistered(ChannelHandlerContext ctx);
void channelActive(ChannelHandlerContext ctx);
void channelRead(ChannelHandlerContext ctx, Object msg);
void userEventTriggered(ChannelHandlerContext ctx, Object evt);
void channelWritabilityChanged(ChannelHandlerContext ctx);
...
}

36. void initChannel(SocketChannel ch) {
ch.pipeline()
.addLast(new MyProtocolDecoder())
.addLast(new MyProtocolEncoder())
.addLast(new MyLogicHandler());
}
Own tcp/ip server

37. Channel
MyProtocolDecoder
MyLogicHandler
Own tcp/ip server
Channel
MyProtocolEncoder
MyLogicHandler

38. Handlers
HttpServerCodec
ChannelTrafficShapingHandler
IdleStateHandler
ReadTimeoutHandler
ChunkedWriteHandler
SslHandler
LoggingHandler
RuleBasedIpFilter
StringDecoder
JsonObjectDecoder
Base64DecoderJZlibDecoder
JZlibDecoder
Lz4FrameDecoder
ProtobufDecoder
ObjectDecoder
XmlFrameDecoder

39. void initChannel(SocketChannel ch) {
ch.pipeline()
.addLast(new HttpRequestDecoder())
.addLast(new HttpResponseEncoder())
.addLast(new MyHttpHandler());
}
Http Server

40. void initChannel(SocketChannel ch) {
ch.pipeline()
.addLast(new HttpServerCodec())
.addLast(new MyHttpHandler());
}
OR

41. void initChannel(SocketChannel ch) {
ch.pipeline()
.addLast(sslCtx.newHandler(ch.alloc()))
.addLast(new HttpServerCodec())
.addLast(new MyHttpHandler());
}
Https Server

42. void initChannel(SocketChannel ch) {
ch.pipeline()
.addLast(sslCtx.newHandler(ch.alloc()))
.addLast(new HttpServerCodec())
.addLast(new HttpContentCompressor())
.addLast(new MyHttpHandler());
}
Https Server + content gzip

43. @Override
public void channelRead(Context ctx, Object msg) {
//pass flow processing to next handler
super.channelRead(ctx, msg);
}
Pipeline flow

44. @Override
public void channelRead(Context ctx, Object msg) {
//stop request processing
return;
}
Pipeline flow

45. public void channelRead(Context ctx, Object msg) {
If (msg instanceOf LoginMessage) {
LoginMessage login = (LoginMessage) msg;
if (isSuperAdmin(login)) {
ctx.pipeline().remove(this);
ctx.pipeline().addLast(new SuperAdminHandler());
}
}
Pipeline flow on the fly

46. public void channelRead(Context ctx, Object msg) {
ChannelFuture cf = ctx.writeAndFlush(response);
cf.addListener(new ChannelFutureListener() {
@Override
public void complete(ChannelFuture future) {
future.channel().close();
}
});
}
Pipeline futures

47. @Override
public void channelRead(Context ctx, Object msg) {
ChannelFuture cf = ctx.writeAndFlush(response);
//close connection after message was delivered
cf.addListener(ChannelFutureListener.CLOSE);
}
Pipeline futures

48. @Override
public void channelRead(Context ctx, Object msg) {
...
ChannelFuture cf = ctx.writeAndFlush(response);
cf.addListener(future -> {
...
});
}
Pipeline futures

49. public void channelRead(Context ctx, Object msg) {
ChannelFuture cf = session.sendMsgToFriend(msg);
cf.addListener(new ChannelFutureListener() {
@Override
public void complete(ChannelFuture future) {
future.channel().writeAndFlush(“Delivered!”);
}
});
}
Pipeline futures

50. Pipeline blocking IO
Non blocking pools Blocking pools
IO Event Loops
DB
Worker Event Loops
Mailing
File system

51. public void channelRead(Context ctx, Object msg) {
if (msg instanceof HttpRequest) {
HttpRequest req = (HttpRequest) msg;
if (req.method() == GET && req.uri().eq(“/users”)) {
Users users = dbManager.userDao.getAllUsers();
ctx.writeAndFlush(new Response(users));
}
}
Pipeline blocking IO

52. public void channelRead(Context ctx, Object msg) {
if (msg instanceof HttpRequest) {
HttpRequest req = (HttpRequest) msg;
if (req.method() == POST && req.uri().eq(“/email”)) {
mailManager.sendEmail();
}
}
Pipeline blocking IO

53. public void channelRead(Context ctx, Object msg) {
if (msg instanceof HttpRequest) {
HttpRequest req = (HttpRequest) msg;
if (req.method() == GET && req.uri().eq(“/property”)) {
String property = fileManager.readProperty();
ctx.writeAndFlush(new Response(property));
}
}
}
Pipeline blocking IO

54. public void channelRead(Context ctx, Object msg) {
...
blockingThreadPool.execute(() -> {
Users users = dbManager.userDao.getAllUsers();
ctx.writeAndFlush(new Response(users));
});
}
Pipeline blocking IO

55. Pipeline blocking IO
● Thread.sleep()

56. Pipeline blocking IO
● Thread.sleep()
● java.util.concurrent.*

57. Pipeline blocking IO
● Thread.sleep()
● java.util.concurrent.*
● Intensive operations

58. Pipeline blocking IO
● Thread.sleep()
● java.util.concurrent.*
● Intensive operations
● Any blocking IO (files, db, smtp, etc)

59. Pipeline blocking IO
● Thread.sleep()
● java.util.concurrent.*
● Intensive operations
● Any blocking IO (files, db, smtp, etc)

60. @Override
public void channelInactive(Context ctx) {
HardwareState state = getState(ctx.channel());
if (state != null) {
ctx.executor().schedule(
new DelayedPush(state), state.period, SECONDS
);
}
}
EventLoop is Executor!

61. public void channelRead(Context ctx, Object msg) {
if (msg instanceof FullHttpRequest) {
FullHttpRequest request = (FullHttpRequest) msg;
User user = sessionDao.checkCookie(request);
...
}
super.channelRead(ctx, msg);
}
Request state

62. private static AttributeKey<User>
USER_KEY = AttributeKey.valueOf("user");
ctx.channel().attr(USER_KEY).set(user);
Request state

63. public void channelRead(Context ctx, Object msg) {
if (msg instanceof FullHttpRequest) {
FullHttpRequest request = (FullHttpRequest) msg;
User user = sessionDao.checkCookie(request);
ctx.channel().attr(USER_KEY).set(user);
}
super.channelRead(ctx, msg);
}
Request state

64. if (isSsl(in)) {
enableSsl(ctx);
} else {
if (isGzip()) {
enableGzip(ctx);
} else if (isHttp(in)) {
switchToHttp(ctx);
}
Port unification

66. Back pressure
if (channel.isWritable()) {
channel.writeAndFlush(msg);
}

67. Back pressure
BackPressureHandler
coming soon...

68. Performance

69. Performance
https://www.techempower.com/benchmarks/#section=data-r13&hw=ph&test=plaintext

70. <dependency>
<groupId>io.netty</groupId>
<artifactId>netty-transport-native-epoll</artifactId>
<version>${netty.version}</version>
<classifier>${os}</classifier>
</dependency>
Native transport

71. Bootstrap b = new Bootstrap();
b.group(new EpollEventLoopGroup());
b.channel(EpollSocketChannel.class);
Native transport

72. SslContextBuilder.forServer()
.sslProvider(SslProvider.OpenSsl);
JNI OpenSslEngine

73. <dependency>
<groupId>io.netty</groupId>
<artifactId>netty-tcnative-boringssl-static</artifactId>
<version>${netty.boring.ssl.version}</version>
<classifier>${os}</classifier>
</dependency>
JNI OpenSslEngine

74. ● Netty-tcnative
● netty-tcnative-libressl
● netty-tcnative-boringssl-static
JNI OpenSslEngine

75. Own ByteBuf

76. Own ByteBuf
● Reference counted
● Pooling by default
● Direct memory by default
● LeakDetector by default
● Reduced branches, range-checks

77. Own ByteBuf
● ByteBufAllocator.buffer(size);
● ctx.alloc().buffer(size);
● channel.alloc().buffer(size);

78. Less system calls
for (Message msg : messages) {
ctx.writeAndFlush(msg);
}

79. Less system calls
for (Message msg : messages) {
ctx.write(msg);
}
ctx.flush();

80. Thread Model
ChannelFuture inCf = ctx.deregister();
inCf.addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture cf) {
targetLoop.register(cf.channel())
.addListener(completeHandler);
}
});

81. Reusing Event Loop
new ServerBootstrap().group(
new EpollEventLoopGroup(1),
new EpollEventLoopGroup()
).bind(80);

82. Reusing Event Loop
EventLoopGroup boss = new EpollEventLoopGroup(1);
EventLoopGroup workers = new EpollEventLoopGroup();
new ServerBootstrap().group(
boss,
workers
).bind(80);
new ServerBootstrap().group(
boss,
workers
).bind(443);

83. Use direct buffers
ctx.writeAndFlush(
new ResponseMessage(messageId, OK)
);

84. Use direct buffers
ByteBuf buf = ctx.alloc().buffer(3);//pool
buf.writeByte(messageId);
buf.writeShort(OK);
ctx.writeAndFlush(buf);

85. Less allocations
ByteBuf msg = makeResponse(...);
msg.retain(targets.size() - 1);
for (Channel ch : targets) {
ch.writeAndFlush(msg);
}

86. Void promise
ctx.writeAndFlush(
response
);

87. Void promise
ctx.writeAndFlush(
response, ctx.voidPromise()
);

88. Reuse handlers
@Sharable
public class StringDecoder extends
MessageToMessageDecoder<ByteBuf> {
...
}

89. Prefer context
ctx.channel().writeAndFlush();

90. Prefer context
ctx.channel().writeAndFlush();
ctx.writeAndFlush();

91. Simpler - faster
ChannelInboundHandlerAdapter
does nothing, but fast

92. Simpler - faster
ByteToMessageDecoder
does some work, but slower

93. Simpler - faster
ReplayingDecoder
does job for you, but slowest

94. Turn off leak detection
ResourceLeakDetector.setLevel(
ResourceLeakDetector.Level.DISABLED);

95. What else?
● ASCIIString
● FastThreadLocal
● Unsafe
● Optimized Encoders

96. ● Really fast
● Low GC load
● Flexible
● Rapidly evolve
● Cool support
Summary

97. ● Hard
● Memory leaks
● Still have issues
Summary

98. https://github.com/blynkkk/blynk-server