We will show how to write application in Java 8 that do not waste resources and which can maximize effective utilization of CPU/RAM. There will be presented comparison of blocking and non-blocking approach for I/O and application services. Based on microservices implementing simple business logic in security/cryptography/payments domain, we will demonstrate following aspects: * NIO at all edges of application * popular libraries that support NIO * single instance scalability * performance metrics (incl. throughput and latency) * resources utilization * code readability with CompletableFuture * application maintenance and debugging All above based on our experiences gathered during development of software platforms at Oberthur Technologies R&D Poland.

1. Practical
non-blocking microservices
in Java 8
Michał Baliński
System Architect
Oleksandr Goldobin
System Architect

2. Cloud of microservices for secure IoT
gateway backing
service
core
service
gateway
gateway
core
service
backing
service

3. The demand and characteristics of our domain
Crowded IoT environment
Slow, long lived connections
Big amount of concurrent connections
Scalability and resilience
Rather I/O intensive than CPU intensive

4. OTA Gateway – Use Case
TSM
Trusted
Service
Manager

5. OTA Gateway – Use Case
TSM
Trusted
Service
Manager
Security Module

6. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB

7. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1. submit
scripts

8. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1. submit
scripts
HTTP
2. encrypt
scripts

9. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1. submit
scripts
HTTP
2. encrypt
scripts
TCP
3. store
scripts

10. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1. submit
scripts
HTTP
2. encrypt
scripts
TCP
4. submition
response
3. store
scripts

11. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1. submit
scripts
HTTP
2. encrypt
scripts
TCP
4. submition
response
HTTP
5. poll
scripts
3. store
scripts

12. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1. submit
scripts
HTTP
2. encrypt
scripts
TCP
3. store
scripts
4. submition
response
HTTP
5. poll
scripts
6. search
scripts

13. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP
1. submit
scripts
HTTP
2. encrypt
scripts
TCP
3. store
scripts
4. submition
response
HTTP
5. poll
scripts
6. search
scripts
7. response
with scripts

14. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
HTTP HTTP
HTTP
TCP

15. OTA Gateway – Use Case
OTA
(Over-the-Air)
Gateway
TSM
Trusted
Service
Manager
Security Module
DB
Log
Storage
HTTP HTTP
HTTP
TCP File I/OMonitoring
System
HTTP

16. Let’s test blocking approach!

17. OTA Gateway Blocking - technologies
17
TSM
DB Log
Storage
HTTP
TCP STDOUT
OTA
Gateway
JAX-RS
Logback
appender
Security Module
JAX-RS
Jedis
JAX-RS Client

18. OTA Gateway – Blocking - test
OTA
(Over-the-Air)
Gateway
send
2 scripts per
request
Security Module
DB
Log
Storage
HTTP HTTP
HTTP
TCP File I/O
emulated
latency
200 ms
expected
latency
< 450 ms
verified with
throughput
over 7k req/s

19. Blocking – 1k threads
OTA
(Over-the-Air)
Gateway
send
2 scripts per
request
Security Module
DB
Log
Storage
HTTP HTTP
HTTP
TCP File I/O
emulated
latency
200 ms
max 1000
connections
max 1000
threads
expected
latency
< 450 ms

20. Blocking – 1k threads
500 req/s 1000 req/s 1500 req/s 2000 req/s

21. The drawbacks of classic synchronous I/O
One thread per connection
Threads waiting instead of running
Context switches
Resource wasting (~1 MB per thread - 64bit)

22. Let’s switch from blocking to non-blocking

23. OTA Gateway Non-blocking - technologies
23
TSM
DB Log
Storage
HTTP
TCP STDOUT
OTA
Gateway
JAX-RS 2.0
Logback
async
appender
Async Http Client 2
(Netty)
Security Module
JAX-RS 2.0
Lettuce
(Netty)

24. Non-blocking – 16 threads
OTA
(Over-the-Air)
Gateway
send
2 scripts per
request
Security Module
DB
Log
Storage
HTTP HTTP
HTTP
TCP File I/O
emulated
latency
200 ms
no limit for
connections
max 16
threads
expected
latency
< 450 ms

25. Non-blocking – 16 threads
1k req/s 2k req/s 2.5k req/s 3k req/s

26. Blocking Non-blocking
1000 threads 56 threads
1.2 GB 0.5 GB
~1.2k r/s 2.5k r/s

27. Let’s talk about challenges

28. OTA Gateway Blocking – sequence diagram
OTA
Gateway
TSM
Security
Module DB Logs
loop
1. submit
scripts
2. encrypt
script
3a. store
script
4. submition
response
3b. count
scripts

29. OTA Gateway Non-blocking – sequence diagram
OTA
Gateway
TSM
Security
Module DB Logs
loop
1. submit
scripts
2. encrypt
script
3a. store
script
4. submition
response
3b. count
scripts

30. OTA Non-blocking – realityOTA
Gateway
TSM
Security
Module DB Logs
„loopedprocessingchain”
1. submit
scripts
4. submition
response
3b. count
scripts
HTTP
Server
2. encrypt
script
3a. store
script
Logging
DB
Client
Security
Client

31. Code. Bird view
3110 October 2016
package org.demo.ota.blocking.rest;
@Path("se")
public class ScriptSubmissionResource extends Application {
private static final Logger log = LoggerFactory.getLogger(ScriptSubmissionResource.class);
private static final ResourceMetrics METRICS = new ResourceMetrics("ota_submission");
private SecureModuleClient secureModuleClient = SecureModuleClient.instance();
private ScriptStorageClient scriptStorageClient = ScriptStorageClient.instance();
@POST
@Path("/{seId}/scripts")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.TEXT_PLAIN)
public long submitScripts(@PathParam("seId") String seId, List<Script> scripts) {
MDC.put("flow", "submission");
MDC.put("se", seId);
return METRICS.instrument(() -> {
log.debug("Processing {} scripts submission", scripts.size(), seId);
for (int i = 0; i < scripts.size(); i++) {
final Script script = scripts.get(i);
log.debug("Encrypting {} script", i);
final String encryptedPayload = secureModuleClient.encrypt(seId, script.getPayload());
script.setPayload(encryptedPayload);
log.debug("Storing encrypted script {}", i);
scriptStorageClient.storeScript(seId, script);
}
long numberOfScripts = scriptStorageClient.numberOfScriptsForSe(seId);
log.debug("Request processed", seId);
return numberOfScripts;
});
}
@Override
public Set<Object> getSingletons() {
return Collections.singleton(this);
}
}
package org.demo.ota.nonblocking.rest;
@Path("se")
public class ScriptSubmissionResource extends Application {
private static final Logger log = LoggerFactory.getLogger(ScriptSubmissionResource.class);
private static final ResourceMetrics METRICS = new ResourceMetrics("ota_submission");
private SecureModuleClient secureModuleClient = SecureModuleClient.instance();
private ScriptStorageClient scriptStorageClient = ScriptStorageClient.instance();
@POST
@Path("/{seId}/scripts")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.TEXT_PLAIN)
public void submitScripts(
@PathParam("seId") String seId,
List<Script> scripts,
@Suspended final AsyncResponse asyncResponse
) {
final DiagnosticContext diagnosticContext = new DiagnosticContext("submission", seId);
METRICS.instrumentStage(() -> {
log.debug("{} Processing {} scripts submission", diagnosticContext, scripts.size());
return
encryptAndStoreAllScripts(diagnosticContext, seId, scripts)
.thenCompose(
ignore -> scriptStorageClient.numberOfScriptsForSe(seId)
);
})
.whenComplete((numberOfScripts, e) -> {
if (e != null) {
asyncResponse.resume(e);
} else {
log.debug("{} Request processed", diagnosticContext);
asyncResponse.resume(numberOfScripts);
}
});
}
private CompletionStage<Void> encryptAndStoreAllScripts(
final DiagnosticContext diagnosticContext,
final String seId,
final List<Script> scripts
) {
CompletionStage<Void> stage = null; // <- non final field, potential concurrent access bug!
for (int i = 0; i < scripts.size(); i++) {
final int scriptIndex = i;
final Script script = scripts.get(scriptIndex);
if (stage == null) {
stage = encryptAndStoreSingleScript(diagnosticContext, seId, scriptIndex, script);
} else {
stage = stage.thenCompose(ignore ->
encryptAndStoreSingleScript(diagnosticContext, seId, scriptIndex, script));
}
}
return stage;
}
private CompletionStage<Void> encryptAndStoreSingleScript(
final DiagnosticContext diagnosticContext,
final String seId,
final int scriptIndex,
final Script script
) {
log.debug("{} Encrypting script {}", diagnosticContext, scriptIndex);
return secureModuleClient
.encrypt(seId, script.getPayload())
.thenCompose(
encryptedPayload -> {
log.debug("{} Storing encrypted script {}", diagnosticContext, scriptIndex);
return scriptStorageClient.storeScript(seId, new Script(encryptedPayload));
}
);
}
@Override
public Set<Object> getSingletons() {
return new HashSet<>(Collections.singletonList(this));
}
}

32. Code. Blocking. Submission 1
10 October 2016 32
@POST
@Path("/{seId}/scripts")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.TEXT_PLAIN)
public
long submitScripts(@PathParam("seId") String seId, List<Script> scripts) {
MDC.put("flow", "submission"); // ß Setting diagnostic context
MDC.put("se", seId);
return METRICS.instrument(() -> { // ß Instrumenting with metrics
//...

33. Code. Blocking. Submission 2
10 October 2016 33
log.debug("Processing {} scripts submission", scripts.size(), seId);
for (int i = 0; i < scripts.size(); i++) { ß Cycle through the
scripts
final Script script = scripts.get(i);
log.debug("Encrypting {} script", i);
final String encryptedPayload = secureModuleClient
.encrypt(seId, script.getPayload()); ß Encrypting the script
script.setPayload(encryptedPayload);
log.debug("Storing encrypted script {}", i);
scriptStorageClient.storeScript(seId, script); ß Saving the script into
DB
}
long numberOfScripts =
scriptStorageClient.numberOfScriptsForSe(seId); ß Getting current number
of scripts in DB
log.debug("Request processed", seId);
return numberOfScripts;

34. Code. Non-blocking. Submission 1
10 October 2016 34
@POST
@Path("/{seId}/scripts")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.TEXT_PLAIN)
public void submitScripts(
@PathParam("seId") String seId,
List<Script> scripts,
@Suspended final AsyncResponse asyncResponse
) {
final DiagnosticContext diagnosticContext =
new DiagnosticContext("submission", seId); ß Creating diagnostic
context
METRICS.instrumentStage(() -> { ß Instrumenting with metrics

35. Code. Non-blocking. Submission 1
10 October 2016 35
@POST
@Path("/{seId}/scripts")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.TEXT_PLAIN)
public void submitScripts(
@PathParam("seId") String seId,
List<Script> scripts,
@Suspended final AsyncResponse asyncResponse
) {
final DiagnosticContext diagnosticContext =
new DiagnosticContext("submission", seId); ß Creating diagnostic
context
METRICS.instrumentStage(() -> { ß Instrumenting with metrics

36. Code. Non-blocking. Submission 2
10 October 2016 36
METRICS.instrumentStage(() -> {
log.debug(
"{} Processing {} scripts submission",
diagnosticContext,
scripts.size());
return
encryptAndStoreAllScripts(diagnosticContext, seId, scripts)
.thenCompose(
ignore ->
scriptStorageClient.numberOfScriptsForSe(seId)
);
})
.whenComplete((numberOfScripts, e) -> {
if (e != null) {
asyncResponse.resume(e);
} else {
log.debug("{} Request processed", diagnosticContext);
asyncResponse.resume(numberOfScripts);
}
});

37. Code. Non-blocking. Submission 3
10 October 2016 37
private CompletionStage<Void> encryptAndStoreAllScripts(
final DiagnosticContext diagnosticContext,
final String seId,
final List<Script> scripts
) {
CompletionStage<Void> stage = null; // <- non final field, potential
// concurrent access bug!
for (int i = 0; i < scripts.size(); i++) { ß Cycle through the
scripts
final int scriptIndex = i;
final Script script = scripts.get(scriptIndex);
if (stage == null) {
stage = encryptAndStoreSingleScript(
diagnosticContext, seId, scriptIndex, script);
} else {
stage = stage.thenCompose(ignore ->
encryptAndStoreSingleScript(
diagnosticContext, seId, scriptIndex, script));
}
}
return stage;
}

38. Code. Non-blocking. Submission 4
10 October 2016 38
private CompletionStage<Void> encryptAndStoreSingleScript(
final DiagnosticContext diagnosticContext,
final String seId,
final int scriptIndex,
final Script script
) {
log.debug("{} Encrypting script {}", diagnosticContext, scriptIndex);
return secureModuleClient
.encrypt(seId, script.getPayload()) ß Encrypting the script
.thenCompose(
encryptedPayload -> {
log.debug(
"{} Storing encrypted script {}",
diagnosticContext,
scriptIndex);
return
scriptStorageClient.storeScript( ß Saving the script into
seId, the DB
new Script(encryptedPayload));
}
);
}

39. Code. Blocking. Integration
10 October 2016 39
private final JedisPool pool;
public void storeScript(String seId, Script script) {
try (Jedis jedis = pool.getResource()) {
jedis.rpush(seId, script.getPayload());
}
}
public long numberOfScriptsForSe(String seId) {
try (Jedis jedis = pool.getResource()) {
return jedis.llen(seId);
}
}
public Optional<Script> nextScript(String seId) {
try (Jedis jedis = pool.getResource()) {
return Optional.ofNullable(jedis.lpop(seId)).map(Script::new);
}
}
public String encrypt(String keyDiversifier, String payload) {
// ...
}

40. Code. Non-Blocking. Integration
10 October 2016 40
private final RedisAsyncCommands<String, String> commands;
public CompletionStage<Void> storeScript(String seId, Script script) {
return commands
.rpush(seId, script.getPayload())
.thenApply(ignore -> null);
}
public CompletionStage<Long> numberOfScriptsForSe(String seId) {
return commands.llen(seId);
}
public CompletionStage<Optional<String>> nextScript(String seId) {
return commands.lpop(seId).thenApply(Optional::ofNullable);
}
public CompletionStage<String> encrypt(String keyDiversifier, String payload)
{
// ...
}

41. Diagnostics in non-blocking systems
No clear stack traces, need for good logs
Name your threads properly
MDC becomes useless (thread locals)
Explicitly pass debug context to trace flows
Be prepared for debuging non-obvious errors

42. NIO technology landscape
JDK 1.4
NIO
JDK 1.7
NIO.2
JAX-RS 2.x
Servlet API 3.x

43. Lessons learned, part 1
Vanila Java 8 for NIO µ-services
Netty best for custom protocols in NIO
Unit tests should be synchronous
Load/stress testing is a must
Make bulkheading and plan your resources

44. Lessons learned, part 2
Functional programming patterns for readability
Immutability as 1-st class citizen
Scala may be good choice ;-)

45. Conclusion
Non-blocking processing can really
save your resources (== money)
But!
Weight all pros and cons and use non-blocking processing
only if you really need it.

46. Thank you.
Michał Baliński
m.balinski@oberthur.com
Oleksandr Goldobin
o.goldobin@oberthur.com
goldobin
@goldobin
balonus
@MichalBalinski
Readings:
• https://github.com/balonus/blocking-vs-nonblocking-demo
• C10k Problem, C10M Problem, Asynchronous I/O
• Boost application performance using asynchronous I/O (M. Tim Jones, 2006)
• Zuul 2 : The Netflix Journey to Asynchronous, Non-Blocking Systems (Netflix, 2016)
• Thousands of Threads and Blocking I/O: The Old Way to Write Java Servers Is New
Again (and Way Better) (Paul Tyma, 2008)
• Why Non-Blocking? (Bozhidar Bozhanov, 2011)