Scale Up with Lock-Free Algorithms, presented at JavaOne 2017

1. Scale Up with
Lock-Free Algorithms
Non-blocking concurrency on JVM
Presented at JavaOne 2017
/Roman Elizarov @ JetBrains

2. Speaker: Roman Elizarov
• 16+ years experience
• Previously developed high-perf trading software
@ Devexperts
• Teach concurrent & distributed programming
@ St. Petersburg ITMO University
• Chief judge
@ Northern Eurasia Contest / ACM ICPC
• Now work on Kotlin
@ JetBrains

3. Shared

4. Shared Mutable

5. Shared Mutable State

8. Shared Mutable State
Why?

9. Big Big Data

10. Data 1 Data 2 Data N

11. Data 1 Data 2 Data N
map map map

12. Data 1 Data 2 Data N
map map map
reduce
answer

13. Embarrassingly parallel
Data 1 Data 2 Data N
map map map
reduce
answer

14. Big Big Data

15. Big Big Data
Real-time

16. Big Big Data
Real-time
Concurrent requests / processing

17. Big Big Data
Real-time
Concurrent requests / processing
Performance Scalability

19. A toy problem

20. A toy problem – stack

21. A toy problem – stack
class Node<T>(val next: Node<T>?, val value: T)

22. public final class Node<T> {
private final Node<T> next;
private final T value;
public Node(Node<T> next, T value) {
this.next = next;
this.value = value;
}
public Node<T> getNext() {
return next;
}
public T getValue() {
return value;
}
}
A toy problem – stack

23. A toy problem – stack
class Node<T>(val next: Node<T>?, val value: T)

24. A toy problem – empty stack
class Node<T>(val next: Node<T>?, val value: T)
top

25. A toy problem – stack push
class Node<T>(val next: Node<T>?, val value: T)
next = null
value = 1
top А

26. A toy problem – stack push
class Node<T>(val next: Node<T>?, val value: T)
next = null
value = 1
top А
next = A
value = 2
B

27. A toy problem – stack push
class Node<T>(val next: Node<T>?, val value: T)
next = null
value = 1
top А
next = A
value = 2
B

28. A toy problem – stack push
class Node<T>(val next: Node<T>?, val value: T)
next = A
value = 2
top
next = null
value = 1
B A

29. A toy problem – stack push
class Node<T>(val next: Node<T>?, val value: T)
class LinkedStack<T> {
private var top: Node<T>? = null
fun push(value: T) {
top = Node(top, value)
}
}

30. A toy problem – stack push
class Node<T>(val next: Node<T>?, val value: T)
class LinkedStack<T> {
private var top: Node<T>? = null
fun push(value: T) {
top = Node(top, value)
}
}

31. A toy problem – stack push
class Node<T>(val next: Node<T>?, val value: T)
class LinkedStack<T> {
private var top: Node<T>? = null
fun push(value: T) {
top = Node(top, value)
}
}

32. A toy problem – stack
class Node<T>(val next: Node<T>?, val value: T)
next = A
value = 2
top
next = null
value = 1
B A

33. A toy problem – stack pop
class Node<T>(val next: Node<T>?, val value: T)
next = A
value = 2
top
next = null
value = 1
B A
cur

34. A toy problem – stack pop
class Node<T>(val next: Node<T>?, val value: T)
next = A
value = 2
top
next = null
value = 1
B A
cur

35. A toy problem – stack pop
class Node<T>(val next: Node<T>?, val value: T)
next = A
value = 2
top
next = null
value = 1
B A
cur
result = 2

36. A toy problem – stack pop
class Node<T>(val next: Node<T>?, val value: T)
class LinkedStack<T> {
private var top: Node<T>? = null
fun push(value: T) {
top = Node(top, value)
}
fun pop(): T? {
val cur = top ?: return null
top = cur.next
return cur.value
}
}

37. A toy problem – stack
class Node<T>(val next: Node<T>?, val value: T)
class LinkedStack<T> {
private var top: Node<T>? = null
fun push(value: T) {
top = Node(top, value)
}
fun pop(): T? {
val cur = top ?: return null
top = cur.next
return cur.value
}
}

38. Does it work?

39. A toy problem – concurrent push
class Node<T>(val next: Node<T>?, val value: T)
next = null
value = 1
top А
next = A
value = 2
B

40. A toy problem – concurrent push
class Node<T>(val next: Node<T>?, val value: T)
next = null
value = 1
top А
next = A
value = 2
B
next = A
value = 3
C

41. A toy problem – concurrent push
class Node<T>(val next: Node<T>?, val value: T)
next = null
value = 1
top А
next = A
value = 2
B
next = A
value = 3
C

42. A toy problem – synchronized stack
class Node<T>(val next: Node<T>?, val value: T)
class LinkedStack<T> {
private var top: Node<T>? = null
@Synchronized
fun push(value: T) {
top = Node(top, value)
}
@Synchronized
fun pop(): T? {
val cur = top ?: return null
top = cur.next
return cur.value
}
}

43. Does it scale?

44. Benchmark
@State(Scope.Benchmark)
open class LinkedStackBenchmark {
private val stack = LinkedStack<Int>()
@Benchmark
fun benchmark() {
stack.push(1)
check(stack.pop() == 1)
}
}

45. Benchmark
@State(Scope.Benchmark)
open class LinkedStackBenchmark {
private val stack = LinkedStack<Int>()
@Benchmark
fun benchmark() {
stack.push(1)
check(stack.pop() == 1)
}
}

46. Benchmark results
0
5
10
15
20
25
1 2 4 8 16 32 64 128
Millions
Number of threads
Throughput (ops/s)
LinkedStack
Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz; 32 HW threads; Java HotSpot(TM) 64-Bit Server VM (build 9+181, mixed mode)

47. Contention
P
Q
pop1

48. Contention
P
Q
pop1

49. Contention
P
Q
pop1
pop2
wait

50. Contention
P
Q work
pop1
pop2
wait

51. Deadlocks

52. Lock-free?

53. Lock-free push
class Node<T>(val next: Node<T>?, val value: T)
next = null
value = 1
top А
next = A
value = 2
B
expect

54. Lock-free push
class Node<T>(val next: Node<T>?, val value: T)
next = null
value = 1
top А
next = A
value = 2
B
update

55. Lock-free push
class Node<T>(val next: Node<T>?, val value: T)
next = null
value = 1
top А
next = A
value = 2
B
update
expect

56. AtomicReference
package java.util.concurrent.atomic;
/** @since 1.5 */
public class AtomicReference<V> {
private volatile V value;
public V get() {
return value;
}
public boolean compareAndSet(V expect, V update) {
// …
}
}

57. AtomicReference
package java.util.concurrent.atomic;
/** @since 1.5 */
public class AtomicReference<V> {
private volatile V value;
public V get() {
return value;
}
public boolean compareAndSet(V expect, V update) {
// …
}
}

58. AtomicReference
package java.util.concurrent.atomic;
/** @since 1.5 */
public class AtomicReference<V> {
private volatile V value;
public V get() {
return value;
}
public boolean compareAndSet(V expect, V update) {
// …
}
}

59. Using AtomicReference
class LockFree<T> {
private val top = AtomicReference<Node<T>?>(null)
fun push(value: T) {
while (true) {
val cur = top.get()
val upd = Node(cur, value)
if (top.compareAndSet(cur, upd)) return
}
}
}

60. Using AtomicReference
class LockFree<T> {
private val top = AtomicReference<Node<T>?>(null)
fun push(value: T) {
while (true) {
val cur = top.get()
val upd = Node(cur, value)
if (top.compareAndSet(cur, upd)) return
}
}
}

61. Using AtomicReference - push
class LockFree<T> {
private val top = AtomicReference<Node<T>?>(null)
fun push(value: T) {
while (true) {
val cur = top.get()
val upd = Node(cur, value)
if (top.compareAndSet(cur, upd)) return
}
}
}
1

62. Using AtomicReference - push
class LockFree<T> {
private val top = AtomicReference<Node<T>?>(null)
fun push(value: T) {
while (true) {
val cur = top.get()
val upd = Node(cur, value)
if (top.compareAndSet(cur, upd)) return
}
}
}
1
2

63. Using AtomicReference
class LockFree<T> {
private val top = AtomicReference<Node<T>?>(null)
fun push(value: T) {
while (true) {
val cur = top.get()
val upd = Node(cur, value)
if (top.compareAndSet(cur, upd)) return
}
}
}
1
2
3

64. Using AtomicReference - push
1
2
3
class LockFree<T> {
private val top = AtomicReference<Node<T>?>(null)
fun push(value: T) {
while (true) {
val cur = top.get()
val upd = Node(cur, value)
if (top.compareAndSet(cur, upd)) return
}
}
}

65. Powerful we have become!

66. Using AtomicReference - pop
class LockFree<T> {
private val top = AtomicReference<Node<T>?>(null)
fun push(value: T) { … }
fun pop(): T? {
while (true) {
val cur = top.get() ?: return null
if (top.compareAndSet(cur, cur.next)) return cur.value
}
}
}

67. Using AtomicReference
class LockFree<T> {
private val top = AtomicReference<Node<T>?>(null)
fun push(value: T) { … }
fun pop(): T? { … }
}

68. It’s a trap

69. Using volatile variable
class LinkedStackLF<T> {
@Volatile
private var top: Node<T>? = null
fun push(value: T) {
// ...
}
}

70. Using AtomicReferenceFieldUpdater
package java.util.concurrent.atomic;
/** @since 1.5 */
public abstract class AtomicReferenceFieldUpdater<T,V> {
public static <U,W> AtomicReferenceFieldUpdater<U,W> newUpdater(
Class<U> tclass, Class<W> vclass, String fieldName;
public abstract boolean compareAndSet(T obj, V expect, V update;
}

71. Using AtomicReferenceFieldUpdater
private volatile Node<T> top;

72. Using AtomicReferenceFieldUpdater
private volatile Node<T> top;
private static final AtomicReferenceFieldUpdater<LockFree, Node>
TOP = AtomicReferenceFieldUpdater
.newUpdater(LockFree.class, Node.class, "top");

73. Using AtomicReferenceFieldUpdater
private volatile Node<T> top;
private static final AtomicReferenceFieldUpdater<LockFree, Node>
TOP = AtomicReferenceFieldUpdater
.newUpdater(LockFree.class, Node.class, "top");
if (TOP.compareAndSet(this, cur, upd)) return;

74. Using VarHandle
package java.lang.invoke;
/** @since 9 */
public abstract class VarHandle {
@MethodHandle.PolymorphicSignature
public native boolean compareAndSet(Object... args);
}

75. Using VarHandle
private volatile Node<T> top;
private static final VarHandle TOP;
static {
try {
TOP = MethodHandles.lookup()
.findVarHandle(LockFree.class, "top", Node.class);
} catch (NoSuchFieldException | IllegalAccessException e) {
throw new InternalError(e);
}
}

76. Using VarHandle
private volatile Node<T> top;
private static final VarHandle TOP;
static {
try {
TOP = MethodHandles.lookup()
.findVarHandle(LockFree.class, "top", Node.class);
} catch (NoSuchFieldException | IllegalAccessException e) {
throw new InternalError(e);
}
}
if (TOP.compareAndSet(this, cur, upd) return;

77. Using AtomicFU J
private val top = atomic<Node<T>?>(null)

78. Using AtomicFU J
private val top = atomic<Node<T>?>(null)
if (top.compareAndSet(cur, upd)) return

79. Using AtomicFU J
private val top = atomic<Node<T>?>(null)
if (top.compareAndSet(cur, upd)) return
Code like
AtomicReference

80. Using AtomicFU J
private val top = atomic<Node<T>?>(null)
if (top.compareAndSet(cur, upd)) return
Bytecode
Code like
AtomicReference
compile

81. Using AtomicFU J
private val top = atomic<Node<T>?>(null)
if (top.compareAndSet(cur, upd)) return
Bytecode
Code like
AtomicReference
AtomicReferenceFUcompile atomicFU

82. Using AtomicFU J
private val top = atomic<Node<T>?>(null)
if (top.compareAndSet(cur, upd)) return
Bytecode
Code like
AtomicReference
VarHandlecompile atomicFU

83. Was it worth it?

84. Benchmark results
0
5
10
15
20
25
30
35
40
1 2 4 8 16 32 64 128
Millions
Number of threads
Throughput (ops/s)
LockFree
LinkedStack

85. 0
5
10
15
20
25
30
35
40
1 2 4 8 16 32 64 128
Millions
Number of threads
Throughput (ops/s)
LockFree
LinkedStack
Benchmark results
Yeh!
Nay…

86. Contention
P
Q retry
pop1
pop2
try update

87. Too toy of a problem?
class LinkedStack<T> {
private var top: Node<T>? = null
@Synchronized
fun push(value: T) { … }
@Synchronized
fun pop(): T? {
val cur = top ?: return null
top = cur.next
return cur.value
}
}

88. Too toy of a problem – make it more real?
class LinkedStack<T> {
private var top: Node<T>? = null
@Synchronized
fun push(value: T) { … }
@Synchronized
fun pop(): T? {
val cur = top ?: return null
top = cur.next
Blackhole.consumeCPU(100L)
return cur.value
}
}

89. Too toy of a problem – make it more real?
class LockFree<T> {
private val top = atomic<Node<T>?>(null)
fun push(value: T) { … }
fun pop(): T? {
while (true) {
val cur = top.value ?: return null
Blackhole.consumeCPU(100L)
if (top.compareAndSet(cur, cur.next)) return cur.value
}
}
}

90. Benchmark results
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
1 2 4 8 16 32 64 128
Millions
Number of threads
Throughput (ops/s)
LockFree
LinkedStack

91. Workload
@State(Scope.Benchmark)
open class LinkedStackBenchmark {
private val stack = LinkedStack<Int>()
@Benchmark
fun benchmark() {
stack.push(1)
check(stack.pop() == 1)
}
}

92. Read-dominated workload
@State(Scope.Benchmark)
open class LinkedStackBenchmark {
private val stack = LinkedStack<Int>()
@Benchmark
fun benchmarkReadDominated() {
stack.push(1)
repeat(10) { check(stack.peek() == 1) }
check(stack.pop() == 1)
}
}

93. Read-dominated workload
@State(Scope.Benchmark)
open class LinkedStackBenchmark {
private val stack = LinkedStack<Int>()
@Benchmark
fun benchmarkReadDominated() {
stack.push(1)
repeat(10) { check(stack.peek() == 1) }
check(stack.pop() == 1)
}
}
class LinkedStack<T> {
@Synchronized
fun peek() = top?.value
}

94. Read-dominated workload
@State(Scope.Benchmark)
open class LockFreeBenchmark {
private val stack = LockFree<Int>()
@Benchmark
fun benchmarkReadDominated() {
stack.push(1)
repeat(10) { check(stack.peek() == 1) }
check(stack.pop() == 1)
}
}
class LockFree<T> {
fun peek() = top.value?.value
}

95. Benchmark results – x10 reads
0
5
10
15
20
25
1 2 4 8 16 32 64 128
Millions
Number of threads
Throughput (ops/s)
LockFree
LinkedStack

96. Benchmark results – x100 reads
0
1
2
3
4
5
6
1 2 4 8 16 32 64 128
Millions
Number of threads
Throughput (ops/s)
LockFree
LinkedStack

97. But … scalability?

98. Real-world workload
@Benchmark
fun benchmarkReadWorld() {
stack.push(1)
repeat(10) {
check(stack.peek() == 1)
Blackhole.consumeCPU(100L)
}
check(stack.pop() == 1)
Blackhole.consumeCPU(100L)
}

99. Benchmark results – real world
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
1 2 4 8 16 32 64 128
Millions
Number of threads
Throughput (ops/s)
LockFree
LinkedStack

100. Learn to ask the right questions
You shall, young Padawan.

102. Links
• JMH http://openjdk.java.net/projects/code-tools/jmh/
• Kotlin https://kotlinlang.org
• AtomicFU https://github.com/Kotlin/kotlinx.atomicfu

103. Thank you
Any questions?
Slides are available at www.slideshare.net/elizarov
email me to elizarov at gmail
relizarov

104. Appendix

105. A toy problem – concurrent pop
class Node<T>(val next: Node<T>?, val value: T)
next = A
value = 2
top
next = null
value = 1
B A
cur1 cur2

106. A toy problem – concurrent pop
class Node<T>(val next: Node<T>?, val value: T)
next = A
value = 2
top
next = null
value = 1
B A
cur1 cur2

107. A toy problem – concurrent pop
class Node<T>(val next: Node<T>?, val value: T)
next = A
value = 2
top
next = null
value = 1
B A
result1 = 2
cur1 cur2
result2 = 2