Apache Flink is a world class stateful stream processor presents a huge variety of optional features and configuration choices to the user. Determining out the optimal choice for any production environment and use-case be challenging. In this talk, we will explore and discuss the universe of Flink configuration with respect to state and state backends. We will start with a closer look under the hood, at core data structures and algorithms, to build the foundation for understanding the impact of tuning parameters and the costs-benefit-tradeoffs that come with certain features and options. In particular, we will focus on state backend choices (Heap vs RocksDB), tuning checkpointing (incremental checkpoints, ...) and recovery (local recovery), serializers and Apache Flink's new state migration capabilities.

1. © 2019 Ververica
Seth Wiesman, Solutions Architect
Deep Dive on Apache Flink State

2. © 2019 Ververica

3. © 2019 Ververica
Agenda
• Serialization
• State Backends
• Checkpoint Tuning
• Schema Migration
• Upcoming Features
3

4. © 2019 Ververica
Serializers

5. © 2019 Ververica
Flink’s Serialization System
• Natively Supported Types
• Primitive Types
• Tuples, Scala Case Classes
• Pojo’s
• Unsupported Types Fall Back to Kryo
5

6. © 2019 Ververica
Flink’s Serialization System
Benchmark Results For Flink 1.8
6
Serializer Ops/s
PojoSerializer 305 / 293*
RowSerializer 475
TupleSerializer 498
Kryo 102 / 67*
Avro (Reflect API) 127
Avro (SpecificRecord API) 297
Protobuf (via Kryo) 376
Apache Thrift (via Kryo) 129 / 112*
public static class MyPojo {
public int id;
private String name;
private String[] operationNames;
private MyOperation[] operations;
private int otherId1;
private int otherId2;
private int otherId3;
private Object someObject; // used with String
}
MyOperation {
int id;
protected String name;
}

7. © 2019 Ververica
Custom Serializers
• registerKryoType(Class<?>)
• Registers a type with Kryo for more compact binary format
• registerTypeWithKryoSerializer(Class<?>, Class<? extends Serializer>)
• Provides a default serializer for the given class
• Provided serializer class must extends com.esotericsoftware.kryo.Serializer
• addDefaultKryoSerializer(Class<?>, Serializer<?> serializer)
• Registers a serializer as the default serializer for the given type
Registration with Kryo via ExecutionConfig
7

8. © 2019 Ververica
Custom Serializer’s
@TypeInfo Annotation
8
@TypeInfo(MyTupleTypeInfoFactory.class)
public class MyTuple<T0, T1> {
public T0 myfield0;
public T1 myfield1;
}
public class MyTupleTypeInfoFactory extends TypeInfoFactory<MyTuple> {
@Override
public TypeInformation<MyTuple> createTypeInfo(Type t, Map<String, TypeInformation<?>> genericParameters) {
return new MyTupleTypeInfo(genericParameters.get("T0"), genericParameters.get("T1"));
}

9. © 2019 Ververica
State Backends

10. © 2019 Ververica10
Task Manager Process Memory Layout
Task Manager JVM Process
Java Heap
Off Heap / Native
Flink Framework etc.
Network Buffers
Timer State
Keyed State
Typical Size

11. © 2019 Ververica11
Task Manager Process Memory Layout
Task Manager JVM Process
Java Heap
Off Heap / Native
Flink Framework etc.
Network Buffers
Timer State
Keyed State
Typical Size

12. © 2019 Ververica12
Task Manager Process Memory Layout
Task Manager JVM Process
Java Heap
Off Heap / Native
Flink Framework etc.
Network Buffers
Timer State
Keyed State
Typical Size

13. © 2019 Ververica13
Keyed State Backends
Based on Java Heap Objects Based on RocksDB

14. © 2019 Ververica
Heap Keyed State Backend
• State lives as Java objects on the heap
• Organized as chained hash table, key ↦ state
• One hash table per registered state
• Supports asynchronous state snapshots
• Data is de / serialized only during state snapshot and restore
• Highest Performance
• Affected by garbage collection overhead / pauses
• Currently no incremental checkpoints
• High memory overhead of representation
• State is limited by available heap memory
14

15. © 2019 Ververica
Heap State Table Architecture
15
- Hash buckets (Object[]), 4B-8B per slot
- Load factor <= 75%
- Incremental rehash
Entry
Entry
Entry

16. © 2019 Ververica
Heap State Table Architecture
16
- Hash buckets (Object[]), 4B-8B per slot
- Load factor <= 75%
- Incremental rehash
Entry
Entry
Entry
▪ 4 References:
▪ Key
▪ Namespace
▪ State
▪ Next
▪ 3 int:
▪ Entry Version
▪ State Version
▪ Hash Code
K
N
S
4 x (4B-8B)
+3 x 4B
+ ~8B-16B (Object overhead)
Object sizes and
overhead.
Some objects might
be shared.

17. © 2019 Ververica
Heap State Table Snapshot
17
Original Snapshot
A C
B
Entry
Entry
Entry
Copy of hash bucket array is snapshot overhead

18. © 2019 Ververica
Heap State Table Snapshot
18
Original Snapshot
A C
B
D
No conflicting modification = no overhead

19. © 2019 Ververica
Heap State Table Snapshot
19
Original Snapshot
A’ C
B
D A
Modifications trigger deep copy of entry - only as much as required. This depends on
what was modified and what is immutable (as determined by type serializer).
Worst case overhead = size of original at time of snapshot.

20. © 2019 Ververica
Heap Backend Tuning Considerations
• Choose TypeSerializers with efficient copy-methods
• Flag immutability of objects where possible to avoid copy completely
• Flatten POJOs / avoid deep objects
• Reduces object overheads and following references
• GC choice / tuning
• Scale out using multiple task managers per node
20

21. © 2019 Ververica
RocksDB Keyed State Backend Characteristics
• State lives as serialized byte-strings in off-heap memory and on local disk
• One column family per registered state (~table)
• Key / Value store, organized as a log-structured merge tree (LSM tree)
• Key: serialized bytes of <keygroup, key, namespace>
• LSM naturally supports MVCC
• Data is de / serialized on every read and update
• Not affected by garbage collection
• Relatively low overhead of representation
• LSM naturally supports incremental snapshots
• State size is limited by available local disk space
• Lower performance (~ order of magnitude compared to Heap state backend)
21

22. © 2019 Ververica
RocksDB Architecture
22
Local Disk
WAL
WAL
Compaction
Memory Persistent Store
Flush
In Flink:
- disable WAL and sync
- persistence via checkpointsActive
MemTable
ReadOnly
MemTable
WriteOp
Full/Switch
SST SST
SSTSST
Merge

23. © 2019 Ververica
RocksDB Architecture
23
Local Disk
WAL
WAL
Compaction
Memory Persistent Store
Flush
In Flink:
- disable WAL and sync
- persistence via checkpointsActive
MemTable
ReadOnly
MemTable
WriteOp
Full/Switch
SST SST
SSTSST
Merge
Set per column
family (~table)

24. © 2019 Ververica
RocksDB Architecture
24
ReadOp
Local Disk
WAL
WAL
Memory Persistent Store
Flush
Merge
Active
MemTable
ReadOnly
MemTable
Full/Switch
WriteOp
SST SST
SSTSST
In Flink:
- disable WAL and sync
- persistence via checkpoints

25. © 2019 Ververica
RocksDB Architecture
25
ReadOp
Local Disk
WAL
WAL
Memory Persistent Store
Flush
Merge
Active
MemTable
ReadOnly
MemTable
Full/Switch
WriteOp
SST SST
SSTSST
In Flink:
- disable WAL and sync
- persistence via checkpointsActive
MemTable
ReadOnly
MemTable
WriteOp
ReadOp
Local Disk
WAL
WAL
Compaction
Memory Persistent Store
Full/Switch
Read Only
Block Cache
Flush
SST SST
SSTSST
Merge
In Flink:
- disable WAL and sync
- persistence via checkpoints

26. © 2019 Ververica
RocksDB Resource Consumption
• One RocksDB instance per operator subtask
• block_cache_size
• Size of the block cache
• write_buffer_size
• Max size of a MemTable
• max_write_buffer_number
• The maximum number of MemTable’s allowed in memory before flush to SST file
• Indexes and bloom filters
• Optional
• Table Cache
• Caches open file descriptors to SST files
• Default: unlimited!
26

27. © 2019 Ververica
Performance Tuning
Amplification Factors
27
Write Amplification
Read Amplification Space Amplification
More details: https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
Parameter
Space

28. © 2019 Ververica
Performance Tuning
Amplification Factors
28
Write Amplification
Read Amplification Space Amplification
More details: https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide
Parameter
Space
Example: More compaction effort =
increased write amplification
and reduced read amplification

29. © 2019 Ververica
General Performance Considerations
• Use efficient TypeSerializer’s and serialization formats
• Decompose user code objects
• ValueState<List<Integer>> ListState<Integer>
• ValueState<Map<Integer, Integer>> MapState<Integer, Integer>
• Use the correct configuration for your hardware setup
• Consider enabling RocksDB native metrics to profile your applications
• File Systems
• Working directory on fast storage, ideally local SSD. Could even be memory.
• EBS performance can be problematic
29

30. © 2019 Ververica
Timer Service

31. © 2019 Ververica
Heap Timers
31
▪ 2 References:
▪ Key
▪ Namespace
▪ 1 long:
▪ Timestamp
▪ 1 int:
▪ Array Index
K
N
Object sizes and
overhead.
Some objects might
be shared.
Binary heap of timers in array
Peek: O(1)
Poll: O(log(n))
Insert: O(log(n))
Delete: O(n)
Contains O(n)
Timer

32. © 2019 Ververica
Heap Timers
32
▪ 2 References:
▪ Key
▪ Namespace
▪ 1 long:
▪ Timestamp
▪ 1 int:
▪ Array Index
K
N
Object sizes and
overhead.
Some objects might
be shared.
Binary heap of timers in array
HashMap<Timer, Timer> : fast deduplication and deletes
Key Value
Peek: O(1)
Poll: O(log(n))
Insert: O(log(n))
Delete: O(log(n))
Contains O(1)
MapEntry
Timer

33. © 2019 Ververica
Heap Timers
33
Binary heap of timers in array
HashMap<Timer, Timer> : fast deduplication and deletes
MapEntry
Key Value
Snapshot (net values of a timer are immutable)
Timer

34. © 2019 Ververica
RocksDB Timers
34
0 20 A X
0 40 D Z
1 10 D Z
1 20 C Y
2 50 B Y
2 60 A X
…
…
Key
Group
Time
stamp
Key
Name
space
…
Lexicographically ordered
byte sequences as key, no value
Column Family - only key, no value

35. © 2019 Ververica
RocksDB Timers
35
0 20 A X
0 40 D Z
1 10 D Z
1 20 C Y
2 50 B Y
2 60 A X
…
…
Key
Group
Time
stamp
Key
Name
space
Column Family - only key, no value
Key group queues
(caching first k timers)
Priority queue of
key group queues

36. © 2019 Ververica
3 Task Manager Memory Layout
36
Task Manager JVM Process
Off Heap / Native
Flink Framework etc.
Network Buffers
Timer State
Keyed State
Task Manager JVM Process
Java Heap
Off Heap / Native
Flink Framework etc.
Network Buffers
Timer State
Keyed State
Task Manager JVM Process
Java Heap
Off Heap / Native
Flink Framework etc.
Network Buffers
Keyed State
Timer State

37. © 2019 Ververica
Full / Incremental Checkpoints

38. © 2019 Ververica
Full Checkpoint
38
G
H
C
D
Checkpoint 1 Checkpoint 2 Checkpoint 3
I
E
A
B
C
D
A
B
C
D
A
F
C
D
E
@t1 @t2 @t3
A
F
C
D
E
G
H
C
D
I
E

39. © 2019 Ververica
Full Checkpoint Overview
• Creation iterates and writes full database snapshots as a stream to stable storage
• Restore reads data as a stream from stable storage and re-inserts into the state backend
• Each checkpoint is self contained, and size is proportional to the size of full state
• Optional: compression with snappy
39

40. © 2019 Ververica
Incremental Checkpoint
40
H
C
D
Checkpoint 1 Checkpoint 2 Checkpoint 3
I
E
A
B
C
D
A
B
C
D
A
F
C
D
E
E
F
G
H
I
@t1 @t2 @t3
builds upon builds upon
𝚫𝚫 𝚫

41. © 2019 Ververica
Incremental Checkpoints with RocksDB
41
Local Disk
WAL
WAL
Compaction
Memory Persistent Store
Flush
Incremental checkpoint:
Observe created/deleted
SST files since last checkpoint
Active
MemTable
ReadOnly
MemTable
WriteOp
Full/Switch
SST SST
SSTSST
Merge

42. © 2019 Ververica
Incremental Checkpoint Overview
• Expected trade-off: faster* checkpoints, slower recovery
• Creation only copies deltas (new local SST files) to stable storage
• Creates write amplification because we also upload compacted SST files so that we can prune checkpoint
history
• Sum of all increments that we read from stable storage can be larger than the full state size
• No rebuild is required because we simply re-open the RocksDB backend from the SST files
• SST files are snappy compressed by default
42

43. © 2019 Ververica
Schema Migration

44. © 2019 Ververica
Anatomy of a Flink Stream Job Upgrade
44
Flink job user code
Local State Backend
Persistent Savepoint
local reads / writes that
manipulate state

45. © 2019 Ververica
Anatomy of a Flink Stream Job Upgrade
45
Flink job user code
Local State Backend
Persistent Savepoint
Application Upgrade

46. © 2019 Ververica
Anatomy of a Flink Stream Job Upgrade
46
Flink job user code
Local State Backend
Persistent Savepoint
Continue To Access State

47. © 2019 Ververica
Upcoming Features

48. © 2019 Ververica
Upcoming Features
• A new state backend
• Unified savepoint binary format
• State Processor API
48

49. © 2019 Ververica
Questions?