Practical learnings from running thousands of Flink jobs
•Transferir como PPTX, PDF•
0 gostou•323 visualizações
Flink Forward San Francisco 2022. Task Managers constantly running out of memory? Flink job keeps restarting from cryptic Akka exceptions? Flink job running but doesn’t seem to be processing any records? We share practical learnings from running thousands of Flink Jobs for different use-cases and take a look at common challenges they have experienced such as out-of-memory errors, timeouts and job stability. We will cover memory tuning, S3 and Akka configurations to address common pitfalls and the approaches that we take on automating health monitoring and management of Flink jobs at scale. by Hong Teoh & Usamah Jassat
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Practical learnings from running thousands of Flink jobs
Semelhante a Practical learnings from running thousands of Flink jobs (20)
Mais de Flink Forward
Mais de Flink Forward (13)
Último
Último (20)
Practical learnings from running thousands of Flink jobs
- 1. © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Practical learnings from running thousands of Flink jobs HongTeoh, Usamah Jassat Amazon Kinesis Data Analytics
- 2. © 2022, Amazon Web Services, Inc. or its affiliates. Outline • Job Stability • State backend selection and tuning
- 3. © 2022, Amazon Web Services, Inc. or its affiliates. Typical Flink cluster set up on Kubernetes Jobmanager Taskmanager Taskmanager Taskmanager
- 4. © 2022, Amazon Web Services, Inc. or its affiliates. Job instability: Memory OOMKilled Using more than allocated to Flink configuration
- 5. © 2022, Amazon Web Services, Inc. or its affiliates. Memory management (Instance/Container/Java) Instance (Kubernetes node) cAdvisor OS oom-killer Taskmanager container Java (Flink process) Other processes (e.g. Python, Kinesis producers)
- 6. © 2022, Amazon Web Services, Inc. or its affiliates. Memory investigation learnings • Record both Java and cAdvisor memory metrics • Record both virtual + real memory use • OOM-Killer can cause “broken” containers
- 7. © 2022, Amazon Web Services, Inc. or its affiliates. Flink – Java memory mapping Flink / Java Heap Metaspace Direct Mapped JNI Overhead Heap Metaspace Network Task Off-heap Managed Overhead -Xmx -XX:MetaspaceSize -XX:MaxDirectMemorySize NOT CONTROLLED Flink controls* NOT CONTROLLED Heap OOM Metaspace OOM OOMKilled Direct Memory OOM
- 8. © 2022, Amazon Web Services, Inc. or its affiliates. Memory configuration learnings • jvm-overhead too low • Investigate native memory use using: • Native MemoryTracking -XX:NativeMemoryTracking=detail jcmd 1 VM.native_memory • jemalloc + jeprof
- 9. © 2022, Amazon Web Services, Inc. or its affiliates. Job instability: “Hung” job / checkpoints
- 10. © 2022, Amazon Web Services, Inc. or its affiliates. Thread dump: Thread stuck on SocketRead
- 11. © 2022, Amazon Web Services, Inc. or its affiliates. Thread dump: Thread stuck on deadlock jackson-databind version < 2.12
- 12. © 2022, Amazon Web Services, Inc. or its affiliates. State backends • State backends in Flink • Comparing HashMap and RocksDB • RocksDB Optimizations
- 13. © 2022, Amazon Web Services, Inc. or its affiliates. State-backends in Flink
- 14. © 2022, Amazon Web Services, Inc. or its affiliates. Comparing HashMap and RocksDB HashMap RocksDB Storage Java Heap Native Memory + Disk Ceiling Java Heap File System Incremental Checkpointing Yes (experimental) Yes R/W Performance Higher Lower* GC Impact Yes No In Flight Data Format Java Objects Serialized
- 15. © 2022, Amazon Web Services, Inc. or its affiliates. RockDB or HashMap? • RocksDB for large state • Hash Map for performance
- 16. © 2022, Amazon Web Services, Inc. or its affiliates. RocksDB Optimization Write Stalls • Increase RocksDB memory • Increase write threads • Increase write buffer memory* • Write buffer size + counts Slow Reads • Local Disk • Increase RocksDB memory • Increase read cache*
- 17. © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Thank you!
- 18. © 2022, Amazon Web Services, Inc. or its affiliates. Q&A
Notas do Editor
- In a typical setup of Flink cluster on Kubernetes JM containers TM containers
- Instance 2 taskmanager containers -> Java process, Python process, Kinesis producers Each measures memory differently: Java process measures heap, non-heap using beans. Only measure memory use of Java process, excludes Python/Kinesis Containers’ memory use is measured using cAdvisor On the instance, the OS bridges the gap between Virtual memory and Real memory Means that if the Java process requests 100 Gb of memory, that memory is allocated in virtual space In reality, it is mapped to a “zero page”, meaning it doesn’t actually take up 100Gb of space, until it is accessed/written to. In Kubernetes set ups, vm.overcommit_memory is recommended to be set to 1 -> Do not limit virtual memory. When all processes use the memory, the oom-killer (OS process) will spin up and terminate a process. Does not care what container it belongs to – can cause containers to be in unhealthy state. Takeaways Measure BOTH Java memory and OS memory (cAdvisor) Use data to drill down into which bucket of memory is over the limit cGroups v2 to treat all processes in container like a single process
- Java memory NOT JUST HEAP Metaspace – for classes Direct – directly use ByteBuffers -> significantly faster than allocating in heap Mapped – useful when reading from file -> Map a file into a ByteBuffer Java Native Interface (JNI) – when running non-Java code (RocksDB state backend) Overhead – used for the JVM itself (GC, thread stacks, Symbols) Map to Flink Taskmanager memory configurations Explain limits for each bucket When each limit is exceeded, this is how it surfaces Gotchas