O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Why My Streaming Job...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Application in a nut...
© 2018 Bloomberg Finance L.P. All rights reserved.
Initial Latency – ~10ms
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Review configuration...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Profiler to rescue
•...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Where is Time Spent?
© 2018 Bloomberg Finance L.P. All rights reserved.
Under The Hood : Persistent State Store
CachingKeyValueStore
ChangelogB...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
WithCacheEnabled?
• ...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Revisit our applicat...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Solution 1 – switch ...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Why bytes conversion...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Solution 2 - Move se...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Solution 3 – Cache b...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
What’s the best way ...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Solution 4 – Applica...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
CacheableStateStore ...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Profile again - The ...
© 2018 Bloomberg Finance L.P. All rights reserved.
Final Latency – ~1ms
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Random latency spike...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Cms-GC to G1GC
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Does this apply to y...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Does this apply to o...
© 2018 Bloomberg Finance L.P. All rights reserved.
© 2018 Bloomberg Finance L.P. All rights reserved.
Questions?
Próximos SlideShares
Carregando em…5
×

Why My Streaming Job is Slow - Profiling and Optimizing Kafka Streams Apps (Lei Chen, Bloomberg L.P.) Kafka Summit London 2019

Kafka Streams performance monitoring and tuning is important for many reasons, including identifying bottlenecks, achieving greater throughput, and capacity planning. In this talk we’ll share the techniques we used to achieve greater performance and save on compute, storage, and cost. We’ll cover: Identifying design bottlenecks in by reviewing logs, metrics, and serdes. State store access patterns, design, and optimization Using profiling tools such as JMX, YourKit etc. Performance tuning of Kafka and Kafka Streams configuration and properties. JVM optimization for correct heap size and garbage collection strategies. Functional programming and imperative programming trade offs.

  • Seja o primeiro a comentar

Why My Streaming Job is Slow - Profiling and Optimizing Kafka Streams Apps (Lei Chen, Bloomberg L.P.) Kafka Summit London 2019

  1. 1. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Why My Streaming Job is Slow A case study on profiling Kafka Streams application Kafka Summit 2019 London Nishchay Sinha, Lei Chen
  2. 2. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Application in a nut shell Transformer Transformer Stage-1 State Stage-N Market data (bid/ask/trade) Kafka Streams Processor API Composite price
  3. 3. © 2018 Bloomberg Finance L.P. All rights reserved. Initial Latency – ~10ms
  4. 4. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Review configuration • Persistent state store • Cache enabled • Changelog enabled • Kryo serde • Cache size • Commit interval • EOS disabled
  5. 5. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Profiler to rescue • Profilers: VisualVm/YourKit/Async-profiler/etc • Async-profiler o https://github.com/jvm-profiling-tools/async-profiler o Profile CPU and memory o Integrated with Intellij 2019
  6. 6. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Where is Time Spent?
  7. 7. © 2018 Bloomberg Finance L.P. All rights reserved. Under The Hood : Persistent State Store CachingKeyValueStore ChangelogBytesStore RocksDBBytesStore TreeMap LRU cache Flush by commit interval PutGet Cache miss Changelog Segments MeteredKeyValueStore (De)Serialization
  8. 8. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. WithCacheEnabled? • A performance optimization • Bytes in, Bytes out! • For Kafka Streams, not RocksDB
  9. 9. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Revisit our application Transformer Transformer Stage-1 State Stage-N Market data (bid/ask/trade) Kafka Streams Processor API Composite price
  10. 10. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Solution 1 – switch to InMemoryKeyValueStore GET/PUT TreeMap • Cannot handle state larger than available RAM Pros • ~10 times faster with logging disabled • Slower with logging enabled Cons ChangeLogs
  11. 11. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Why bytes conversion? Can it be deferred?
  12. 12. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Solution 2 - Move serde to after caching store • More memory • Cache size measurement • Increase commit time burden • Library change Pros • Faster • Transparent to application Cons Cache (De)Serialization Persistence Changelogs Async flush GET/PUT
  13. 13. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Solution 3 – Cache both Bytes and Object • More memory • Cache size measurement • Put not optimized • Library change Pros • Faster • Get is optimized • Transparent to application Cons Cache Object Persistence Changelogs Async flush Bytes GET/PUT
  14. 14. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. What’s the best way to pass transient state across processors?
  15. 15. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Solution 4 – Application layer cache • Faster • More flexible caching/flushing strategy • Coarse grained size estimator • No library change Application cache Kafka Streams State Pros • Need to flush explicitly • More memory ConsGET/PUT
  16. 16. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. CacheableStateStore - explained • CachableStateStore extends KeyValueStore[K, V] • persistentKeyValueStore as backend • On top of Guava cache • Tunable per state store, not global setting • Only serialize till end of topology • Partition specific
  17. 17. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Profile again - The hot spot was gone!
  18. 18. © 2018 Bloomberg Finance L.P. All rights reserved. Final Latency – ~1ms
  19. 19. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Random latency spikes? • Punctuators • Commit • State directory • GC pressure
  20. 20. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Cms-GC to G1GC
  21. 21. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Does this apply to you?
  22. 22. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Does this apply to only Kafka Streams?
  23. 23. © 2018 Bloomberg Finance L.P. All rights reserved. © 2018 Bloomberg Finance L.P. All rights reserved. Questions?

×