O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Elastic Streams at Scale @ Flink Forward 2018 Berlin

457 visualizações

Publicada em

One of the big operational challenges when running streaming applications is to cope with varying workloads. Variations, e.g. daily cycles, seasonal spikes or sudden events, require that allocated resources are constantly adapted. Otherwise, service quality deteriorates or money is wasted. Apache Flink 1.5 includes a lot of enhancements to support full resource elasticity on cluster management frameworks such as Apache Mesos. With the latest version, it is now possible to build elastic applications which can be programmatically scaled up or down in order to react to changing workloads. In this talk, we will discuss recent improvements to Flink's deployment model which also enables full resource elasticity. In particular, we will discuss how Flink leverages cluster management frameworks, e.g. Mesos, and already-introduced features like scalable state to support elastic streaming applications. We will conclude the presentation with a short demo showing how a stateful Flink application can be rescaled on top of Mesos.

Publicada em: Tecnologia
  • Entre para ver os comentários

  • Seja a primeira pessoa a gostar disto

Elastic Streams at Scale @ Flink Forward 2018 Berlin

  1. 1. Elastic Streams at Scale @stsffap @joerg_schad
  2. 2. time Workload Resources time Workload Resources
  3. 3. • •
  4. 4.
  5. 5. • • • •
  6. 6. • • •
  7. 7. • •
  8. 8. • • • #1 #2 #3
  9. 9. partitionId: 1, offset: 42 partitionId: 3, offset: 10 partitionId: 6, offset: 27 partitionId: 1, offset: 42 partitionId: 3, offset: 10 partitionId: 6, offset: 27 • Store offset for each partition • Individual entries are repartitionable
  10. 10. • • • •
  11. 11. • • •
  12. 12. • • • • • • •
  13. 13. • • • • • •
  14. 14. • • • • • • • • • • • •
  15. 15. © 2018 Mesosphere, Inc. All Rights Reserved. Flink Mesos Integration Mesos Master Mesos Cluster Client (2) HTTP POST JobGraph/Jars Flink Master Process Flink Mesos ResourceManager JobManager (4) Start Process (and supervise) (8) Deploy Tasks (7) Register (5) Request slots Flink Mesos Dispatcher (3) Allocate container for Flink master (6) Allocate containers for TaskManagers Marathon (1) Start and monitor dispatcher Mesos Task TaskManager Mesos Task TaskManager
  16. 16. Mesos Master Mesos Cluster Cluster Client (1) Submit Mesos App. (JobGraph / JARs) Application Master Flink-Mesos ResourceManager JobManager TaskManager TaskManager TaskManager (2) Spawn Application Master (4) Start TaskManagers (6) Deploy Tasks (5) Register (3) Request slots
  17. 17. • Monitor
  18. 18. • • • • •
  19. 19. THANK YOU! @stsffap @dataArtisans @ApacheFlink WE ARE HIRING data-artisans.com/careers

×