O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Building Event-Driven Applications with Apache Kafka & Confluent Platform

806 visualizações

Publicada em

Apache Kafka® has become the de facto technology for real-time event streaming. Confluent Platform, developed by the creators of Apache Kafka, is an event-streaming platform that enables the ingest and processing of massive amounts of data in real time.

In this session, we will cover the easiest ways to start developing event-driven applications with Apache Kafka using Confluent Platform. We will also demo a contextual event-driven application built using our ecosystem of connectors, REST proxy, and a variety of native clients.

View now to learn:
-How to create Apache Kafka topics in minutes and process event streams in real time
-Check the health of an Apache Kafka broker using Confluent Control Center
-The latest enhancements to Confluent Platform that make it easier to run Apache Kafka at scale
-How to use KSQL, streaming SQL for Apache Kafka, to process event streams in real time using simple SQL queries

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Building Event-Driven Applications with Apache Kafka & Confluent Platform

  1. 1. @gamussa | #eventdriven | @ConfluentINc Building Event-Driven Apps with Apache Kafka & Confluent Platform
  2. 2. @gamussa | #eventdriven | @ConfluentINc 2
  3. 3. @gamussa | #eventdriven | @ConfluentINc 3 I build highly scalable Hello World apps @kennybastani
  4. 4. Photoby KhaiSzeOng on Unsplash
  5. 5. Photoby RainierRidao on Unsplash
  6. 6. Photoby RohitTandon on Unsplash
  7. 7. Photoby TheodoreMoore on Unsplash
  8. 8. Photoby CristianGrecu on Unsplash
  9. 9. @gamussa | #eventdriven | @ConfluentINc 9 Normal Applications (i.e., monoliths)
  10. 10. @gamussa | #eventdriven | @ConfluentINc 10 Monoliths are hard to think about
  11. 11. @gamussa | #eventdriven | @ConfluentINc 11 Monoliths are hard to change
  12. 12. Refactoring to microservices
  13. 13. @gamussa | #eventdriven | @ConfluentINc 13 Re- integration?
  14. 14. @gamussa | #eventdriven | @ConfluentINc 14 Re- integration
  15. 15. Photoby DevaDarshan from Pexels
  16. 16. @gamussa | #eventdriven | @ConfluentINc 16 There are no good ways to integrate microservices
  17. 17. @gamussa | #eventdriven | @ConfluentINc 17 There are no-good ways to integrate microservices
  18. 18. @gamussa | #eventdriven | @ConfluentINc 18 Filesystem
  19. 19. @gamussa | #eventdriven | @ConfluentINc 19
  20. 20. @gamussa | #eventdriven | @ConfluentINc 20 Database
  21. 21. @gamussa | #eventdriven | @ConfluentINc 21
  22. 22. @gamussa | #eventdriven | @ConfluentINc 22 Integrating Microservices through the database ● “I have a database and I know how to use it.” ● Eventually causes services to co-mingle. ● Violation the «bounded context» ● Great to use inside a service boundary! ● Terrible for sharing data or negotiating change.
  23. 23. @gamussa | #eventdriven | @ConfluentINc 23 RPC
  24. 24. @gamussa | #eventdriven | @ConfluentINc 24
  25. 25. @gamussa | #eventdriven | @ConfluentINc 25 Integrating microservices via RPC ● Avoids problems of database integration ● Feels natural ● Aligns with the request/response paradigm ● Problem: cascading failures ● Question: how do you debug this system? ● Answer: you build a log. 🤔
  26. 26. @gamussa | #eventdriven | @ConfluentINc 26 Log4j type of log? 🤔
  27. 27. @gamussa | #eventdriven | @ConfluentINc 27 Events?
  28. 28. @gamussa | #eventdriven | @ConfluentINc 28 What’s an event?
  29. 29. @gamussa | #eventdriven | @ConfluentINc A shared narrative describing the evolution of the business over time 29
  30. 30. @gamussa | #eventdriven | @ConfluentINc A combination of: Notification State transfer 30
  31. 31. @gamussa | #eventdriven | @ConfluentINc Also, events are immutable. 31
  32. 32. @gamussa | #eventdriven | @ConfluentINc 32 Core abstraction ● DB - table ● Hadoop - file ● Kafka - ?
  33. 33. @gamussa | #eventdriven | @ConfluentINc 33 LOG
  34. 34. @gamussa | #eventdriven | @ConfluentINc 34 A Rich Microservices Application
  35. 35. @gamussa | #eventdriven | @ConfluentINc 35
  36. 36. @gamussa | #eventdriven | @ConfluentINc 36 Ratings Service operating on the event stream Order Requested ratings raw-rating Ratings Service KStreams API used to count and sum ratings as they arrive Browser Webserver KAFKA ratings counts ratings sums ratings
  37. 37. @gamussa | #eventdriven | @ConfluentINc 37 Schema Management Order Requested ratings raw-rating Ratings Service Browser Webserver KAFKA ratings counts ratings sums ratings Schema Registry KStreams API used to count and sum ratings as they arrive
  38. 38. @gamussa | #eventdriven | @ConfluentINc 38 Access legacy relation data Order Requested ratings raw-rating Ratings Service Browser Webserver KAFKA ratings counts ratings sums ratings Schema Registry Connect Movies KAFKA KStreams API used to count and sum ratings as they arrive
  39. 39. @gamussa | #eventdriven | @ConfluentINc 39 Materialize tables inside the app Order Requested ratings raw-rating Ratings Service Browser Webserver KAFKA ratings counts ratings sums ratings Schema Registry Connect Movies KAFKA Lookup table created inside the Rating Service Movies
  40. 40. @gamussa | #eventdriven | @ConfluentINc 40 Create a new table, persist it to the log Order Requested ratings raw-rating Ratings Service Browser Webserver KAFKA ratings counts ratings sums ratings Schema Registry Connect Movies KAFKA Movies rated-movies rated-movies Writable table created for rated movies
  41. 41. @gamussa | #eventdriven | @ConfluentINc 41 Create a new table, persist it to the log Order Requested ratings raw-rating Ratings Service Browser Webserver KAFKA ratings counts ratings sums ratings Schema Registry Connect Movies KAFKA Movies rated-movies rated-movies TRANSACTION
  42. 42. @gamussa | #eventdriven | @ConfluentINc 42 Hydrate a materialized view Order Requested ratings raw-rating Ratings Service Browser Webserver KAFKA ratings counts ratings sums ratings Schema Registry Connect Movies KAFKA rated-movies an embedded database CDC
  43. 43. @gamussa | #eventdriven | @ConfluentINc 43 What’s new in Cp 5.2?
  44. 44. @gamussa | #eventdriven | @ConfluentINc 44 Message browser enhancements Why Makes reading messages easier for application development & troubleshooting Features ● Feed auto-load, scroll, pause, and resume ● Seek to specific offsets (per partition) ● Card & table formats ● Message metadata
  45. 45. @gamussa | #eventdriven | @ConfluentINc 45 KSQL UI enhancements Why Improves KSQL usability & overall UX Features ● Feed pause & resume ● Card & table formats ● Clearer query status ● Improved error messaging ● Output metadata
  46. 46. @gamussa | #eventdriven | @ConfluentINc 46 Schema management Why Enables schema editing & policy management Features ● Create and edit schemas ● Validate schemas against compatibility 
 policy ● Manage compatibility policy
  47. 47. @gamussa | #eventdriven | @ConfluentINc 47 Dynamic broker configuration Why Helps operators manage their brokers Features ● Change broker configs that don’t require restarts
  48. 48. @gamussa | #eventdriven | @ConfluentINc 48 License management Why Makes managing the subscription license easier and prevents unexpected expirations Features ● License checking ● License submission ● Pending expiration alerts (3mo out, 1mo out, weekly for the last month, daily for the last week) ● Expired license notice
  49. 49. @gamussa | #eventdriven | @ConfluentINc 49 Multi-cluster KSQL Why Runs queries on 1+ KSQL clusters Features ● Many-to-1 mapping for KSQL-to-Kafka Configuration ● confluent.controlcenter.ksql.<name of cluster>.url = <list of urls> ● confluent.controlcenter.ksql.<name of cluster>.advertised.url = <list of urls>
  50. 50. @gamussa | #eventdriven | @ConfluentINc 50 Multi-cluster Connect Why Run, monitor, and manage connectors on 1+ Connect clusters Features ● Many-to-1 mapping for Connect-to-Kafka Configuration ● confluent.controlcenter.connect.<name>.
 cluster = <list of urls>
  51. 51. @gamussa | #eventdriven | @ConfluentINc 51 My personal favorite
  52. 52. @gamussa | #eventdriven | @ConfluentINc 52 Kafka-topics use AdminClient API
  53. 53. CP 5.2 is Free forever on a single Kafka broker
  54. 54. 54 Stay in touch cnfl.io/meetupscnfl.io/slackcnfl.io/blog
  55. 55. 55

×