Deep Dive into Kafka Connect Protocol with Catalin Pop
Denunciar
HostedbyConfluentSeguir
2 de Jun de 2023•0 gostou•161 visualizações
Deep Dive into Kafka Connect Protocol with Catalin Pop
2 de Jun de 2023•0 gostou•161 visualizações
Baixar para ler offline
Denunciar
Tecnologia
"Kafka Connect is the highway to our central nervous system for streaming data into Kafka. This highway has allowed us to connect hundreds of external services without the need to write code for each individual integration. Kafka Connect is based on the concept of having Connectors and tasks that do the work for us. There’s two fundamental questions that come up when it comes to our highway: • How is this highway made? How are connectors created/started? • How do we make sure there’s no traffic jams? Also known as, • How are connectors evenly distributed across nodes? In this presentation, we will answer the above questions and cover the following topics: • Breaking down how a Connector/task is created • Breaking down each Kafka Connect protocol(eager, compatible, sessioned) • Walk through rebalances for each protocol • Pros/cons of each protocol"
HostedbyConfluentSeguir
Recomendados
Scaling your apps with Kubernetes and Docker - TheConf 2018Erick Wendel
Continuous Delivery @ Codemotion Miguel Serrano Milano
Continuous Deployment of Architectural ChangeMatt Graham
Distributed Versioning Tools, BeJUG 2010Pursuit Consulting
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...confluent
Trunk-Based Development and TogglingBryan Liu
Mais conteúdo relacionado
Similar a Deep Dive into Kafka Connect Protocol with Catalin Pop
Edition Based Redefinition Made Easy - Oren NakdimonOren Nakdimon
flowr streamlining computing workflowssahil seth
GraphQL vs. (the) RESTcoliquio GmbH
The Power of RefactoringStefan Koopmanschap
Deployment With Subversion - Lorna Mitchelldpc
Version Control ThinkVitaminAlex Hillman
Mais de HostedbyConfluent
Chill, Distill, No Overkill: Best Practices to Stress Test Kafka with Siva Ku...HostedbyConfluent
Manage Consistent Configurations Across Multiple Kafka Environments with Naga...HostedbyConfluent
The Cost of Kafka’s High Availability on Cloud with Geetha AnneHostedbyConfluent
Case-Study: Building Real-Time Applications at Scale-Cyclist Crash Detection ...HostedbyConfluent
Migrating Your System to the Different Kafka Platforms Available wtih Bhavesh...HostedbyConfluent
Eliminating the Double Write Problem in Apache Kafka Using the Outbox Pattern...HostedbyConfluent
Último
Salesforce Miami User Group Event - 3rd QuarterSkyPlanner
UiPath Tips and Techniques for Debugging - Session 3DianaGray10
"Stateful app as an efficient way to build dispatching for riders and drivers...Fwdays
"From Orchestration to Choreography and Back", Yevhen Bobrov Fwdays
h2 meet pdf test.pdfJohnLee971654
9C Monthly Newsletter - SEPT 2023PublishingTeam
Deep Dive into Kafka Connect Protocol with Catalin Pop
- 1. Deep Dive in Connect Protocol @cdpop_ Catalin Pop
- 2. ❏ High level overview of Connect ❏ Breaking down creation of Connectors ❏ Breaking down Connect Protocols ❏ Walk through rebalance ❏ Pros/Cons of each protocol @cdpop_
- 3. High level overview of Connect @cdpop_
- 4. High level overview of Connect @cdpop_
- 5. High level overview of Connect @cdpop_
- 6. High level overview of Connect @cdpop_
- 7. High level overview of Connect @cdpop_
- 8. High level overview of Connect @cdpop_
- 9. High level overview of Connect @cdpop_
- 10. High level overview of Connect @cdpop_
- 11. High level overview of Connect @cdpop_
- 12. ✓ High level overview of Connect ❏ Breaking down creation of Connectors ❏ Breaking down Connect Protocols ❏ Walk through rebalance ❏ Pros/Cons of each protocol @cdpop_
- 37. ✓ High level overview of Connect ✓ Breaking down creation of Connectors ❏ Breaking down Connect Protocols ❏ Walk through rebalance ❏ Pros/Cons of each protocol @cdpop_ @cdpop_
- 38. Breaking down connect protocols @cdpop_
- 39. Breaking down connect protocols @cdpop_
- 40. Eager Protocol ● Revoke all Connectors ● Perform round-robin assignment ● Send Connector assignment to queue ● Workers pickup assignment ● Start assigned connectors @cdpop_
- 41. Compatible Protocol ● Revoke connectors when the worker left the group or connector/task change ● Generate connector assignment only for the revoked connectors ● Send Connector assignment to queue ● Workers pickup assignment ● Start assigned connectors @cdpop_
- 42. Sessioned Protocol ● Setup Session key for encrypted authentication ● Revoke connectors when the worker left the group or connector/task change ● Generate connector assignment only for the revoked connectors ● Send Connector assignment to queue ● Workers pickup assignment ● Start assigned connectors @cdpop_
- 43. ✓ High level overview of Connect ✓ Breaking down creation of Connectors ✓ Breaking down Connect Protocols ❏ Walk through rebalance ❏ Pros/Cons of each protocol @cdpop_
- 57. Walk through rebalance - Eager @cdpop_
- 58. Walk through rebalance - Eager @cdpop_
- 59. Walk through rebalance - Eager @cdpop_
- 60. Walk through rebalance - Eager @cdpop_
- 61. Walk through rebalance - Eager @cdpop_
- 62. Walk through rebalance - Compatible/Sessioned @cdpop_
- 63. Walk through rebalance - Compatible/Sessioned @cdpop_
- 64. Walk through rebalance - Compatible/Sessioned @cdpop_
- 65. Walk through rebalance - Compatible/Sessioned @cdpop_
- 66. ✓ High level overview of Connect ✓ Breaking down creation of Connectors ✓ Breaking down Connect Protocols ✓ Walk through rebalance ❏ Pros/Cons of each protocol @cdpop_
- 67. Pros/Cons of protocols Eager Pro Evenly distributed Con Not secured Stops all connectors @cdpop_
- 68. Pros/Cons of protocols Eager Pro Evenly distributed Con Not secured Stops all connectors Compatible Pro Incremental rebalance Con Not secured Uneven distribution @cdpop_
- 69. Pros/Cons of protocols Eager Pro Evenly distributed Con Not secured Stops all connectors Sessioned Pro Secure Incremental rebalance Con Uneven distribution Compatible Pro Incremental rebalance Con Not secured Uneven distribution @cdpop_