Migrating Your System to the Different Kafka Platforms Available wtih Bhavesh Sooka & Jonathan Lew
Migrating Your System to the Different Kafka Platforms Available wtih Bhavesh Sooka & Jonathan Lew
"Migrating a core technology like Kafka can be daunting. Whether it is moving from queues to stream processing, Apache Kafka to Confluent Platform or even Confluent Platform to Confluent Cloud there are many obstacles that need to be overcome, both expected and unexpected. In this talk we will go over our experience in migrating all these technologies and the pitfalls and powerups that we encountered along the way. We will touch on the planning processes we went through for each migration and the time we needed to get each migration from conception to deployment in production. We will also look at: - RBAC - Networking - Producer and consumer code and configuration changes - Ideas on automation - Possible architecture changes Lastly, we will touch on the business conversations that were needed to get the green light for each of these migrations, and the decisions made for migrating to each technology. We will analyze the things we did right along our migration journeys and the things we would improve for the next time we need to migrate a system."
Recomendados
Mais conteúdo relacionado
Similar a Migrating Your System to the Different Kafka Platforms Available wtih Bhavesh Sooka & Jonathan Lew
Similar a Migrating Your System to the Different Kafka Platforms Available wtih Bhavesh Sooka & Jonathan Lew(20)
![[Spark Summit EU 2017] Apache spark streaming + kafka 0.10 an integration story](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkstreamingkafka0-171026072955-thumbnail.jpg?width=768&fit=bounds)
Mais de HostedbyConfluent
Mais de HostedbyConfluent(20)
Último
Último(20)
Migrating Your System to the Different Kafka Platforms Available wtih Bhavesh Sooka & Jonathan Lew
- 1. The Chronicles of Kafka: A Migration Story Bhavesh Sooka and Jonathan Lew
- 3. Our Goal For the audience to learn something from our migration experience and take it back to their migration journey.
- 5. Gamified Fitness VDP & V1 - 17 clusters - 40 Markets Globally using 8 AWS Regions - 27 million monthly users - 20 Device Manufacturers - 10 million workouts/day - 124m messages in a day - 1000s of microservices
- 6. Prologue 2015 – V1: Kafka 0.10 on EC2 2019 – V1: Upgrade Apache Kafka 0.10 -> CP 5.0.1 2021: - VDP: Tibco EMS -> CP 6.0.1 2014 – V1: Kafka on K8s 2015 – VDP: Tibco EMS (Event Messaging Service) 2020 – V1 Upgrade (x2) CP 5.0.1 -> 5.4.1 -> CP 6.0.1 2021 – The Migration: Enter your speakers 1 2 3 5 7 6 4 7 Years
- 7. Nov 2021 – Feb 2022: ARB x2 June – Sept: Execution 4 years: Lead up Feb - June: Planning Oct: Hyper Care 4 5 3 2 1 Migration Timeline 11 Months
- 8. ARB VS
- 9. Planning Planning took 4 Months Planning Outcomes - Design decisions - Configuration - Secret Storage - Networking - RBAC - Automation - Deployment plan
- 10. Planning: Design Decisions Two DD’s we want to mention in specific are: Code and config split Vault Module
- 11. Planning: Networking Private Link TGW VPC Peering
- 12. SSO enabled and LDAP connectivity Planning: RBAC Split into 2 problems to Solve People Applications
- 13. Planning: Automation Topic creation Cluster and Networking
- 14. Planning: Deployment Day Goal: 100% Uptime During Migration • Move Producer • Drain • Move Consumer • Repeat
- 15. Line Manager • Decision Maker • Autonomy • On our Side Operational Requests SNAG: CIO - Flight plan - Commercials Business Decisions/Red Tape
- 16. This was child’s play!!!!! 4 Environments = 4 Months 1 month of Hyper Care Execution
- 17. • Agile vs Waterfall • Stakeholders buy in and trust • Contextualized planning • Team dynamic • Business vs IT • Industry Experts in Kafka Learnings
- 18. Thank You Confluent, Jack and Dan Vitality Group/Vitality Global Synthesis You
- 19. And so, they lived happily ever after… #KafkaSummit2024