O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Apache Kafka on Kubernetes

988 visualizações

Publicada em

Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Pivotal Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value.

Apache Kafka® is providing developers a critically important component as they build and modernize applications to cloud-native architecture.

This talk will explore:
• Why cloud-native platforms and why run Apache Kafka on Kubernetes?
• What kind of workloads are best suited for this combination?
• Tips to determine the path forward for legacy monoliths in your application portfolio
• Demo: Running Apache Kafka as a Streaming Platform on Kubernetes

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Apache Kafka on Kubernetes

  1. 1. Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Kafka on Kubernetes
  2. 2. Speakers Kamala Dasika Product Team Cloud Platform and Ecosystem Pivotal @DasikaKN Michael Ng Product Manager Confluent @Micha8LNg
  3. 3. Changing Model for Application Delivery Development Deployment Sparingly at designated times Ready for prod at any time Architecture Abstraction Layer “Day 2” Ops App Server on Machine App on “disposable” infrastructure App and Data Monolith Microservices / Composite app Linear / Sequential Short cycles, test driven, iterative Many tools, ad hoc automation Manage services, not servers
  4. 4. Driven by Modern Cloud Native Platform Cont inuous Del ivery An idea in the morning can ship by evening Microservices Release #1 Microservices Release #2
  5. 5. Know your abstractions
  6. 6. Hardware IaaS CaaS PaaS FaaS HPE, Dell, IBM, Lenovo Cloud providers: AWS, GCP, Microsoft Azure, VMware Kubernetes Cloud Foundry, Heroku AWS Lambda, Azure Functions, OpenWhisk, kubeless, PFS
  7. 7. Hardware IaaS CaaS PaaS FaaS Strategic goal: Push as many workloads as technically feasible to the top of the platform hierarchy Higher flexibility and less enforcement of standards Lower development complexity and higher operational efficiency
  8. 8. Hardware IaaS CaaS PaaS FaaS • Data warehouses • Storage with regional implications • Hypervisors • Relational DBs • Packaged software • Virtual appliances • Application platforms (cloud abstraction) • Containerized Monoliths • Stateful workloads • Databases • Messaging • 12 -15 factor apps • stateful and stateless options • APIs • Streaming apps • DB copies • Change synchronization • Real time stream analytics
  9. 9. Where to run what?
  10. 10. Goal: Operational consistency AND multiple abstractions FREEDOM: Run Your App on Any Cloud AUTOMATION: Lowers Ops Cost and Increases Velocity RESILIENCY: Auto-Scale, Blue-green Deploys, Health Mgmt. GOVERNANCE: Common Developer-Self-Serviced Frameworks VISIBILITY: Full Stack Logging, Monitoring & Auditability SPEED & SCALE: Cut Cycle Time from Months to Days/Hours Built for Cloud Native New Initiatives Existing Workloads Modernize to Cloud-Native?
  11. 11. Outcomes trump technology bingo Needs High Dev Efficiency Needs Low Dev Efficiency App Rarely changes App Frequently changes • Traditional apps • Leave as-is • Re-architect • Replatform • CI/CD • Evaluate • Evaluate
  12. 12. Evaluating the Enterprise App Portfolio Business Technical Economic Score • Business criticality • Risk tolerance • Change frequency • Lifecycle stage • Domain expert availability • Licensing costs (HW / SW) • Time-to-market implications • Revenue impact-fullness • Codebase • Suitable framework / runtime • Relatively lightweight footprint • No “hard wired” file system dependencies • Usage / workload For each app
  13. 13. Application Transformation Strategies Re-Host Containerize aka “lift and shift” to CaaS and gain infrastructure + platform ops benefits Re-Platform Minimal readiness effort to run on PaaS (4 -7 factor apps + data service) that can run on PaaS Re-Factor Convert to cloud- native/microservice (12-15 factor app + data service) runs well on PaaS/FaaS Re-Build Decompose and rewrite from the ground up Application Portfolio Funneling Process Learn More: Tools and Recipes to Replatform Monolithic Apps to Modern Cloud Environments
  14. 14. Evolution of the enterprise application portfolio Learn More: Microservices, Events, and Breaking the Data Monolith with Kafka Microservices or Dist. Monolith
  15. 15. Now what about Kafka?
  16. 16. “The world’s most valuable resource is no longer oil, but data” - The Economist, May 6 2017
  17. 17. Apache Kafka is a Distributed Event Streaming Platform Process streams of events In real time, as they occur 110101 010111 001101 100010 Publish and subscribe to streams of events Similar to a message queue or enterprise messaging system 110101 010111 001101 100010 Store streams of events In a fault tolerant way 110101 010111 001101 100010
  18. 18. Events A Sale An Invoice A Trade A Customer Experience
  19. 19. What is a company? A business is a series of events and reacting to those events.
  20. 20. Events enable Microservice Architectures - Let’s refactor! Submit Order Orders Service Shipping Service Customer Service Webserver E-Commerce Microservices (with RPC) - Orders Service calls Shipping Service to tell it to ship item. - Shipping service looks up address to ship to (from Customer Service) - No Kafka 😢shipOrder() getCustomer()
  21. 21. Refactoring Orders and Shipping Message Broker (Kafka) Submit Order Order Created getCustomer() RPC Orders Service Shipping Service Customer Service Webserver - Orders Service no longer knows about the Shipping service (or any other service). - Events are fire and forget.
  22. 22. Refactoring Customers Customer Updated Submit Order Order Created Webserver KAFKA Orders Service Shipping Service Customer Service - Call to Customer service is gone. - Data is replicated, as events, into the shipping service, where it is queried locally.
  23. 23. Events are the key to scalable microservices Orders Service
  24. 24. Kafka on Kubernetes
  25. 25. Why run Kafka on Kubernetes? 33% of Confluent Customers plan to run Production Kafka on Kubernetes
  26. 26. Microservices - driving container and container orchestration adoption Breaking down the monolith - enables dev teams to work and scale independently µ Containers allow services to be independently deployable services ● Each service has its own purpose & implementation language ● Each service has its own dependencies Orchestration - running containers at scale ● Microservices = container deployments at scale ● Service discovery ● Scale services independently ● High Availability, Self Healing of services ● Hardware Abstraction
  27. 27. The world seems to think Kafka on Kubernetes is Difficult
  28. 28. There are some challenges Translating an existing architecture to Kubernetes: K8s pods, nodes, stateful sets, custom resources External access: What do we do with advertised listeners on Kubernetes? Managing configuration of Kafka, Zookeeper and all Confluent Components Persistent Storage options on prem and clouds Security Configuration and Upgrades
  29. 29. Confluent Operator enables you to: Automate provisioning of Kafka pods in minutes Monitor SLAs through Confluent Control Center or Prometheus Scale Kafka elastically & Automate rolling updates Built on our first hand knowledge of running Confluent at scale Confluent Operator is our Kubernetes Operator and our first step towards becoming cloud native CONFIDENTIAL
  30. 30. Helm - The Confluent Operator Package Manager ● Confluent Operator leverages Helm Charts to deploy, upgrade and uninstall Confluent Platform custom resources and pods ● Configuration front end for users to specify how a Confluent Platform Cluster is deployed: ○ # of replicas for Kafka, Zookeeper ○ Security and Authentication configuration ○ Persistent Storage configuration ● Cluster configuration edits are also performed using Helm Operator Helm Charts - yaml
  31. 31. Confluent Operator - Automated Provisioning
  32. 32. Confluent Operator - Scale Horizontally Automate Scaling: Spin up new brokers, connect workers easily Distribute partitions to new brokers: Determine balancing plan Execute balancing plan Monitor Resources
  33. 33. Confluent Operator - Rolling Upgrade of all components Automated Rolling Upgrades of all components - Kafka Brokers, Zookeeper, Connect, Control Center Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero under-replicated partitions 5. Upgrade the next broker
  34. 34. Confluent Operator - Automated Security Configuration SASL PLAIN and Mutual TLS Authentication Automate configuration of truststores and keystores with secret objects Automate configuration of Kafka and all Confluent Platform Components
  35. 35. Confluent Operator GA Plans We are in private Preview Release now 24 customers testing the Operator in Preview: ● Global customers ● Banks, Fin Tech, Retailers, Consumer Tech We are in the final stages of Preview and about to launch soon.
  36. 36. Make this easy!!!
  37. 37. Leverage multiple platforms/abstractions
  38. 38. Demo – Provision a Complex Cluster in K8s with Confluent Operator ● Provision: ○ 3 Kafka Brokers, 3 Zookeeper Nodes ○ Confluent Control Center ○ Provisioned on GKE (7 node cluster) ● Mutual TLS Authentication, External Access via Service Endpoints, DNS ● Pub/Sub example (remote access to Kafka Cluster)