O SlideShare utiliza cookies para otimizar a funcionalidade e o desempenho do site, assim como para apresentar publicidade mais relevante aos nossos usuários. Se você continuar a navegar o site, você aceita o uso de cookies. Leia nosso Contrato do Usuário e nossa Política de Privacidade.
O SlideShare utiliza cookies para otimizar a funcionalidade e o desempenho do site, assim como para apresentar publicidade mais relevante aos nossos usuários. Se você continuar a utilizar o site, você aceita o uso de cookies. Leia nossa Política de Privacidade e nosso Contrato do Usuário para obter mais detalhes.
Building distributed systems is challenging. Luckily, Apache Kafka provides a powerful toolkit for putting together big services as a set of scalable, decoupled components. In this talk, I'll describe some of the design tradeoffs when building microservices, and how Kafka's powerful abstractions can help. I'll also talk a little bit about what the community has been up to with Kafka Streams, Kafka Connect, and exactly-once semantics.
Presentation by Colin McCabe, Confluent, Big Data Day LA
● Example network service
• Why microservices?
• Why Kafka?
● Apache Kafka background
● How Kafka helps scale microservices
● Kafka APIs
• Kafka Connect API
• Kafka Streams API
● Wrap up
● New Kafka features and improvements
● Improving Decoupling
• Everything in one big app: no decoupling
• Microservices with REST: multiple services
• Microservices with Kafka: decoupled services sharing
● Improving Scalability
• Everything in one big app: single node
• Microservices with REST: one node per service
• Microservices with Kafka: scalable microservices
● A distributed streaming platform
● Kafka was built at LinkedIn around 2010
● Multi-platform: clients in Java, Scala, C, C++, Python, Go, C#, …
Kafka is Durable
● Data is
and persisted to
● Configurable log
● Consumers can
read from any
part of the log.
Scaling with Kafka
● Can have multiple producers writing to a topic
● Can have multiple consumers reading from a topic
● Can add new microservices to consume data easily
• Example: add more microservices processing views
• Organize microservices around data, rather than APIs
● Can add more Kafka brokers to handle more messages and
• Horizontal scalability
Scaling a Topic with Multiple Partitions
Backend Backend Backend
Load Balancing with Multiple Consumers
Connecting to External Services
Kafka Connect API
● Responsible for
between Kafka and
an external system
Kafka Streams API
● Process streams of data.
● Fault-tolerant and
Calculating News Reader Metrics
clicks locations clicks per location
Kafka Streams API
● Inputs and outputs are
provided by Kafka
New Kafka Features and Improvements
● Exactly once semantics in Kafka 0.11
● Consumer and producer performance improvements
• Up to +20% producer throughput
• Up to +50% consumer throughput
● Better CLASSPATH isolation for Kafka Connect connectors
● The loose coupling, deployability, and testability of
microservices makes them a great way to scale.
● Apache Kafka is an incredibly useful building block for many
● Kafka is reliable and does the heavy lifting
● Kafka Connect is a great API for connecting with external
databases, Hadoop clusters, and other external systems.
● Kafka Streams can process data in realtime.