Scaling big with Apache Kafka

•

2 gostaram•261 visualizações

Apache Kafka sits at the core of the modern scalable event driven architecture. It’s no longer used only as logging infrastructure, but as a core component in thousands of companies around the world. It has the unique capability to provide low-latency, fault-tolerant pipeline at scale that is very important for today’s world of big data. During this talk we’ll see what makes Apache Kafka perfect for the job. We’ll explore how to optimize it for throughput or for durability. And we’ll also go over the messaging semantics it provides. Last but not least, we’ll see how Apache Kafka can help us solve some everyday problems that we face when we build large scale systems in an elegant way.

Software

Topic
Message Message Message Message Message Message

Partition - ordered, immutable sequence
Message Message Message Message Message
Message Message Message Message Message
Message Message Message Message Message
Partition 0
Partition 1
Partition 2

Offset
Message Message Message Message MessagePartition 0
Offset: 1

Offset
Message Message Message Message MessagePartition 0
Offset: 2

Multi Subscriber
Message Message Message Message Message
Offset: 5Offset: 3
Consumer: 2 Consumer: 1

Broker
Broker
Broker
P0
P0
P0
P1
P1
P1
P2
P2
P2
Partitioned and Replicated

Broker
Broker
Broker
P0
P0
P0
P1
P1
P1
P2
P2
P3
Fault Tolerant

Broker
Broker
Broker
P0
P0
P0
P1
P1
P1
P2
P2
P2
Producers
Producer
Producer

Broker
Broker
Broker
P0
P0
P0
P1
P1
P1
P2
P2
P2
Producer
Producer
Consumer
Consumer
Consumer Group 1
Consumer
Consumer
Consumer
Consumer Group 2
Consumers

Broker
Broker
P0
P0
P1
P1
P2
P2
Producer
Consumer
Consumer
Consumer Group 1
ZooKeeper
Get Broker ID Update Offset

System for “collecting and delivering
high volumes of log data with low
latency”

Publish - Subscribe
Or Kafka for interservice communication

Built-in partitioning, replication, and
fault-tolerance

One leader for every partition
Follower
Leader
Follower
Follower
Producer Consumer

In-sync Replicas
Follower
Leader
Follower
Follower
Producer Consumer

In-sync Replicas
Leader
Follower
Follower
Producer Consumer

Tune for lower latency
● Acknowledgement after persisted on the leader
● Can lost message on leadership changes
● At-most-once semantic

Tune for durability
● Acknowledgement after persisted on all ISR
(after committed)
● No data loss
● At-least-once sematic

Kafka as a message bus
KafkaUpstream Downstream

KafkaUpstream
Downstrea
m
Schema
Registry
Publish Schema Fetch Schema
https://docs.confluent.io/current/schema-registry/docs/index.html

How to make sure something is
durably stored and will be processed
exactly once?

Idempotency + at least once delivery
Process 1 Process 2Kafka Consumer

Out of the box exactly once delivery
after 0.11

https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/

Order
Service
Kafka
Payment
Consumer
Payment
Provider
Handling Failures

Order
Service
Kafka
Payment
Consumer
Payment
Provider
Clogged processing

Order
Service
Kafka
Payment
Consumer
Payment
Provider
Dead Letter Queue
Payment Retry 0
Payment Retry 1
Dead Letter Queue

Order
Service
Kafka Payment Consumer
Payment
Provider
Dead Letter Queue
Payment Retry 0
Payment Retry 1
Dead Letter Queue
Payment Consumer
Payment Consumer

Regional Cluster
Local Producer Regional Kafka
DC1
Local Producer Regional Kafka
DC2

Aggregated Cluster
Local Producer Regional Kafka
DC1
Local Producer Regional Kafka
DC2
Kafka Replicator
Aggregated
Kafka
DC3

https://github.com/confluentinc/kafka-rest

Kafka
Upstream
Downstream
Kafka REST
Proxy
Kafka REST Proxy

Audit Library
Regional Kafka
Service
Kafka REST
Proxy
Audit
Library
Audit
Library

Chaperone Service
Regional Kafka
Service
Kafka REST
Proxy
Audit
Library
Audit
Library
Chaperone
Service

Chaperone Collector
Regional Kafka
Service
Kafka REST
Proxy
Audit
Library
Audit
Library
Aggregate Kafka
Chaperone
Service
Chaperone
Service
Chaperone
Collector
DB

Summary
Tune for durability
Define Avro Schemas

Summary
Tune for durability
Define Avro Schemas
Use Kafka REST Proxy

Summary
Tune for durability
Define Avro Schemas
Use Kafka REST Proxy
Add idempotency checks

Summary
Tune for durability
Define Avro Schemas
Use Kafka REST Proxy
Add idempotency checks
Use Dead Letter Queue

Summary
Tune for durability
Define Avro Schemas
Use Kafka REST Proxy
Add idempotency checks
Use Dead Letter Queue
Monitor everything

THANK YOU!
Nikolay Stoitsev
Sr. Software Engineer @ Uber

Mais conteúdo relacionado

Mais procurados

Modern businesses have data at their core, and this data is changing continuously. How can we harness this torrent of continuously changing data in real time? The answer is stream processing, and one system that has become a core hub for streaming data is Apache Kafka. This presentation will give a brief introduction to Apache Kafka and describe its usage as a platform for streaming data. It will explain how Kafka serves as a foundation for both streaming data pipelines and applications that consume and process real-time data streams. It will introduce some of the newer components of Kafka that help make this possible, including Kafka Connect, a framework for capturing continuous data streams, and Kafka Streams, a lightweight stream processing library. This is talk 1 out of 6 from the Kafka Talk Series. http://www.confluent.io/apache-kafka-talk-series/introduction-to-stream-processing-with-apache-kafka

Introduction To Streaming Data and Stream Processing with Apache Kafka

confluent

Kevin Jones, NGNIX - NGINX is one of the most popular images on Docker Hub and has been at the forefront of the web since the early 2000's. In this talk we will discuss how and why NGINX's lightweight and powerful architecture makes it a very popular choice for securing containerized applications as a sidecar reverse proxy within containers. We will highlight important aspects of application security that NGINX can help with, such as TLS, HTTP, AuthN, AuthZ and traffic control.

Securing Your Containerized Applications with NGINX

Docker, Inc.

Introducing Exactly Once Semantics To Apache Kafka

Apurva Mehta

RedisConf17 - Pain-free Pipelining

Redis Labs

A 9 line shop powered by armeria

LINE Corporation

Kafka Summit SF 2017 - Running Kafka as a Service at Scale

confluent

Apache kafka

MvkZ

Openzipkin conf: Zipkin at Yelp

Prateek Agarwal

B 2 line game cloud - our personal ec2

LINE Corporation

Cloud migration: it's practically a rite of passage for anyone who's built infrastructure on bare metal. When we migrated our 5-year-old Kafka deployment from the datacenter to GCP, we were faced with the task of making our highly mutable server infrastructure more cloud-friendly. This led to a surprising decision: we chose to run our Kafka cluster on Kubernetes. I'll share war stories from our Kafka migration journey, explain why we chose Kubernetes over arguably simpler options like GCP VMs, and present the lessons we learned while making our way toward a stable and self-healing Kubernetes deployment. I'll also go through some improvements in the more recent Kafka releases that make upgrades crucial for any Kafka deployment on immutable and ephemeral infrastructure. You'll learn what happens when you try to run one complex distributed system on top of another, and come away with some handy tricks for automating cloud cluster management, plus some migration pitfalls to avoid. And if you're not sure whether running Kafka on Kubernetes is right for you, our experiences should provide some extra data points that you can use as you make that decision.

Kafka on Kubernetes: Keeping It Simple (Nikki Thean, Etsy) Kafka Summit SF 2019

confluent

Apache Kafka – (Pattern and) Anti-Pattern

confluent

Codeless pipelines with pulsar and flink

Timothy Spann

Kafka on Pulsar

StreamNative

In this talk I go over the basic theory of messaging in distributed systems, the different message delivery guarantees in Kafka and the to use them. I focus on exactly once delivery guarantees and the way Kafka implements it with transaction based messaging protocol. Including a discussion of the latency/throughput trade-offs, resource utilisation and its overall advantages and shortcomings. Finally, I show a use-case at Wix where exactly once delivery helped us solve a big problem.

Exactly Once Delivery with Kafka - Kafka Tel-Aviv Meetup

Natan Silnitsky

Robust Stream Processing with Apache Flink

Jamie Grier

Mais procurados (15)

Introduction To Streaming Data and Stream Processing with Apache Kafka

Securing Your Containerized Applications with NGINX

Introducing Exactly Once Semantics To Apache Kafka

RedisConf17 - Pain-free Pipelining

A 9 line shop powered by armeria

Kafka Summit SF 2017 - Running Kafka as a Service at Scale

Apache kafka

Openzipkin conf: Zipkin at Yelp

B 2 line game cloud - our personal ec2

Kafka on Kubernetes: Keeping It Simple (Nikki Thean, Etsy) Kafka Summit SF 2019

Apache Kafka – (Pattern and) Anti-Pattern

Codeless pipelines with pulsar and flink

Kafka on Pulsar

Exactly Once Delivery with Kafka - Kafka Tel-Aviv Meetup

Robust Stream Processing with Apache Flink

Semelhante a Scaling big with Apache Kafka

Trivadis TechEvent 2016 Apache Kafka - Scalable Massage Processing and more! ...

Trivadis

https://www.meetup.com/new-york-city-apache-pulsar-meetup/events/283837865/ Learn how to use Apache Pulsar and Apache NiFi to Stream to your Data Lake Discover how to stream data to and from your data lake or data mart using Apache Pulsar™ and Apache NiFi®. Learn how these cloud-native, scalable open-source projects built for streaming data pipelines work together to enable you to quickly build applications with minimal coding. |WHAT THE SESSION WILL COVER| Best Practices for using Pulsar and NiFi A deep dive on Apache NiFi's Pulsar connector and demos Building an End-to-End Application in the Hybrid Cloud Attend for a chance to win a We <3 Pulsar t-shirt! The first 50 registrants who register through here [https://hubs.ly/Q013LTpn0] will be entered in a drawing! —------------------------ |AGENDA| 6:00 - 7:00 PM EST: Presentation - Tim Spann, StreamNative Developer Advocate 7:00 - 8:00 PM EST: Presentation - John Kuchmek, Cloudera Principal Solutions Engineer 8:00 - 8:30 PM EST: Q&A + Networking —------------------------ |ABOUT THE SPEAKERS| John Kuchmek is a Principal Solutions Engineer for Cloudera. Before joining Cloudera, John transitioned to the Autonomous Intelligence team where he was in charge of integrating the platforms to allow data scientists to work with various types of data. Tim Spann is a Developer Advocate for StreamNative. He works with StreamNative Cloud, Apache Pulsar™, Apache Flink®, Flink® SQL, Big Data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, messaging, streaming technologies, and Java programming. Previously, he was a Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Engineer at Hortonworks, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on Big Data, Cloud, IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as ApacheCon, DeveloperWeek, Pulsar Summit and many more. He holds a BS and MS in computer science. He is currently working on a book about the FLiP Stack.

[March sn meetup] apache pulsar + apache nifi for cloud data lake

Timothy Spann

Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning

Guido Schmutz

After a quick overview and introduction of Apache Kafka, this session cover two components which extend the core of Apache Kafka: Kafka Connect and Kafka Streams/KSQL. Kafka Connects role is to access data from the out-side-world and make it available inside Kafka by publishing it into a Kafka topic. On the other hand, Kafka Connect is also responsible to transport information from inside Kafka to the outside world, which could be a database or a file system. There are many existing connectors for different source and target systems available out-of-the-box, either provided by the community or by Confluent or other vendors. You simply configure these connectors and off you go. Kafka Streams is a light-weight component which extends Kafka with stream processing functionality. By that, Kafka can now not only reliably and scalable transport events and messages through the Kafka broker but also analyse and process these event in real-time. Interestingly Kafka Streams does not provide its own cluster infrastructure and it is also not meant to run on a Kafka cluster. The idea is to run Kafka Streams where it makes sense, which can be inside a “normal” Java application, inside a Web container or on a more modern containerized (cloud) infrastructure, such as Mesos, Kubernetes or Docker. Kafka Streams has a lot of interesting features, such as reliable state handling, queryable state and much more. KSQL is a streaming engine for Apache Kafka, providing a simple and completely interactive SQL interface for processing data in Kafka.

Kafka Connect & Kafka Streams/KSQL - the ecosystem around Kafka

Guido Schmutz

Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can me make sure that all these event are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amount of messages between a source and a target. This session will start with an introduction into Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table. Additionally the Kafka ecosystem will be covered as well as the integration of Kafka in the Oracle Stack, with products such as Golden Gate, Service Bus and Oracle Stream Analytics all being able to act as a Kafka consumer or producer.

Apache Kafka - Scalable Message-Processing and more !

Guido Schmutz

[Big Data Spain] Apache Spark Streaming + Kafka 0.10: an Integration Story

Joan Viladrosa Riera

Apache Kafka - Scalable Message-Processing and more !

Guido Schmutz

Confluent Cloud runs a modified version of Apache Kafka - redesigned to be cloud-native and deliver a serverless user experience. In this talk, we will discuss key improvements we've made to Kafka and how they contribute to Confluent Cloud availability, elasticity, and multi-tenancy. You'll learn about innovations that you can use on-prem, and everything you need to make the most of Confluent Cloud.

Not Your Mother's Kafka - Deep Dive into Confluent Cloud Infrastructure | Gwe...

HostedbyConfluent

ndependent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. How can me make sure that all these event are accepted and forwarded in an efficient and reliable way? This is where Apache Kafaka comes into play, a distirbuted, highly-scalable messaging broker, build for exchanging huge amount of messages between a source and a target. This session will start with an introduction into Apache and presents the role of Apache Kafka in a modern data / information architecture and the advantages it brings to the table. Additionally the Kafka ecosystem will be covered as well as the integration of Kafka in the Oracle Stack, with products such as Golden Gate, Service Bus and Oracle Stream Analytics all being able to act as a Kafka consumer or producer.

Apache Kafka - Scalable Message-Processing and more !

Guido Schmutz

https://github.com/tspannhw/SpeakerProfile/tree/main/2022/talks Fast Streaming into Clickhouse with Apache Pulsar https://github.com/tspannhw/FLiPC-FastStreamingIntoClickhouseWithApachePulsar https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup/events/285271332/ Fast Streaming into Clickhouse with Apache Pulsar - Meetup 2022 StreamNative - Apache Pulsar - Stream to Altinity Cloud - Clickhouse May the 4th Be With You! 04-May-2022 Clickhosue Meetup CREATE TABLE iotjetsonjson_local ( uuid String, camera String, ipaddress String, networktime String, top1pct String, top1 String, cputemp String, gputemp String, gputempf String, cputempf String, runtime String, host String, filename String, host_name String, macaddress String, te String, systemtime String, cpu String, diskusage String, memory String, imageinput String ) ENGINE = MergeTree() PARTITION BY uuid ORDER BY (uuid); CREATE TABLE iotjetsonjson ON CLUSTER '{cluster}' AS iotjetsonjson_local ENGINE = Distributed('{cluster}', default, iotjetsonjson_local, rand()); select uuid, top1pct, top1, gputempf, cputempf from iotjetsonjson where toFloat32OrZero(top1pct) > 40 order by toFloat32OrZero(top1pct) desc, systemtime desc select uuid, systemtime, networktime, te, top1pct, top1, cputempf, gputempf, cpu, diskusage, memory,filename from iotjetsonjson order by systemtime desc select top1, max(toFloat32OrZero(top1pct)), max(gputempf), max(cputempf) from iotjetsonjson group by top1 select top1, max(toFloat32OrZero(top1pct)) as maxTop1, max(gputempf), max(cputempf) from iotjetsonjson group by top1 order by maxTop1 Tim Spann Developer Advocate StreamNative

Fast Streaming into Clickhouse with Apache Pulsar

Timothy Spann

F_1330_Narkhede_Kafka .pptx

NIMITJAIN71

Ran Silberman, developer & technical leader at LivePerson presents how LivePerson moved their data platform from a legacy ETL concept to new "Data Integration" concept of our era. Kafka is the main infrastructure that holds the backbone for data flow in the new Data Integration. Having that said, Kafka cannot come by itself. Other supporting systems like Hadoop, Storm, and Avro protocol were also integrated. In this lecture Ran will describe the implementation in LivePerson and will share some tips and how to avoid pitfalls. Read More: https://connect.liveperson.com/community/developers/blog/2013/11/21/from-a-kafkaesque-story-to-the-promised-land

From a Kafkaesque Story to The Promised Land at LivePerson

Apache kafka

Apache kafka

Apache kafka

Apache kafka

Apache kafka

Apache kafka

Spark Streaming has supported Kafka since it's inception, but a lot has changed since those times, both in Spark and Kafka sides, to make this integration more fault-tolerant and reliable.Apache Kafka 0.10 (actually since 0.9) introduced the new Consumer API, built on top of a new group coordination protocol provided by Kafka itself. So a new Spark Streaming integration comes to the playground, with a similar design to the 0.8 Direct DStream approach. However, there are notable differences in usage, and many exciting new features. In this talk, we will cover what are the main differences between this new integration and the previous one (for Kafka 0.8), and why Direct DStreams have replaced Receivers for good. We will also see how to achieve different semantics (at least one, at most one, exactly once) with code examples. Finally, we will briefly introduce the usage of this integration in Billy Mobile to ingest and process the continuous stream of events from our AdNetwork.

Spark streaming + kafka 0.10

Joan Viladrosa Riera

How to build 1000 microservices with Kafka and thrive

Natan Silnitsky

Semelhante a Scaling big with Apache Kafka (20)

Trivadis TechEvent 2016 Apache Kafka - Scalable Massage Processing and more! ...

[March sn meetup] apache pulsar + apache nifi for cloud data lake

Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning

Kafka Connect & Kafka Streams/KSQL - the ecosystem around Kafka

Apache Kafka - Scalable Message-Processing and more !

[Big Data Spain] Apache Spark Streaming + Kafka 0.10: an Integration Story

Apache Kafka - Scalable Message-Processing and more !

Not Your Mother's Kafka - Deep Dive into Confluent Cloud Infrastructure | Gwe...

Apache Kafka - Scalable Message-Processing and more !

Fast Streaming into Clickhouse with Apache Pulsar

F_1330_Narkhede_Kafka .pptx

From a Kafkaesque Story to The Promised Land at LivePerson

Apache kafka

Spark streaming + kafka 0.10

How to build 1000 microservices with Kafka and thrive

Mais de Nikolay Stoitsev

As engineers, we like to solve problems by building solutions from scratch. Even though on some occasions it’s better to buy and integrate existing software. But how come? Are engineers who don’t always deliver from scratch real engineers? The goal of this talk is to answer all important questions about making build vs buy decisions. We’ll see how to define a clear strategy for making such decisions. And we’ll explore how to select and integrate existing software efficiently. Even if your company doesn’t have the habit of doing it.

Building vs Buying Software

Scaling big with Apache Kafka

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (15)

Semelhante a Scaling big with Apache Kafka

Semelhante a Scaling big with Apache Kafka (20)

Mais de Nikolay Stoitsev

Mais de Nikolay Stoitsev (20)

Último

Último (20)

Scaling big with Apache Kafka