Kafka in practice

•

6 likes•644 views

Caique Lima

How we use Apache Kafka

Data & Analytics

KAFKA in practice
Caique Lima @ Easy Taxi

Apache Kafka
distributed, partitioned, replicated
commit log service

Kafka at Easy Taxi
• +600 mi unique events/day
• Currently on 0.9.0
• 3 brokers
• 10 topics
• X consumers

Architecture
Api Kafka
Consumer 1
Consumer 2
Consumer 3
S3
Redis
Consumer N
BQ
Api
…
Redash

Do’s
• Rsyslog as a scalable simple producer
• Use BigQuery to allow SQL
• Start building the event based culture
• Right-time before real-time

Dont’s
• Doesn't run stress tests
• Increase architecture’s complexity
• Produce useless data
• Go live with default configs

Supply demand analysis
• Allow us to understand the area behaviour
• Generate a “health indicator” based on
predefined parameters
• Take actions without any human interaction

References
• Confluent blog
• Kafka Summit
• Jay Kreps interview
• Jepsen: Kafka
• Documentation
• Dados Brasil Slack

What's hot

Feedback on AWS re:invent 2016Laurent Bernaille

(New)SQL on AWS: Aurora serverlessClaudio Pontili

The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018iguazio

Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020confluent

Elastic Search Meetup Special - Yann Cluchey, Cogenta Internet World

Introduction to Big Data InfrastructureSilota Inc.

Building Event Streaming Microservices with Spring Boot and Apache Kafka | Ja...HostedbyConfluent

Cost Optimization Best Practices from Trend Micro Cliff Chao-kuan Lu

Kafka Summit NYC 2017 - The Real-time Event Driven Bank: A Kafka Story confluent

Real Time Data Infrastructure team overviewMonal Daxini

Snowplow and Kinesis - Presentation to the inaugural Amazon Kinesis London Us...Alexander Dean

Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...HostedbyConfluent

Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...iguazio

AWS re:Invent 2016: Content and Data Platforms at Vevo: Rebuilding and Scalin...AwsReinventSlides

Serverless in the Azure WorldKasun Kodagoda

AWS for Java Developers workshopRory Preddy

Icinga Camp Bangalore - Icinga and Icinga Director Icinga

Nordstrom's Event-Sourced Architecture and Kafka-as-a-Service | Adam Weyant a...HostedbyConfluent

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...HostedbyConfluent

Azuresatpn19 - An Introduction To Azure Data FactoryRiccardo Perico

What's hot (20)

Feedback on AWS re:invent 2016

(New)SQL on AWS: Aurora serverless

The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018

Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020

Elastic Search Meetup Special - Yann Cluchey, Cogenta

Introduction to Big Data Infrastructure

Building Event Streaming Microservices with Spring Boot and Apache Kafka | Ja...

Cost Optimization Best Practices from Trend Micro

Kafka Summit NYC 2017 - The Real-time Event Driven Bank: A Kafka Story

Real Time Data Infrastructure team overview

Snowplow and Kinesis - Presentation to the inaugural Amazon Kinesis London Us...

Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...

Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...

AWS re:Invent 2016: Content and Data Platforms at Vevo: Rebuilding and Scalin...

Serverless in the Azure World

AWS for Java Developers workshop

Icinga Camp Bangalore - Icinga and Icinga Director

Nordstrom's Event-Sourced Architecture and Kafka-as-a-Service | Adam Weyant a...

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...

Azuresatpn19 - An Introduction To Azure Data Factory

Viewers also liked

SQL for Everything at CWT2014N Masahiro

PrestoMK JUNG

Scaling Business Intelligence: from infrastruture to visualization (pt-br)Fernando Spanó Junqueira de Paiva

Presto - SQL on anythingGrzegorz Kokosiński

チームを動かすデザイナーKeisuke Tsukayoshi

金融機関でのHive/Presto事例紹介Amazon Web Services Japan

Presto As A Service - Treasure DataでのPresto運用事例Taro L. Saito

爆速クエリエンジン”Presto”を使いたくなる話Kentaro Yoshida

Apache Sparkを用いたスケーラブルな時系列データの異常検知モデル学習ソフトウェアの開発Ryo 亮 Kawahara 河原

re:dash is awesomeHiroshi Toyama

Viewers also liked (10)

SQL for Everything at CWT2014

Presto

Scaling Business Intelligence: from infrastruture to visualization (pt-br)

Presto - SQL on anything

チームを動かすデザイナー

金融機関でのHive/Presto事例紹介

Presto As A Service - Treasure DataでのPresto運用事例

爆速クエリエンジン”Presto”を使いたくなる話

Apache Sparkを用いたスケーラブルな時系列データの異常検知モデル学習ソフトウェアの開発

re:dash is awesome

Similar to Kafka in practice

Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...confluent

Serverless Stream Processing with Bill Bejeckconfluent

Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)Spark Summit

introductiontoapachekafka-201102140206.pdfTarekHamdi8

Introduction to Apache KafkaAIMDek Technologies

Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...Amazon Web Services

Reducing Microservice Complexity with Kafka and Reactive Streamsjimriecken

CouchbasetoHadoop_Matt_Michael_Justin v4Michael Kehoe

Microservices Manchester: Serverless Architectures By Rafal GancarzOpenCredo

Introduction to KafkaAkash Vacher

AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017Monal Daxini

Kafka Streams for Java enthusiastsSlim Baltagi

Intro to AWS Lambda Amazon Web Services

REDSHIFT - AmazonDouglas Bernardini

Akka Streams And Kafka Streams: Where Microservices Meet Fast DataLightbend

Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...confluent

Integrating Apache Kafka Into Your Environmentconfluent

Building a Streaming Pipeline on Kubernetes Using Kafka Connect, KSQLDB & Apa...HostedbyConfluent

Micro services ArchitectureJenis Dharmadurai

GraphQL API on a Serverless EnvironmentItai Yaffe

Similar to Kafka in practice (20)

Bringing Streaming Data To The Masses: Lowering The “Cost Of Admission” For Y...

Serverless Stream Processing with Bill Bejeck

Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)

introductiontoapachekafka-201102140206.pdf

Introduction to Apache Kafka

Building a Real Time Dashboard with Amazon Kinesis, Amazon Lambda and Amazon ...

Reducing Microservice Complexity with Kafka and Reactive Streams

CouchbasetoHadoop_Matt_Michael_Justin v4

Microservices Manchester: Serverless Architectures By Rafal Gancarz

Introduction to Kafka

AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017

Kafka Streams for Java enthusiasts

Intro to AWS Lambda

REDSHIFT - Amazon

Akka Streams And Kafka Streams: Where Microservices Meet Fast Data

Kafka: Journey from Just Another Software to Being a Critical Part of PayPal ...

Integrating Apache Kafka Into Your Environment

Building a Streaming Pipeline on Kubernetes Using Kafka Connect, KSQLDB & Apa...

Micro services Architecture

GraphQL API on a Serverless Environment

Recently uploaded

Data Factory in Microsoft Fabric (MsBIP #82)Cathrine Wilhelmsen

办理学位证中佛罗里达大学毕业证,UCF成绩单原版一比一F sss

PKS-TGC-1084-630 - Stage 1 Proposal.pptxPramod Kumar Srivastava

2006_GasProcessing_HB (1).pdf HYDROCARBON PROCESSINGmarianagonzalez07

NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...Amil Baba Dawood bangali

20240419 - Measurecamp Amsterdam - SAM.pdfHuman37

Multiple time frame trading analysis -brianshannon.pdfchwongval

Advanced Machine Learning for Business ProfessionalsVICTOR MAESTRE RAMIREZ

Call Girls in Saket 99530🔝 56974 Escort Service9953056974 Low Rate Call Girls In Saket, Delhi NCR

Building on a FAIRly Strong Foundation to Connect Academic Research to Transl...Jack DiGiovanna

Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024thyngster

科罗拉多大学波尔得分校毕业证学位证成绩单-可办理e4aez8ss

Identifying Appropriate Test Statistics Involving Population MeanMYRABACSAFRA2

Deep Generative Learning for All - The Gen AI Hype (Spring 2024)Universitat Politècnica de Catalunya

RS 9000 Call In girls Dwarka Mor (DELHI)⇛9711147426🔝Delhijennyeacort

Biometric Authentication: The Evolution, Applications, Benefits and Challenge...GQ Research

Predicting Salary Using Data Science: A Comprehensive Analysis.pdfBoston Institute of Analytics

How we prevented account sharing with MFAAndrei Kaleshka

Generative AI for Social Good at Open Data Science East 2024Colleen Farrelly

GA4 Without Cookies [Measure Camp AMS]📊 Markus Baersch

Recently uploaded (20)

Data Factory in Microsoft Fabric (MsBIP #82)

办理学位证中佛罗里达大学毕业证,UCF成绩单原版一比一

PKS-TGC-1084-630 - Stage 1 Proposal.pptx

2006_GasProcessing_HB (1).pdf HYDROCARBON PROCESSING

NO1 Certified Black Magic Specialist Expert Amil baba in Lahore Islamabad Raw...

20240419 - Measurecamp Amsterdam - SAM.pdf

Multiple time frame trading analysis -brianshannon.pdf

Advanced Machine Learning for Business Professionals

Call Girls in Saket 99530🔝 56974 Escort Service

Building on a FAIRly Strong Foundation to Connect Academic Research to Transl...

Consent & Privacy Signals on Google *Pixels* - MeasureCamp Amsterdam 2024

科罗拉多大学波尔得分校毕业证学位证成绩单-可办理

Identifying Appropriate Test Statistics Involving Population Mean

Deep Generative Learning for All - The Gen AI Hype (Spring 2024)

RS 9000 Call In girls Dwarka Mor (DELHI)⇛9711147426🔝Delhi

Biometric Authentication: The Evolution, Applications, Benefits and Challenge...

Predicting Salary Using Data Science: A Comprehensive Analysis.pdf

How we prevented account sharing with MFA

Generative AI for Social Good at Open Data Science East 2024

GA4 Without Cookies [Measure Camp AMS]