Apache Kafka® at Dropbox
Peng Kang, Software Engineer, Dropbox + Richi Gupta, Engineering Manager, Dropbox As a scalable and reliable data streaming solution with a rich ecosystem, Kafka is widely adopted in Dropbox infrastructure in various scenarios. It is part of Dropbox’s analytics data pipeline, stream processing platform and more mission critical systems. Jetstream is the team that provides Kafka as a service in Dropbox infrastructure. We manage the clusters, develop tooling, and enforce policies, so that our users can enjoy a highly available and reliable service. In this talk, we will share our experiences and learnings running Kafka clusters, pipelines that enable high durability (direct writes to kafka) and availability (goscribe), the policies we enforce for high reliability, the tooling we have for maintenance and stress testing, and finally an overview of Dropbox’s next generation queueing service built on top Kafka. https://www.meetup.com/KafkaBayArea/events/266327152/
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Apache Kafka® at Dropbox
Semelhante a Apache Kafka® at Dropbox (20)
Mais de confluent
Mais de confluent (20)
Último
Último (20)
Apache Kafka® at Dropbox
- 2. Kafka@dropbox Peng Kang Richi Gupta “Achieving highly reliable message transfer pipeline with low operational cost “
- 3. Tired: Cloud file storage service provider Wired: Smart workspace to • Organize your content • Connect your tools • Bring everyone together, wherever you are Learn more about it @ https://blog.dropbox.com/topics/company/dropbox--the-smart-workspace https://blog.dropbox.com/topics/product/introducing-dropbox-spaces
- 4. Agenda 1. Use cases overview 2. Configurations & Scale & Trade-offs 3. Challenges 4. Highlights of our works
- 5. Use cases: Async Analytics • Availability SLA – 99.9% • Durability SLA – 99.9% GoScribe leaf GoScribe leaf GoScribe leaf GoScribe leaf GoScribe agregators GoScribe aggregators Kafka connect Hive Other consumers Kafka
- 6. Use cases: Sync Search indexing pipeline, campaigns, cape • Availability SLA – 99.9% • Durability SLA – 99.99% Python/Go producers Python/Go consumersKafka
- 7. Configuration and Scale • Kafka version – 0.10.2.1, H/W and S/W raid • Clients – Go and Python • Replication=3, min-isr=2 • >40 clusters • >400 brokers • 15M mps • 4GB/s • Avg produce time < 10 ms
- 8. Feature Support • No keyed messages • No ordering • At-least once message delivery
- 9. The great battle of reliability • KTLO Then: 80% of our engineer time • KTLO Now: 10% of our engineer time
- 10. Challenges Our Kafka clients aren’t robust enough Long & dangerous maintenance operations • Data retention – 1-3 day • Message size – 1MB • Cluster size - ~15 Insufficient understanding of failure scenarios • Regular automated DRTs (rolling restart, one broker down, broker replacement)
- 11. Challenges Keeping up with organic cluster growth • Semi-automatic topic scaling • Set cluster scaling policy Cannot tolerate cluster outage • Sharding • Emergency failover • Dynamic failover
- 12. Deep dives • Automated stress tester • Sharding for async data pipeline • Emergency failover for sync use cases • Queue service
- 13. Automated stress testing • Automatically find throughput limit of Kafka clusters • Its results are used for making Kafka usage policies • Standard procedure of verifying new hardware, version, setups, etc. • blog
- 14. Setup
- 15. Algorithm
- 16. Results
- 19. Queue service – a proxy of Kafka Kafka Kafka Kafka Kafka Queue service produce consume
- 20. Queue service – why are we building it? • Real time traffic control • Enforce safe traffic pattern • Easy to customize with in house technologies
- 21. Queue service - topology
- 22. Queue service – a modular design
- 23. Queue service – future development • Dynamic failover • Transparently back up Kafka data in cold storage • Pub sub service
- 24. Questions?
- 25. Thanks to Grzegorz Gronkowski Kedar Rudre Kelsey Fix Kevin Murphy Shuyao Bi
- 27. Thank You!