Building data pipelines is pretty hard! Building a multi-datacenter active-active real time data pipeline for multiple classes of data with different durability, latency and availability guarantees is much harder. Real time infrastructure powers critical pieces of Uber (think Surge) and in this talk we will discuss our architecture, technical challenges, learnings and how a blend of open source infrastructure (Apache Kafka and Samza) and in-house technologies have helped Uber scale.

1. Real Time Data Analytics @
UberAnkur Bansal
Apache Big Data Europe
November 14, 2016

2. About Me
● Sr. Software Engineer, Streaming Team @ Uber
○ Streaming team supports platform for real time data
analytics: Kafka, Samza, Flink, Pinot.. and plenty more
○ Focused on scaling Kafka at Uber’s pace
● Staff software Engineer @ Ebay
○ Build & scale Ebay’s cloud using openstack
● Apache Kylin: Committer, Emeritus PMC

3. Agenda
● Real time Use Cases
● Kafka Infrastructure Deep Dive
● Our own Development:
○ Rest Proxy & Clients
○ Local Agent
○ uReplicator (Mirrormaker)
○ Chaperone (Auditing)
● Operations/Tooling

4. Important Use Cases

5. Stream
Processing
Real-time Price Surging
SURGE
MULTIPLIERS
Rider eyeballs
Open car information
KAFKA

6. Real-time Machine Learning - UberEats ETD

8. ● Fraud detection
● Share my ETA
And many more ...

9. Apache Kafka is Uber’s Lifeline

10. Kafka ecosystem @ Uber

11. 100s of billion
100s TB
Messages/day
bytes/day
Kafka cluster stats
Multiple data centers

12. Kafka Infrastructure Deep Dive

13. Requirements
● Scale to 100s Billions/day → 1 Trillion/day
● High Throughput ( Scale: 100s TB → PB)
● Low Latency for most use cases(<5ms )
● Reliability - 99.99% ( #Msgs Available /#Msgs Produced)
● Multi-Language Support
● Tens of thousands of simultaneous clients.
● Reliable data replication across DC

14. Kafka Pipeline
Local
Agent
uReplicator

15. Kafka Pipeline: Data Flow
1
2
3 5 7
64 8

16. Kafka Clusters
Local
Agent
uReplicator

17. Kafka Clusters
● Use case based clusters
○ Data (async, reliable)
○ Logging (High throughput)
○ Time Sensitive (Low Latency e.g. Surge, Push
notifications)
○ High Value Data (At-least once, Sync e.g. Payments)
● Secondary cluster as fallback
● Aggregate clusters for all data topics.

18. Kafka Clusters
● Scale to 100s Billions/day → 1 Trillion/day
● High Throughput ( Scale: 100s TB → PB)
● Low Latency for most use cases(<5ms )
● Reliability - 99.99% ( #Msgs Available /#Msgs Produced)
● Multi-Language Support
● Tens of thousands of simultaneous clients.
● Reliable data replication across DC

19. Kafka Rest Proxy
Local
Agent
uReplicator

20. Why Kafka Rest Proxy ?
● Simplified Client API
● Multi-lang support (Java, NodeJs, Python, Golang)
● Decouple client from Kafka broker
○ Thin clients = operational ease
○ Less connections to Kafka brokers
○ Future kafka upgrade
● Enhanced Reliability
○ Primary & Secondary Kafka Clusters

21. Kafka Rest Proxy: Internals

22. Kafka Rest Proxy: Internals

23. Kafka Rest Proxy: Internals
● Based on Confluent’s open sourced Rest Proxy
● Performance enhancements
○ Simple http servlets on jetty instead of Jersey
○ Optimized for binary payloads.
○ Performance increase from 7K* to 45-50K QPS/box
● Caching of topic metadata.
● Reliability improvements*
○ Support for Fallback cluster
○ Support for multiple Producers (SLA based segregation)
● Plan to contribute back to community
*Based on benchmarking & analysis done in Jun ’2015

24. Rest Proxy: performance (1 box)
Message rate (K/second) at single node
End-endLatency(ms)

25. Kafka Clusters + Rest Proxy
● Scale to 100s Billions/day → 1 Trillion/day
● High Throughput ( Scale: 100s TB → PB)
● Low Latency for most use cases(<5ms )
● Reliability - 99.99% ( #Msgs Available /#Msgs Produced)
● Multi-Language Support
● Tens of thousands of simultaneous clients.
● Reliable data replication across DC

26. Kafka Clients
Local
Agent
uReplicator

27. Client Libraries
● Support for multiple clusters.
● High Throughput
○ Non-blocking, async, batching
○ <1ms produce latency for clients
○ Handles Throttling/BackOff signals from Rest Proxy
● Topic Discovery
○ Discovers the kafka cluster a topic belongs
○ Able to multiplex to different kafka clusters
● Integration with Local Agent for critical data

28. Client Libraries
Add
Figure
What if there is
network glitch /
outage?

29. Client Libraries
Add
Figure

30. Kafka Clusters + Rest Proxy + Clients
● Scale to 100s Billions/day → 1 Trillion/day
● High Throughput ( Scale: 100s TB → PB)
● Low Latency for most use cases(<5ms )
● Reliability - 99.99% ( #Msgs Available /#Msgs Produced)
● Multi-Language Support
● Tens of thousands of simultaneous clients.
● Reliable data replication across DC

31. Local Agent
Local
Agent
uReplicator

32. Local Agent
● Local spooling in case of downstream outage/backpressure
● Backfills at the controlled rate to avoid hammering
infrastructure recovering from outage
● Implementation:
○ Reuses code from rest-proxy and kafka’s log module.
○ Appends all topics to same file for high throughput.

33. Local Agent Architecture
Add
Figure

34. Local Agent in Action
Add
Figure

35. Kafka Clusters + Rest Proxy + Clients + Local Agent
● Scale to 100s Billions/day → 1 Trillion/day
● High Throughput ( Scale: 100s TB → PB)
● Low Latency for most use cases(<5ms )
● Reliability - 99.99% ( #Msgs Available /#Msgs Produced)
● Multi-Language Support
● Tens of thousands of simultaneous clients.
● Reliable data replication across DC

36. uReplicator
Local
Agent
uReplicator

37. Multi-DC data flow

38. CONFIDENTIAL
>> INSERT SCREENSHOT HERE <<
Mirrormaker : existing problems
● New Topic added
● New partitions added
● Mirrormaker bounced
● New mirrormaker added

39. uReplicator: In-house solution
Zookeeper
Helix MM
Controller
Helix
Agent
Thread 1
Thread N
Topic-partition
Helix
Agent
Thread 1
Thread N
Topic-partition
Helix
Agent
Thread 1
Thread N
Topic-partition
MM worker1 MM worker2 MM worker3

40. uReplicator
Zookeeper
Helix MM
Controller
Helix
Agent
Thread 1
Thread N
Topic-partition
Helix
Agent
Thread 1
Thread N
Topic-partition
Helix
Agent
Thread 1
Thread N
Topic-partition
MM worker1 MM worker2 MM worker3

41. Kafka Clusters + Rest Proxy + Clients + Local Agent
● Scale to 100s Billions/day → 1 Trillion/day
● High Throughput ( Scale: 100s TB → PB)
● Low Latency for most use cases(<5ms )
● Reliability - 99.99% ( #Msgs Available /#Msgs Produced)
● Multi-Language Support
● Tens of thousands of simultaneous clients.
● Reliable data replication across DC

42. uReplicator
● Running in production for 1+ year
● Open sourced: https://github.com/uber/uReplicator
● Blog: https://eng.uber.com/ureplicator/

43. Chaperone - E2E Auditing

44. Chaperone Architecture

45. CONFIDENTIAL
>> INSERT SCREENSHOT HERE <<
Chaperone : Track counts

46. CONFIDENTIAL
>> INSERT SCREENSHOT HERE <<
Chaperone : Track Latency

47. Chaperone
● Running in production for 1+ year
● Planning to open source in ~2 Weeks

48. At-least Once Kafka

49. Why do we need it?
1
2
3 5 7
64 8
● Most of infrastructure tuned for high throughput
○ Batching at each stage
○ Ack before produce (ack’ed != committed)
● Single node failure in any stage leads to data loss
● Need a reliable pipeline for High Value Data e.g. Payments

50. How did we achieve it?
● Brokers:
○ min.insync.replicas=2, can only torrent one node failure
○ unclean.leader.election= false, need to wait until the old
leader comes back
● Rest Proxy:
○ Partition Failover
● Improved Operations:
○ Replication throttling, to reduce impact of node bootstrap
○ Prevent catching up nodes to become ISR

51. Operations/Tooling

52. Partition Rebalancing
Add
Figure

53. Partition Rebalancing
● Calculates partition
imbalance and inter-broker
dependency.
● Generates & Executes
Rebalance Plan.
● Rebalance plans are
incremental, can be stopped
and resumed.
● Currently on-demand,
Automated in the future.

54. XFS vs EXT4
Add
Figure

55. Summary: Scale
● Kafka Brokers:
○ Multiple Clusters per DC
○ Use case based tuning
● Rest Proxy to reduce connections and better batching
● Rest Proxy & Clients
○ Batch everywhere, Async produce
○ Replace Jersey with Jetty
● XFS

56. Summary: Reliability
● Local Agent
● Secondary Clusters
● Multi Producer support in Rest Proxy
● uReplicator
● Auditing via Chaperone

57. Future Work
● Open source contribution
○ Chaperone
○ Toolkit
● Data Lineage
● Active Active Kafka
● Chargeback
● Exactly once mirroring via uReplicator

58. Questions ?
ankur@uber.com

59. Extra Slides

60. Kafka Durability (acks=1)
Broker 1
100
101
102
103
Broker 2
100
101
Broker 3
100
101
Leader
Committed
Producer
Acked

61. Kafka Durability (acks=1)
Broker 1
100
101
102
103
Broker 2
100
101
Broker 3
100
101
Leader
Committed
Producer
Failed
Acked

62. Kafka Durability (acks=1)
Broker 1
100
101
102
103
Broker 2
100
101
Broker 3
100
101
Leader
Committed
Producer

63. Kafka Durability (acks=1)
Broker 1
100
101
102
103
Broker 2
100
101
104
105
106
Broker 3
100
101
104
105
Leader
Committed
Producer
Old HW

64. Kafka Durability (acks=1)
Broker 1
100
101
102
103
Broker 2
100
101
104
105
106
Broker 3
100
101
104
105
Leader
Committed
Producer
X
Old HW
X

65. Kafka Durability (acks=1)
Broker 1
100
101
104
105
106
Broker 2
100
101
104
105
106
Broker 3
100
101
105
106
Leader
Committed
Producer
data loss!!

66. Distributed Messaging system
* Supported in Kafka 0.8+
● High throughput
● Low latency
● Scalable
● Centralized
● Real-time

67. What is Kafka?
● Distributed
● Partitioned
● Replicated
● Commit Log
Broker 1 Broker 2 Broker 3
ZooKeeper

68. What is Kafka?
● Distributed
● Partitioned
● Replicated
● Commit Log
Broker 1
Partition 0
Broker 2
Partition 1
Broker 3
Partition 2
ZooKeeper

69. What is Kafka?
● Distributed
● Partitioned
● Replicated
● Commit Log
Broker 1
Partition 0
Partition 2
Broker 2
Partition 1
Partition 0
Broker 3
Partition 2
Partition 1
ZooKeeper

70. What is Kafka?
● Distributed
● Partitioned
● Replicated
● Commit Log
Broker 1
Partition 0
0 1 2 3
Partition 2
0 1 2 3
Broker 2
Partition 1
0 1 2 3
Partition 0
0 1 2 3
Broker 3
Partition 2
0 1 2 3
Partition 1
0 1 2 3
ZooKeeper

71. Kafka Concepts