by Ben Fleis - Nodeday Amsterdam Feb 11, 2016

1. EXPLORATIONS IN COOPERATIVE
DISTRIBUTED SYSTEMS
WITH UBER’S RINGPOP

2. WHAT IS RINGPOP?
HIGH-LEVEL FACTS
It is a library written in Node.js and Go.
It is...
“...scalable, fault-tolerant application layer sharding.”
"...a library that brings cooperation and coordination to distributed applications.”
“...a hash ring.”
It is open source.

3. WHAT IS RINGPOP?
… A LITTLE MAGIC?
Lightweight: Write your service. Add Ringpop. Stir. Serve.
Need more instances? Add more instances.
Deploy a cluster. It will mostly take care of itself.
No leaders or referees; just workers.
Stands on the shoulders of giants.

4. WHAT IS UBER?
REALTIME DISPATCH ENGINEERING
The Monolith past: rigid, manual, limited scale.
The SOA future:
Platform for growth: 100x scale
Self healing
Automatic instance integration
Application neutral

5. RINGPOP @UBER
FAST FACTS
>25 realtime services
core + infrastructural
instances in the thousands
QPS in the millions

6. ENTER RINGPOP
REQUEST ROUTING
CONSISTENT HASHING
MEMBERSHIP PROTOCOL

7. h1 h2 h3
DATABASE DATABASE
DISCRETE APPLICATION INSTANCES
ENTER RINGPOP: MEMBERSHIP PROTOCOL
FE1 FE2 FE3 FE4 FE5
FRONT-END
APPLICATION
STORAGE

8. COOPERATIVE APPLICATION INSTANCES
ENTER RINGPOP: MEMBERSHIP PROTOCOL
h1
h2
h3
DATABASE DATABASE
FE1 FE2 FE3 FE4 FE5
FRONT-END
APPLICATION
STORAGE

9. REQUEST ROUTING
RINGPOP

10. REQUEST ROUTING
APPLICATION LAYER MIDDLEWARE
HTTP / THRIFT / ETC
ROUTING
HASH RING
BUSINESS LOGIC
MEMBERSHIP
h1
{RINGPOP
PROCESS
APPLICATION
FRONT-END
STORAGE

11. CONSISTENT HASHING
RINGPOP

12. Hash Ring:
[0, 232)
Example:
[0-100)
Ringpop - Consistent Hashing
0
50
2575
99

13. Hashing instances
Ringpop - Consistent Hashing
hash(“h1”)
hash(“h2”)hash(“h3”)

14. Ringpop - Consistent Hashing
hash(“h1”)Ownership: instances
Hash Ring slices for
“h1”, “h2”, and “h3”
hash(“h2”)
hash(“h3”)

15. Ownership: requests
“Alice”,
“Bob”, and
“Charlie”
Ringpop - Consistent Hashing
hash(“h1”)
“Alice” ∈ (“h1”, “h2”] → h2
“Bob” → h3
hash(“h2”)
“Charlie” → h3
hash(“h3”)

16. Ringpop - Consistent Hashing
hash(“h1”)Losing capacity:
“h3” is faulty
“Alice” → h2
“Bob” → h1
hash(“h2”)
“Charlie” → h1
hash(“h3”)

17. Ringpop - Consistent Hashing
hash(“h1”)Original cluster of 3
“Alice” → h2
“Bob” → h3
hash(“h2”)
“Charlie” → h3
hash(“h3”)

18. Ringpop - Consistent Hashing
hash(“h1”)Adding capacity:
“h4” joins
“Alice” → h2
“Bob” → h4
hash(“h2”)
hash(“h3”)
hash(“h4”)
“Charlie” → h3

19. MEMBERSHIP PROTOCOL
RINGPOP

20. SWIM GOSSIP PROTOCOL
ENTER RINGPOP: MEMBERSHIP PROTOCOL
PING
h1 h2
h1
h2
h3
h1
h2
h3
h3
h1
h2
h3
PING PING
Membership

21. ENTER RINGPOP: MEMBERSHIP PROTOCOL
PING
h1 h2
h1
h2
h3
h1
h2
h3
h3
h1
h2
h3
FAILURE
DETECTION

22. ENTER RINGPOP: MEMBERSHIP PROTOCOL
h1 h2
h1
h2
h3
h1
h2
h3
h3
h1
h2
h3
PING-REQ PING
AN INDIRECT PING

23. ENTER RINGPOP: MEMBERSHIP PROTOCOL
h1 h2
h1
h2
h3
h1
h2
h3
h3
h1
h2
h3
PING-REQ
DECLARE h2 SUSPECT

24. ENTER RINGPOP: MEMBERSHIP PROTOCOL
h1 h2
h1
h2
h3
h1
h2
h3
h3
h1
h2
h3
PING
h2
PIGGYBACK MEMBERSHIP UPDATES

25. ENTER RINGPOP: MEMBERSHIP PROTOCOL
h1 h2
h1
h2
h3
h1
h2
h3
h3
h1
h2
h3
PING
h2
INFECTION-STYLE DISSEMINATION

26. ENTER RINGPOP: MEMBERSHIP PROTOCOL
h1 h2
h1
h2
h3
h1
h2
h3
h3
h1
h2
h3
PING
DETECTING A FAILURE

27. PROGRAMMING RINGPOP
INSTANTIATE, BOOTSTRAP, LOOKUP

28. A TYPICAL WEB APP
PROGRAMMING RINGPOP

29. INSTANTIATING RINGPOP
PROGRAMMING RINGPOP

30. BOOTSTRAPPING RINGPOP
PROGRAMMING RINGPOP

31. RING LOOKUPS
PROGRAMMING RINGPOP

32. APPLICATIONS
RINGPOP

33. LONG LIVED WEBSOCKETS
APPLICATION OF RINGPOP
h1
h2 h3
Alarm service
Chat service
websockets server pushes
Ringpop
forwarding

34. APPLICATIONS OF RINGPOP
RINGPOP IN PRODUCTION
Geospatial sharding
Work delegation
Server-side push / long-polling
Caching
Aggregation
Mailboxes
Database
Service Discovery and Routing

35. LESSONS LEARNED
RINGPOP

36. Verifying correctness.
Scaling and Failing.
DEVELOPMENT AND STAGING
LESSONS LEARNED

37. Convergence
Cross-pollination
Flappy nodes
Hard to forget
Slow start times
Anti-entropy
Tooling
Backwards compatibility
PRODUCTION
LESSONS LEARNED

38. DynamoDB
by Amazon
Riak
by Basho
Serf
by Hashicorp
Cassandra
by Apache
Orleans
by Microsoft
Akka
by Typesafe

39. CONCLUSION
RINGPOP

40. Thank You!
Ben Fleis
benfleis@uber.com
Uber is hiring.
Come work with me in our Amsterdam office!