true

1. CMB
An Open Source Alternative
for Queuing and Notification
Cloud Services
Ryan King
Comcast Silicon Valley
OpenStack Summit – San Diego 2012

3. Intro

4. Summary
A team inside Comcast open sourced a
compatible version of Amazon SNS and
SQS on top of Cassandra & Redis

5. Who are we?
• Comcast Silicon Valley
– Consumer-facing internet products for Comcast
– Web, Mobile, TV
• Silicon Valley
– Product Development & Innovation Center

6. CMB
CMB – Message Bus
CQS – Queue Service
CNS – Notification Service

7. But, wait.. Why build your own???

8. Comcast building Private Cloud
(… using OpenStack!)

9. To power next-gen services
(… including TV!)

10. Latency extremely important

11. Every millisecond counts

12. Needs to be cost effective
(… for 50MM users)

13. What else is out there?

14. None of those could meet
our requirements 

15. Requirements
• Compatible with AWS
• Support Active-Active Multi-Data-Center
• Horizontal Scalability
• Guaranteed Delivery
• Very Low Latency

16. NOT Required
1. Order is NOT guaranteed
2. Duplicate messages CAN happen
2. Duplicate messages CAN happen

17. First, the Queue
CMB – Message Bus
CQS – Queue Service
CNS – Notification Service

18. What is Simple Queue Service?
(from a Popular Cloud Service Provider)
“Simple Queue Service offers a reliable, highly
scalable, hosted queue for storing messages as
they travel between computers”

19. Simple Queue Service

20. Main SQS Methods
• CreateQueue
• SendMessage
• ReceiveMessage
• DeleteMessage

21. Our implementation:
CQS

22. Redis
Lists
CQS API Hashtables
CQS API
API
Payload cache
Cassandra

23. Cassandra Persistence
• Queue sharded over N=100 rows
–Avoids wide rows (> 500K items)
–Reduces “churn” (Tombstones)
–Distributes queue among nodes

24. Redis
Lists
CQS API Hashtables
CQS API
API
Payload cache

25. Redis Caching
• Message IDs: List
• Visibility: Hashtable
• Payload cache: Set

26. CQS Notes
• Minimal duplication and good ordering
– When things are happy..
• Cassandra tombstones degrade message
performance
– Normally reads => Redis

27. CQS Performance
90
80
70
Latency (ms)
60
50
40
30
20
10
0
10 50 75 100 250 500 1000
Msg/Sec

28. Why use Cassandra?
• very low latency
• very high write throughput
• extremely high availability (across multi-dc)
• Near linear horizontal scalability

29. Why add Redis?
• Queue metadata:
– changes frequently
– not durable
• Needed extremely low latency reads
• Reduces “churn” in Cassandra

30. Next, Notifications
CMB – Message Bus
CQS – Queue Service
CNS – Notification Service

31. What is Simple Notification Service?
(from a Popular Cloud Services Provider … in
Seattle… )
“Simple Notification Service is a web service that
makes it easy to set up, operate, and send
notifications from the cloud”

32. Simple Notification Service

33. Main SNS Methods
•CreateTopic
•Subscribe
•Publish

34. Our Implementation:
CNS

36. Scalability
• Publish Requests
=> API servers
• # subscribers
=> Delivery Producers
• Publishing to endpoints
=> Delivery Consumers
• Overall
=> CQS & Cassandra

37. Throughput Scalability
1200
1000
Latency (ms)
800
600
6 workers
400
3 workers
200
0
500 1000 2000 4000 8000
Throughput (Msgs/Sec)

38. Subscriber Scalability
3000
2500
Latency (ms)
2000
1500
3 workers
1000
6 workers
500
0
100 250 500 750 1000 1250 1500
# of Subscribers per Topic

39. What are we using this for?

41. Now Open Source!
github.com/Comcast/cmb

42. Next steps
• Does this fit into OpenStack?
• Get Feedback from the community (you!)
• Questions?
• ryan@sv.comcast.com
• @ryanking
• PS - We’re hiring @ Comcast SV!