Comcast is developing a highly scalable cloud DVR scheduling system on top of Cassandra. The system is responsible for managing all DVR data and scheduling logic for devices on the X1 platform. This talk will cover the overall architecture of the scheduling system, data model, message queue and notification software that have been developed as part of this ambitious project. We'll take a deep dive into the details of our data model and review the implementation of Comcast's open-source, Cassandra-based clones of Amazon SQS and SNS.

1. Using Cassandra for DVR Scheduling at Comcast
VERSION 1.0
DATE 3/20/213
AUTHOR John McCann | @dangermccann
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 1

2. The X1 Platform
XFINITY On Demand
Choose from thousands of movies and
shows.
More than a back button
See the last nine shows and channels
you've watched with a single tap.
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 2

3. The X1 Platform
The ultimate scoreboard
Track multiple games at once and check
the latest scores, standings, and schedules.
Apps right on your TV
Now your TV is for more than just
watching.
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 3

4. The X1 Platform
• Next generation of the XFINITY TV experience
• New line of set-top boxes and technologies that lets XFINITY
TV run in the cloud
• Premium content viewing experience, with access to XFINITY
On Demand, DVR and Pay-Per-View video
• De-coupled, service-based architecture where services are
developed and deployed independently
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 4

5. What Comcast Stores in Cassandra
• Customer viewing history
• Resume points for DVR and XFINITY On Demand content
• Device and account preferences
• Pay-Per-View purchases
• DVR recordings and scheduling instructions
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 5

6. Why Comcast Uses Cassandra
• Multi-data center (active / active)
• Performance of write-heavy use cases
– DVR status updates
– Remembering resume points during content playback
• Developers (not DBAs) model and manage the data
• Open source encourages in-house expertise
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 6

7. Redesigned DVR
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 7

8. DVR System Architecture
East Data Center West Data Center
API Server Worker API Server Worker
Worker Queue
Worker Queue
API Server Worker API Server Worker
API Server Worker API Server Worker
Cassandra Cassandra
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 8

9. Cassandra Configuration
• 4-node rings in 2 data centers
• NetworkTopologyStrategy replication strategy
• Replication factor is 3 (per data center)
• LOCAL_QUORUM consistency level for most operations
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 9

10. Comcast Message Bus
https://github.com/Comcast/cmb
CQS Endpoint
• Horizontally scalable queuing and
notification service
• Compatible with Amazon SQS and SNS Cassandra Redis
• Cassandra used to persist messages
• Redis used as caching layer
• Open source (Apache license)
CNS Endpoint
CNS Worker
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 10

11. Use Case Exploration
Scheduling a Series Recording
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 11

12. User Interaction
Set-top box
User
UI Server
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 12

13. User Interaction
Set-top box
API Server Worker
Worker Queue
User
API Server Worker
UI Server API Server Worker
Cassandra
Memcached
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 13

14. Recording Schedule Update
API Server Worker
Worker Queue
API Server Worker
API Server Worker
Cassandra
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 14

15. Recording Schedule Update
Set-top box
API Server Worker
Worker Queue
API Server Worker
API Server Worker
Cassandra
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 15

16. Recording Schedule Update
Set-top box
API Server Worker
Worker Queue
API Server Worker
API Server Worker
Cassandra
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 16

17. UI Update
API Server Worker
Worker Queue
API Server Worker
UI Server API Server Worker
Cassandra
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 17

18. UI Update
API Server Worker
Worker Queue
API Server Worker
UI Server API Server Worker
Cassandra
Memcached
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 18

19. System Diagram
Set-top box
API Server Worker
Worker Queue
User
API Server Worker
UI Server API Server Worker
Cassandra
Memcached
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 19

20. Data Model Analysis
1. Recording Instructions
2. Recording Schedule
3. Completed Recordings
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 20

21. Recording Instructions Schema
• Stores the instructions used to produce the recording schedule
in the order that the instructions were generated.
ScheduleInstruction
AccountID/DeviceID TimeUUID : [instruction] TimeUUID : [instruction] …
Composite row key Instructions sorted by creation time using TimeUUID as
the column name. The column value contains the instruction data.
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 21

22. Recording Schedule Schema
• Stores the data for the most recent recording schedule that
has been delivered to the Recorder on the STB.
DvrSchedule
AccountID/DeviceID
Properties of each recording stored as separate columns using
composite column names: recording_ID/[property_name]
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 22

23. Completed Recordings Schema
• Stores the data for all of the completed recordings on the STB.
Recording
AccountID/DeviceID
Properties of each recording stored as separate columns using
composite column names: recording_ID/[property_name]
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 23

24. Hardware Configuration
• HP DL360 G8
• 64GB RAM
• 2 x 600 GB SATA Hard Drive
• RAID 1 for OS Partition
• LSI controller
• 2 x Samsung 840 512GB SSD (room for up to 4 SSDs)
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 24

25. Solid State Drives
Samsung 840 512GB
• 50% cost increase per node
• 400% capacity increase per node
• Smaller rings, better read performance
• Read performance consistency (flat 99th percentiles for reads)
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 25

26. Lessons Learned
• If you're using Hector, pay close attention to
CassandraHostConfigurator.maxActive.
• Don’t enable the row cache if you need to perform slice
queries.
• Don’t delete a row if you plan to write columns to the same row
later (better in 1.2).
• Don’t run Cassandra on shared storage if you can avoid it.
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 26

27. Q&A
Thanks for your attention!
http://xfinity.comcast.net/x1
John McCann
@dangermccann
Using Cassandra for DVR Scheduling at Comcast 4/9/2013 27