Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1yyaHb8. The authors discuss Netflix's new stream processing system that supports a reactive programming model, allows auto scaling, and is capable of processing millions of messages per second. Filmed at qconsf.com. Danny Yuan is an architect and software developer in Netflix’s Platform Engineering team. Justin Becker is Senior Software Engineer at Netflix.

1. Mantis: Netflix's Event
Stream Processing System
Justin Becker Danny Yuan
10/26/2014

2. Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/mantis
InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month

3. Presented at QCon San Francisco
www.qconsf.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. Motivation

5. Traditional TV just works

6. Netflix wants Internet TV to work just as well

8. Especially when things aren’t working

10. Tracking “big signals” is not
enough

11. Need to track all kinds of permutations

12. Detect quickly, to resolve ASAP

13. Cheaper than product services

14. Here comes the problem

15. 12 ~ 20 million metrics updates per
second

16. 750 billion metrics updates per day

19. 4 ~ 18 Million App Events Per Second
> 300 Billion Events Per Day

28. They Solve Specific Problems

29. They Require Lots of Domain Knowledge

30. A System to Rule Them All

33. Mantis: A Stream Processing System

35. 3

36. 1. Versatile User Demands

42. Asynchronous Computation Everywhere

48. 3. Same Source, Multiple Consumers

52. We want Internet TV to just work

53. One problem we need to solve,
detect movies that are failing?

54. Do it fast → limit impact, fix early
Do it at scale → for all permutations
Do it cheap → cost detect <<< serve

55. Work through the details for how
to solve this problem in Mantis

56. Goal is to highlight unique and
interesting design features

57. … begin with batch approach,
the non-Mantis approach
Batch algorithm, runs every N minutes
for(play in playAttempts()){
Stats movieStats = getStats(play.movieId);
updateStats(movieStats, play);
if (movieStats.failRatio > THRESHOLD){
alert(movieId, failRatio, timestamp);
}
}

58. First problem, each run requires
reads + writes to data store per run
Batch algorithm, runs every N minutes
for(play in playAttempts()){
Stats movieStats = getStats(play.movieId);
updateStats(movieStats, play);
if (movieStats.failRatio > THRESHOLD){
alert(movieId, failRatio, timestamp);
}
}

59. For Mantis don’t want to pay that
cost: for latency or storage
Batch algorithm, runs every N minutes
for(play in playAttempts()){
Stats movieStats = getStats(play.movieId);
updateStats(movieStats, play);
if (movieStats.failRatio > THRESHOLD){
alert(movieId, failRatio, timestamp);
}
}

60. Next problem, “pull” model great for
batch processing, bit awkward for
stream processing
Batch algorithm, runs every N minutes
for(play in playAttempts()){
Stats movieStats = getStats(play.movieId);
updateStats(movieStats, play);
if (movieStats.failRatio > THRESHOLD){
alert(movieId, failRatio, timestamp);
}
}

61. By definition, batch processing
requires batches. How do I chunk
my data? Or, how often do I run?
Batch algorithm, runs every N minutes
for(play in playAttempts()){
Stats movieStats = getStats(play.movieId);
updateStats(movieStats, play);
if (movieStats.failRatio > THRESHOLD){
alert(movieId, failRatio, timestamp);
}
}

62. For Mantis, prefer “push” model,
natural approach to data-in-motion
processing
Batch algorithm, runs every N minutes
for(play in playAttempts()){
Stats movieStats = getStats(play.movieId);
updateStats(movieStats, play);
if (movieStats.failRatio > THRESHOLD){
alert(movieId, failRatio, timestamp);
}
}

63. For our “push” API we decided
to use Reactive Extensions (Rx)

64. Two reasons for choosing Rx:
theoretical, practical
1. Observable is a natural abstraction for
stream processing, Observable = stream
2. Rx already leveraged throughout the
company

65. So, what is an Observable?
A sequence of events, aka a stream
Observable<String> o =
Observable.just(“hello”,
“qcon”, “SF”);
o.subscribe(x->{println x;})

66. What can I do with an Observable?
Apply operators → New observable
Subscribe → Observer of data
Operators, familiar lambda functions
map(), flatMap(), scan(), ...

67. What is the connection with Mantis?
In Mantis, a job (code-to-run) is the
collection of operators applied to a
sourced observable where the
output is sinked to observers

68. Think of a “source” observable as
the input to a job.
Think of a “sink” observer as the
output of the job.

69. Let’s refactor previous problem
using Mantis API terminology
Source: Play attempts
Operators: Detection logic
Sink: Alerting service

70. Sounds OK, but how will this scale?
For pull model luxury of requesting
data at specified rates/time
Analogous to drinking
from a straw

71. In contrast, push is the firehose
No explicit control to limit the
flow of data

72. In Mantis, we solve this problem
by scaling horizontally

73. Horizontal scale is accomplished
by arranging operators into
logical “stages”, explicitly by job
writer or implicitly with fancy
tooling (future work)

74. A stage is a boundary between
computations. The boundary
may be a network boundary,
process boundary, etc.

75. So, to scale, Mantis job is really,
Source → input observable
Stage(s) → operators
Sink → output observer

76. Let’s refactor previous problem to
follow the Mantis job API structure
MantisJob
.source(Netflix.PlayAttempts())
.stage({ // detection logic })
.sink(Alerting.email())

77. We need to provide a computation
boundary to scale horizontally
MantisJob
.source(Netflix.PlayAttempts())
.stage({ // detection logic })
.sink(Alerting.email())

78. For our problem, scale is a function
of the number of movies tracking
MantisJob
.source(Netflix.PlayAttempts())
.stage({ // detection logic })
.sink(Alerting.email())

79. Lets create two stages, one producing
groups of movies, other to run detection
MantisJob
.source(Netflix.PlayAttempts())
.stage({ // groupBy movieId })
.stage({ // detection logic })
.sink(Alerting.email())

80. OK, computation logic is split, how is the
code scheduled to resources for execution?
MantisJob
.source(Netflix.PlayAttempts())
.stage({ // groupBy movieId })
.stage({ // detection logic })
.sink(Alerting.email())

81. One, you tell the Mantis Scheduler explicitly
at submit time: number of instances, CPU
cores, memory, disk, network, per instance

82. Two, Mantis Scheduler learns how to
schedule job (work in progress)
Looks at previous run history
Looks at history for source input
Over/under provision, auto adjust

83. A scheduled job creates a topology
Stage 1 Stage 2 Stage 3
Worker
Worker Worker
Worker
Worker
Worker
Worker
Application 1
Mantis cluster
Application
clusters
Application 2
Application 3
cluster boundary

84. Computation is split, code is scheduled,
how is data transmitted over stage
boundary?
Worker Worker ?

85. Depends on the service level agreement
(SLA) for the Job, transport is pluggable
Worker Worker ?

86. Decision is usually a trade-off
between latency and fault tolerance
Worker Worker ?

87. A “weak” SLA job might trade-off fault
tolerance for speed, using TCP as the
transport
Worker Worker ?

88. A “strong” SLA job might trade-off speed
for fault tolerance, using a queue/broker as
a transport
Worker Worker ?

89. Forks and joins require data partitioning,
collecting over boundaries
Worker
Worker
Worker
Fork Worker
Worker
Worker
Join
Need to partition data Need to collect data

90. Mantis has native support for partitioning,
collecting over scalars (T) and groups (K,V)
Worker
Worker
Worker
Fork Worker
Worker
Worker
Join
Need to partition data Need to collect data

91. Let’s refactor job to include SLA, for the
detection use case we prefer low latency
MantisJob
.source(Netflix.PlayAttempts())
.stage({ // groupBy movieId })
.stage({ // detection logic })
.sink(Alerting.email())
.config(SLA.weak())

92. The job is scheduled and running what
happens when the input-data’s volume
changes?
Previous scheduling decision may not hold
Prefer not to over provision, goal is for cost
insights <<< product

93. Good news, Mantis Scheduler has the
ability to grow and shrink (autoscale) the
cluster and jobs

94. The cluster can scale up/down for two
reasons: more/less job (demand) or jobs
themselves are growing/shrinking
For cluster we can use submit pending
queue depth as a proxy for demand
For jobs we use backpressure as a proxy to
grow shrink the job

95. Backpressure is “build up” in a system
Imagine we have a two stage Mantis job,
Second stage is performing a complex
calculation, causing data to queue up in the
previous stage
We can use this signal to increase nodes at
second stage

96. Having touched on key points in Mantis
architecture, want to show a complete job
definition

97. Source
Stage 1
Stage 2
Sink

98. Play Attempts
Grouping by
movie Id
Detection
algorithm
Email alert

99. Sourcing play attempts
Static set of
sources

100. Grouping by movie Id
GroupBy operator
returns key
selector function

101. Simple detection algorithms
Windows for 10 minutes
or 1000 play events
Reduce the window to
an experiment, update
counts
Filter out if less than
threshold

102. Sink alerts

103. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/mantis