Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1bJsakO.
Ben Christensen describes how the Netflix API evolved from a typical one-size-fits-all RESTful API designed to support public developers into a web service platform optimized to handle the diversity and variability of each device and user experience. The talk will also address the challenges involving operations, deployment, performance, fault-tolerance, and rate of innovation at massive scale. Filmed at qconsf.com.
Ben Christensen is a software engineer on the Netflix Edge Services Platform team responsible for fault tolerance, performance, architecture and scale while enabling millions of customers to access the Netflix experience across more than 1,000 different device types. Prior to Netflix, Ben was at Apple making iOS apps and media available on the iTunes store.
1. Evolution of the Netflix API
QCon San Francisco - November 2013
Ben Christensen
Software Engineer – Edge & Playback Services at Netflix
@benjchristensen
!
!
!
!
http://techblog.netflix.com/
2. Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/netflix-api-evolution
InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
3. Presented at QCon San Francisco
www.qconsf.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
5. More than 40 million Subscribers
in 50+ Countries and Territories
6. Netflix accounts for 33% of Peak Downstream
Internet Traffic in North America
Netflix subscribers are watching
more than 1 billion hours a month
7. API traffic has grown from
~20 million/day in 2010 to >2 billion/day
millions of API requests per day
2000
1500
1000
500
0
2010
2011
2012
Today
21. Netflix API
Dependency A
Dependency B
Dependency D
Dependency C
Dependency E
Dependency G
Dependency F
Dependency H
Dependency J
Dependency M
Dependency P
Dependency I
Dependency K
Dependency N
Dependency Q
Dependency L
Dependency O
Dependency R
22. Netflix API
Dependency A
Dependency B
Dependency D
Dependency C
Dependency E
Dependency G
Dependency F
Dependency H
Dependency J
Dependency M
Dependency P
Dependency I
Dependency K
Dependency N
Dependency Q
Dependency L
Dependency O
Dependency R
32. User Request
Dependency A
Dependency B
Dependency D
Dependency C
Dependency E
Dependency G
Dependency F
Dependency H
Dependency J
Dependency M
Dependency P
Dependency I
Dependency K
Dependency N
Dependency Q
Dependency L
Dependency O
Dependency R
33. User Request
Dependency A
Dependency B
Dependency D
Dependency C
Dependency E
Dependency G
Dependency F
Dependency H
Dependency J
Dependency M
Dependency P
Dependency I
Dependency K
Dependency N
Dependency Q
Dependency L
Dependency O
Dependency R
User request
blocked by
latency in
single
network call
34. Dependency A
Dependency B
Dependency D
Dependency C
Dependency E
Dependency G
Dependency F
Dependency H
Dependency J
Dependency M
Dependency P
Dependency I
Dependency K
Dependency N
Dependency Q
Dependency L
Dependency O
Dependency R
User Request
User Request
User Request
User Request
User Request
User Request
User Request
.............................................
At high
volume
all request
threads can
block in
seconds
35. !
Dozens of dependencies.
!
One going bad takes everything down.
!
30
99.99%
= 99.7% uptime
!
0.3% of 1 billion = 3,000,000 failures
!
2+ hours downtime/month
!
!
!
Reality is generally worse.
40. Dependency A
Dependency B
Dependency D
Dependency C
Dependency E
Dependency G
Dependency F
Dependency H
Dependency J
Dependency M
Dependency P
Dependency I
Dependency K
Dependency N
Dependency Q
Dependency L
Dependency O
Dependency R
User Request
User Request
User Request
User Request
User Request
User Request
User Request
.............................................
41. User Request
User Request
User Request
Logic - argument validation, caches, metrics, logging,
multivariate testing, routing, etc
Serialization - URL and/or body generation
Network Request - TCP/HTTP, latency, 4xx, 5xx, etc
.............................................
Deserialization - JSON/XML/Thrift/Protobuf/etc
Dependency B
cy D
Dependency C
Dependency E
dency G
ependency J
Dependency M
Logic - validation, decoration, object model, caching,
metrics, logging, etc
Dependency F
Dependency H
Dependency K
Dependency N
Dependency I
Dependency L
Dependency O
42. > 80% of requests rejected
Median
Latency
[Sat Jun 30 04:01:37 2012] [error] proxy: HTTP: disabled connection for (127.0.0.1)
"Timeout guard" daemon prio=10 tid=0x00002aaacd5e5000 nid=0x3aac runnable [0x00002aaac388f000] java.lang.Thread.State: RUNNABLE!
at java.net.PlainSocketImpl.socketConnect(Native Method)!
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)!
- locked <0x000000055c7e8bd8> (a java.net.SocksSocketImpl)!
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)!
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)!
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)!
at java.net.Socket.connect(Socket.java:579)!
at java.net.Socket.connect(Socket.java:528)!
at java.net.Socket.(Socket.java:425)!
at java.net.Socket.(Socket.java:280)!
at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)!
at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$1.doit(ControllerThreadSocketFactory.java:91)!
at org.apache.commons.httpclient.protocol.ControllerThreadSocketFactory$SocketTask.run(ControllerThreadSocketFactory.java:158) at java.lang.Thread.run(Thread.java:722)
43. User Request
Dependency A
Dependency B
Dependency D
Dependency C
Dependency E
Dependency G
Dependency F
Dependency H
Dependency J
Dependency M
Dependency P
Dependency I
Dependency K
Dependency N
Dependency Q
Dependency L
Dependency O
Dependency R
44. User Request
Dependency A
Dependency B
Dependency D
Dependency C
Dependency E
Dependency G
Dependency F
Dependency H
Dependency J
Dependency M
Dependency P
Dependency I
Dependency K
Dependency N
Dependency Q
Dependency L
Dependency O
Dependency R
45. User Request
Dependency A
Dependency B
Dependency D
Dependency C
Dependency E
Dependency G
Dependency F
Dependency H
Dependency J
Dependency M
Dependency P
Dependency I
Dependency K
Dependency N
Dependency Q
Dependency L
Dependency O
Dependency R
46. User Request
Dependency A
Dependency B
Dependency D
Dependency C
Dependency E
Dependency G
Dependency F
Dependency H
Dependency J
Dependency M
Dependency P
Dependency I
Dependency K
Dependency N
Dependency Q
Dependency L
Dependency O
Dependency R
51. circle color and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
52. circle color and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
hosts reporting from cluster
53. circle color and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
hosts reporting from cluster
last minute latency percentiles
54. circle color and size represent
health and traffic volume
2 minutes of request rate to
show relative changes in traffic
hosts reporting from cluster
Circuit-breaker
status
last minute latency percentiles
55. circle color and size represent
health and traffic volume
Request rate
2 minutes of request rate to
show relative changes in traffic
hosts reporting from cluster
Circuit-breaker
status
last minute latency percentiles
56. circle color and size represent
health and traffic volume
Error percentage of
last 10 seconds
Request rate
2 minutes of request rate to
show relative changes in traffic
hosts reporting from cluster
Circuit-breaker
status
last minute latency percentiles
57. Error percentage of
last 10 seconds
circle color and size represent
health and traffic volume
Request rate
2 minutes of request rate to
show relative changes in traffic
Circuit-breaker
status
hosts reporting from cluster
last minute latency percentiles
Rolling 10 second counters
with 1 second granularity
Successes
Short-circuited (rejected)
Thread timeouts
Thread-pool Rejections
Failures/Exceptions
58. User Request
Dependency A
Dependency B
Dependency D
Dependency C
Dependency E
Dependency G
Dependency F
Dependency H
Dependency J
Dependency M
Dependency P
Dependency I
Dependency K
Dependency N
Dependency Q
Dependency L
Dependency O
Dependency R
65. User Request
Dependency A
Dependency B
Dependency D
Dependency C
Dependency E
Dependency G
Dependency F
Dependency H
Dependency J
Dependency M
Dependency P
Dependency I
Dependency K
Dependency N
Dependency Q
Dependency L
Dependency O
Dependency R
System
Relationship
Over
Network
without
Bulkhead
86. Netflix API
Dependency A
Dependency B
Dependency D
Dependency C
Dependency E
Dependency G
Dependency F
Dependency H
Dependency J
Dependency M
Dependency P
Dependency I
Dependency K
Dependency N
Dependency Q
Dependency L
Dependency O
Dependency R
87. Netflix API
Dependency A
Dependency B
Dependency D
Dependency C
Dependency E
Dependency G
Dependency F
Dependency H
Dependency J
Dependency M
Dependency P
Dependency I
Dependency K
Dependency N
Dependency Q
Dependency L
Dependency O
Dependency R
88. One Size Fits All RESTful API
millions of API requests per day
2000
1500
1000
500
0
2010
2011
2012
Today
89. One Size Fits All RESTful API
millions of API requests per day
2000
1500
1000
500
0
2010
2011
2012
Today
90. One Size Fits All RESTful API
1000+ Devices
millions of API requests per day
2000
1500
1000
500
0
2010
2011
2012
Today
91. We wanted to re-architecture our call patterns ...
92. ... to collapse network traffic into coarse API calls ...
nested, conditional, concurrent execution
93. ... and we wanted to allow
anybody to create
endpoints, not just the
“API Team”
95. Concurrency without each engineer
reading and re-reading this →
!
(awesome book ... everybody isn’t going to - or
should have to - read it though, that’s the point)
96. Owner of api should retain control
of concurrency behavior.
97. Owner of api should retain control
of concurrency behavior.
public
Data
getData();
What if the implementation needs to change
from synchronous to asynchronous?
!
How should the client execute that method
without blocking? spawn a thread?
98. public
Data
getData();
public
void
getData(Callback<T>
c);
!
public
Future<T>
getData();
!
public
Future<List<Future<T>>>
getData();
!
!
other options ... ?
99. Reactive Programming with Rx
Iterable
pull
T next()
throws Exception
returns;
Observable
push
onNext(T)
onError(Exception)
onCompleted()
100. (Functional) Reactive Programming with RxJava
Iterable
pull
T next()
throws Exception
returns;
Observable
push
onNext(T)
onError(Exception)
onCompleted()
101. (Functional) Reactive Programming with RxJava
Iterable
pull
T next()
throws Exception
returns;
Observable
push
onNext(T)
onError(Exception)
onCompleted()
102. (Functional) Reactive Programming with RxJava
Iterable
pull
T next()
throws Exception
returns;
Observable
push
onNext(T)
onError(Exception)
onCompleted()
103. Iterable
pull
T next()
throws Exception
returns;
!
//
Iterable<String>
//
that
contains
75
Strings
getDataFromLocalMemory()
.skip(10)
.take(5)
.map({
s
-‐>
return
s
+
"_transformed"})
.forEach(
{
println
"next
=>
"
+
it})
Observable
push
onNext(T)
onError(Exception)
onCompleted()
!
//
Observable<String>
//
that
emits
75
Strings
getDataFromNetwork()
.skip(10)
.take(5)
.map({
s
-‐>
return
s
+
"_transformed"})
.subscribe(
{
println
"onNext
=>
"
+
it})
104. Iterable
pull
T next()
throws Exception
returns;
!
//
Iterable<String>
//
that
contains
75
Strings
getDataFromLocalMemory()
.skip(10)
.take(5)
.map({
s
-‐>
return
s
+
"_transformed"})
.forEach(
{
println
"onNext
=>
"
+
it})
Observable
push
onNext(T)
onError(Exception)
onCompleted()
!
//
Observable<String>
//
that
emits
75
Strings
getDataFromNetwork()
.skip(10)
.take(5)
.map({
s
-‐>
return
s
+
"_transformed"})
.subscribe(
{
println
"onNext
=>
"
+
it})
121. instead of a blocking api ...
class
VideoService
{
def
VideoList
getPersonalizedListOfMovies(userId);
def
VideoBookmark
getBookmark(userId,
videoId);
def
VideoRating
getRating(userId,
videoId);
def
VideoMetadata
getMetadata(videoId);
}
... create an observable api:
class
VideoService
{
def
Observable<VideoList>
getPersonalizedListOfMovies(userId);
def
Observable<VideoBookmark>
getBookmark(userId,
videoId);
def
Observable<VideoRating>
getRating(userId,
videoId);
def
Observable<VideoMetadata>
getMetadata(videoId);
}
122. RxJava
http://github.com/Netflix/RxJava
“a library for composing
asynchronous and event-based
programs using observable
sequences for the Java VM”
A Java port of Rx (Reactive Extensions)
https://rx.codeplex.com (.Net and Javascript by Microsoft)
123. client code treats all interactions
with the api as asynchronous
!
!
the api implementation chooses
whether something is
blocking or non-blocking
and
what resources it uses
142. public class HelloEndpoint extends APIEndpoint {
@Override
public void execute(APIRequest api) throws Throwable {
// get a request parameter from servlet request
String alias = api.servletRequest.getParameter("name");
!
// set content type and write something to servlet response
api.servletResponse.setContentType("application/json");
api.servletResponse.writer.print(
JsonUtility.toJson(
[ alias : alias, name : api.user.firstName ]
)
)
!
}
}
143. public class HelloEndpoint extends APIEndpoint {
@Override
public void execute(APIRequest api) throws Throwable {
// get a request parameter from servlet request
String alias = api.servletRequest.getParameter("name");
!
// set content type and write something to servlet response
api.servletResponse.setContentType("application/json");
api.servletResponse.writer.print(
JsonUtility.toJson(
[ alias : alias, name : api.user.firstName ]
)
)
!
}
}
runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
144. public class HelloEndpoint extends APIEndpoint {
@Override
public void execute(APIRequest api) throws Throwable {
// get a request parameter from servlet request
String alias = api.servletRequest.getParameter("name");
!
// set content type and write something to servlet response
api.servletResponse.setContentType("application/json");
api.servletResponse.writer.print(
JsonUtility.toJson(
[ alias : alias, name : api.user.firstName ]
)
)
!
}
}
runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
{"alias":"bob","name":"Old"}
145. public class HelloEndpoint extends APIEndpoint {
@Override
public void execute(APIRequest api) throws Throwable {
// get a request parameter from servlet request
String alias = api.servletRequest.getParameter("name");
!
// set content type and write something to servlet response
api.servletResponse.setContentType("application/json");
api.servletResponse.writer.print(
JsonUtility.toJson(
[ alias : alias, name : api.user.firstName ]
)
)
!
}
}
runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
{"alias":"bob","name":"Old"}
146. public class HelloEndpoint extends APIEndpoint {
@Override
public void execute(APIRequest api) throws Throwable {
// get a request parameter from servlet request
String alias = api.servletRequest.getParameter("name");
!
// set content type and write something to servlet response
api.servletResponse.setContentType("application/json");
api.servletResponse.writer.print(
JsonUtility.toJson(
[ alias : alias, name : api.user.firstName ]
)
)
!
}
}
runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
{"alias":"bob","name":"Old"}
147. public class HelloEndpoint extends APIEndpoint {
@Override
public void execute(APIRequest api) throws Throwable {
// get a request parameter from servlet request
String alias = api.servletRequest.getParameter("name");
!
// set content type and write something to servlet response
api.servletResponse.setContentType("application/json");
api.servletResponse.writer.print(
JsonUtility.toJson(
[ alias : alias, name : api.user.firstName ]
)
)
!
}
}
runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
{"alias":"bob","name":"Old"}
148. public class HelloEndpoint extends APIEndpoint {
@Override
public void execute(APIRequest api) throws Throwable {
// get a request parameter from servlet request
String alias = api.servletRequest.getParameter("name");
!
// set content type and write something to servlet response
api.servletResponse.setContentType("application/json");
api.servletResponse.writer.print(
JsonUtility.toJson(
[ alias : alias, name : api.user.firstName ]
)
)
!
}
}
runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
{"alias":"bob","name":"Old"}
149. public class HelloEndpoint extends APIEndpoint {
@Override
public void execute(APIRequest api) throws Throwable {
// get a request parameter from servlet request
String alias = api.servletRequest.getParameter("name");
!
// set content type and write something to servlet response
api.servletResponse.setContentType("application/json");
api.servletResponse.writer.print(
JsonUtility.toJson(
[ alias : alias, name : api.user.firstName ]
)
)
!
}
}
runScript.py -e TEST --userId 1189658154 --args "name=bob" HelloEndpoint.groovy
{"alias":"bob","name":"Old"}
150. public Observable getVideoSummary(APIVideo video) {
// get id, title
def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)]
!
// get bookmark
def bookmarkObservable = getBookmark(video)
!
// get artwork
def artworkObservable = getArtworkImageUrl(video)
!
// merge them and accumulate into the seed.
return Observable.merge(bookmarkObservable, artworkObservable)
.reduce(seed, { aggregate, current -> aggregate << current})
.map({ [ (video.id.toString()) : it]})
}
151. public Observable getVideoSummary(APIVideo video) {
// get id, title
def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)]
!
// get bookmark
def bookmarkObservable = getBookmark(video)
!
// get artwork
def artworkObservable = getArtworkImageUrl(video)
!
// merge them and accumulate into the seed.
return Observable.merge(bookmarkObservable, artworkObservable)
.reduce(seed, { aggregate, current -> aggregate << current})
.map({ [ (video.id.toString()) : it]})
}
152. public Observable getVideoSummary(APIVideo video) {
// get id, title
def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)]
!
// get bookmark
def bookmarkObservable = getBookmark(video)
!
// get artwork
def artworkObservable = getArtworkImageUrl(video)
!
// merge them and accumulate into the seed.
return Observable.merge(bookmarkObservable, artworkObservable)
.reduce(seed, { aggregate, current -> aggregate << current})
.map({ [ (video.id.toString()) : it]})
}
153. public Observable getVideoSummary(APIVideo video) {
// get id, title
def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)]
!
// get bookmark
def bookmarkObservable = getBookmark(video)
!
// get artwork
def artworkObservable = getArtworkImageUrl(video)
!
// merge them and accumulate into the seed.
return Observable.merge(bookmarkObservable, artworkObservable)
.reduce(seed, { aggregate, current -> aggregate << current})
.map({ [ (video.id.toString()) : it]})
}
154. public Observable getVideoSummary(APIVideo video) {
// get id, title
def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)]
!
// get bookmark
def bookmarkObservable = getBookmark(video)
!
// get artwork
def artworkObservable = getArtworkImageUrl(video)
!
// merge them and accumulate into the seed.
return Observable.merge(bookmarkObservable, artworkObservable)
.reduce(seed, { aggregate, current -> aggregate << current})
.map({ [ (video.id.toString()) : it]})
}
155. public Observable getVideoSummary(APIVideo video) {
// get id, title
def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)]
!
// get bookmark
def bookmarkObservable = getBookmark(video)
!
// get artwork
def artworkObservable = getArtworkImageUrl(video)
!
// merge them and accumulate into the seed.
return Observable.merge(bookmarkObservable, artworkObservable)
.reduce(seed, { aggregate, current -> aggregate << current})
.map({ [ (video.id.toString()) : it]})
}
156. public Observable getVideoSummary(APIVideo video) {
// get id, title
def seed = [id: video.id, title : video.getTitle(APIVideo.TitleType.REGULAR)]
!
// get bookmark
def bookmarkObservable = getBookmark(video)
!
// get artwork
def artworkObservable = getArtworkImageUrl(video)
!
// merge them and accumulate into the seed.
return Observable.merge(bookmarkObservable, artworkObservable)
.reduce(seed, { aggregate, current -> aggregate << current})
.map({ [ (video.id.toString()) : it]})
}
157. $ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov 11 05:38:51 UTC 2013",
"revision": 1,
"userAuthorizationRequired": true,
"userAuthorizationType": "https"
}
158. $ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov 11 05:38:51 UTC 2013",
"revision": 1,
"userAuthorizationRequired": true,
"userAuthorizationType": "https"
}
159. $ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov 11 05:40:26 UTC 2013",
"revision": 2,
"userAuthorizationRequired": true,
"userAuthorizationType": "https"
}
160. $ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov 11 05:40:26 UTC 2013",
"revision": 2,
"userAuthorizationRequired": true,
"userAuthorizationType": "https"
}
$ activateScript.py -e TEST --revision 2 /test/hello
{
"active": true,
"allocationPercentage": 100,
"creationDate": "Mon Nov 11 05:40:26 UTC 2013",
"previousRevision": null,
"revision": 2,
"userAuthorizationRequired": true,
"userAuthorizationType": "https"
}
161. $ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov 11 05:40:26 UTC 2013",
"revision": 2,
"userAuthorizationRequired": true,
"userAuthorizationType": "https"
}
$ activateScript.py -e TEST --revision 2 /test/hello
{
"active": true,
"allocationPercentage": 100,
"creationDate": "Mon Nov 11 05:40:26 UTC 2013",
"previousRevision": null,
"revision": 2,
"userAuthorizationRequired": true,
"userAuthorizationType": "https"
}
162. $ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov 11 05:40:26 UTC 2013",
"revision": 2,
"userAuthorizationRequired": true,
"userAuthorizationType": "https"
}
$ activateScript.py -e TEST --revision 2 /test/hello
{
"active": true,
"allocationPercentage": 100,
"creationDate": "Mon Nov 11 05:40:26 UTC 2013",
"previousRevision": null,
"revision": 2,
"userAuthorizationRequired": true,
"userAuthorizationType": "https"
}
163. $ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov 11 05:40:26 UTC 2013",
"revision": 2,
"userAuthorizationRequired": true,
"userAuthorizationType": "https"
}
$ activateScript.py -e TEST --revision 2 /test/hello
{
"active": true,
"allocationPercentage": 100,
"creationDate": "Mon Nov 11 05:40:26 UTC 2013",
"previousRevision": null,
"revision": 2,
"userAuthorizationRequired": true,
"userAuthorizationType": "https"
}
164. $ uploadScript.py -e TEST /test/hello sample.groovy
{
"active": false,
"allocationPercentage": 0,
"creationDate": "Mon Nov 11 05:40:26 UTC 2013",
"revision": 2,
"userAuthorizationRequired": true,
"userAuthorizationType": "https"
}
$ activateScript.py -e TEST --revision 3 /test/hello
{
"active": true,
"allocationPercentage": 100,
"creationDate": "Mon Nov 11 05:42:05 UTC 2013",
"previousRevision": 2,
"revision": 3,
"userAuthorizationRequired": true,
"userAuthorizationType": "https"
}
166. millions of API requests per day
2000
1500
1000
500
0
2008
2009
2010
2011
2012
Today
174. /tv/home
/android/home
/ps3/home
Functional Reactive Dynamic Endpoints
Asynchronous Java API
Dependency A
10 Threads
Dependency F
10 Threads
Dependency K
15 Threads
Dependency P
10 Threads
Dependency B
8 Threads
Dependency G
10 Threads
Dependency L
4 Threads
Dependency Q
8 Threads
Dependency C
10 Threads
Dependency H
10 Threads
Dependency M
5 Threads
Dependency R
10 Threads
Dependency D
15 Threads
Dependency I
5 Threads
Dependency N
10 Threads
Dependency S
8 Threads
Dependency E
5 Threads
Dependency J
8 Threads
Dependency O
10 Threads
Dependency T
10 Threads
175. /tv/home
/android/home
/ps3/home
Hystrix
Functional Reactive Dynamic Endpoints
fault-isolation layer
Asynchronous Java API
Dependency A
10 Threads
Dependency F
10 Threads
Dependency K
15 Threads
Dependency P
10 Threads
Dependency B
8 Threads
Dependency G
10 Threads
Dependency L
4 Threads
Dependency Q
8 Threads
Dependency C
10 Threads
Dependency H
10 Threads
Dependency M
5 Threads
Dependency R
10 Threads
Dependency D
15 Threads
Dependency I
5 Threads
Dependency N
10 Threads
Dependency S
8 Threads
Dependency E
5 Threads
Dependency J
8 Threads
Dependency O
10 Threads
Dependency T
10 Threads
176. +
Observable<User>
u
=
new
GetUserCommand(id).observe();
Observable<Geo>
g
=
new
GetGeoCommand(request).observe();
!
Observable.zip(u,
g,
{user,
geo
-‐>
return
[username:
user.getUsername(),
currentLocation:
geo.getCounty()]
})
RxJava in Hystrix 1.3+
https://github.com/Netflix/Hystrix
179. Ben Christensen
@benjchristensen
http://www.linkedin.com/in/benjchristensen
jobs.netflix.com
!
Functional Reactive in the Netflix API with RxJava
http://techblog.netflix.com/2013/02/rxjava-netflix-api.html
!
Optimizing the Netflix API
http://techblog.netflix.com/2013/01/optimizing-netflix-api.html
!
Application Resilience in a Service-oriented Architecture
http://programming.oreilly.com/2013/06/application-resilience-in-a-service-oriented-architecture.html
!
Fault Tolerance in a High Volume, Distributed System
http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html
!
Making the Netflix API More Resilient
http://techblog.netflix.com/2011/12/making-netflix-api-more-resilient.html
180. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/netflixapi-evolution