Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1VmbI3t. Nitesh Kant describes how embracing asynchrony in the Netflix applications, from networking to business processing, creates gracefully degrading and highly resilient applications. Filmed at qconsf.com. Nitesh Kant is an engineer in Netflix’s Edge Gateway team, working on Netflix’s asynchronous Inter Process Communication stack. He is the author of RxNetty which forms the core of this stack and is currently moving Zuul to this new architecture.

1. Crossroads of asynchrony
and graceful degradation
Nitesh Kant,!
Software Engineer, !
Netflix Edge Engineering.!
!
@NiteshKant

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/netflix-asynchronous-apps

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

7. Nitesh Kant
Who Am I?
❖ Engineer, Edge Engineering, Netflix.!
❖ Core contributor, RxNetty*!
❖ Contributor, Zuul**
* https://github.com/ReactiveX/RxNetty
** https://github.com/Netflix/zuul
@NiteshKant

9. How do systems fail?

10. A simple example. Showing a movie on Netflix.

11. Video Metadata

12. Video Bookmark

13. Video Rating

14. !
public Movie getMovie(String movieId) {!
Metadata metadata = getMovieMetadata(movieId);!
Bookmark bookmark = getBookmark(movieId, userId);!
Rating rating = getRatings(movieId);!
return new Movie(metadata, bookmark, rating);!
}!
Disclaimer: This is an example and not an exact representation of the processing

15. Synchronicity
!
public Movie getMovie(String movieId) {!
Metadata metadata = getMovieMetadata(movieId);!
Bookmark bookmark = getBookmark(movieId, userId);!
Rating rating = getRatings(movieId);!
return new Movie(metadata, bookmark, rating);!
}!
Disclaimer: This is an example and not an exact representation of the processing

16. The bigger picture Price of being synchronous?
!
public Movie getMovie(String movieId) {!
Metadata metadata = getMovieMetadata(movieId);!
Bookmark bookmark = getBookmark(movieId, userId);!
Rating rating = getRatings(movieId);!
return new Movie(metadata, bookmark, rating);!
}!
Disclaimer: This is an example and not an exact representation of the processing

17. In a microservices world
Edge Service
Ratings ServiceVideo Metadata Service
Bookmarks Service
Disclaimer: This is an example and not an exact representation of the processing

18. In a microservices world
Edge Service
Server threadpool
Thread
Thread
Thread
Thread
Thread
getMovieMetadata(movieId)
Disclaimer: This is an example and not an exact representation of the processing

19. In a microservices world
Edge Service
Server threadpool
Thread
Thread
Thread
Thread
Thread
getMovieMetadata(movieId)
getBookmark(movieId, userId)
Disclaimer: This is an example and not an exact representation of the processing

20. In a microservices world
Edge Service
Server threadpool
Thread
Thread
Thread
Thread
Thread
getMovieMetadata(movieId)
getBookmark(movieId, userId)
getRatings(movieId)
Disclaimer: This is an example and not an exact representation of the processing

21. Busy thread time
=
Sum of the time taken to
make all 3 service calls

22. How do systems fail?
1. Latency Latency is your worst enemy in a
synchronous world.

23. Edge Service
Server threadpool
Thread
Thread
Thread
Thread
Thread
getRatings(movieId)
Disclaimer: This is an example and not an exact representation of the processing

24. Disclaimer: This is an example and not an exact representation of the processing
Ratings Service
Edge Service
Server threadpool
Thread
Thread
Thread
Thread
Thread
getRatings(movieId)

25. Disclaimer: This is an example and not an exact representation of the processing
Edge Service
getRatings(movieId)
Server threadpool
Thread
Thread
Thread
Thread
Thread

26. Edge Service
Disclaimer: This is an example and not an exact representation of the processing
getRatings(movieId)
Server threadpool
Thread
Thread
Thread
Thread
Thread

27. Edge Service
Server threadpool
Thread
Thread
Thread
Thread
Thread
Disclaimer: This is an example and not an exact representation of the processing
Client Threadpool
Thread
Thread
Thread
Thread
Thread
getRatings(movieId)

28. Edge Service
Server threadpool
Thread
Thread
Thread
Thread
Thread
Disclaimer: This is an example and not an exact representation of the processing
Client Threadpool
Thread
Thread
Thread
Thread
Thread
getRatings(movieId)

29. Edge Service
Server threadpool
Thread
Thread
Thread
Thread
Thread
Disclaimer: This is an example and not an exact representation of the processing
Client Threadpool
Thread
Thread
Thread
Thread
Thread
getRatings(movieId)

30. Managing client thread pools
Disclaimer: This is an example and not an exact representation of the processing
Client Threadpool
Thread
Thread
Thread
Thread
Thread

31. Managing client thread pools

32. Managing client thread pools

33. Managing client thread pools

34. Managing client thread pools

35. Clients have become our babies

36. Clients have become our babies
Edge Service
Server threadpool
Thread
Thread
Thread
Thread
Thread
getMovieMetadata(movieId)
Disclaimer: This is an example and not an exact representation of the processing
Client Threadpool
Thread
Thread
Thread
Thread
Thread
Client Threadpool
Thread
Thread
Thread
Thread
Thread
Client Threadpool
Thread
Thread
Thread
Thread
Thread
getBookmark(movieId, userId)
getRatings(movieId)

37. Clients have become our babies

38. Untuned/Wrongly tuned
clients cause many outages.

39. Have we exchanged a bigger problem with a
smaller one?

40. How do systems fail?
2. Overload Abusive clients, recovery spikes, special
events ….

41. We did a load test… https://github.com/Netflix-Skunkworks/
WSPerfLab
Hello Netflix!

42. Detailed analysis available online:
https://github.com/Netflix-Skunkworks/WSPerfLab/blob/master/test-results/RxNetty_vs_Tomcat_April2015.pdf

44. ! Graceful
This isn’t graceful degradation!

45. This happens at high CPU usage.

46. This happens at high CPU usage.
So, don’t let the system reach that limit…

47. This happens at high CPU usage.
So, don’t let the system reach that limit…
a.k.a Throttling.

48. Fairness?
One abusive request type can penalize other request
paths.

49. How do systems fail?
3. Thundering herds The failure after recovery….

50. Retries
Edge Service
Video Metadata Service
Disclaimer: This is an example and not an exact representation of the processing

51. Retries
Edge Service
Disclaimer: This is an example and not an exact representation of the processing
Video Metadata Service Cluster

52. Retries
Edge Service
Disclaimer: This is an example and not an exact representation of the processing
Video Metadata Service Cluster

53. Retries
Edge Service
Disclaimer: This is an example and not an exact representation of the processing
Video Metadata Service Cluster

54. Retries are useful in steady state….
…but…

55. Retries
Edge Service
Disclaimer: This is an example and not an exact representation of the processing
Video Metadata Service Cluster

57. Our systems are missing empathy.

58. Because they lack knowledge about the peers.

59. Knowledge comes from various signals..

60. Ability to adapt to those signals is important.

61. This can not adapt…
!
public Movie getMovie(String movieId) {!
Metadata metadata = getMovieMetadata(movieId);!
Bookmark bookmark = getBookmark(movieId, userId);!
Rating rating = getRatings(movieId);!
return new Movie(metadata, bookmark, rating);!
}!
Disclaimer: This is an example and not an exact representation of the processing

62. Asynchrony It is the key to success.

63. What should be async?

64. What should be async?
Edge Service
Video Metadata Service

65. What should be async?
Edge Service
Video Metadata Service
getMovieMetadata(movieId)
getBookmark(movieId, userId)
getRatings(movieId)
Application logic

66. What should be async?
Edge Service
Video Metadata Service
getMovieMetadata(movieId)
getBookmark(movieId, userId)
getRatings(movieId)
I/O
I/O
I/O
Application logic
I/O

67. What should be async?
Edge Service
Video Metadata Service
getMovieMetadata(movieId)
getBookmark(movieId, userId)
getRatings(movieId)
I/O
I/O
I/O
Application logic
I/O
Network protocol

68. Key aspects of being async.

69. Key aspects of being async.
1. Lifecycle control

70. Lifecycle control
Start processing Stop processing

71. Key aspects of being async.
2. Flow control

72. Flow control
When How much

73. Key aspects of being async.
3. Function composition

74. Function composition
!
public Movie getMovie(String movieId) {!
Metadata metadata = getMovieMetadata(movieId);!
Bookmark bookmark = getBookmark(movieId, userId);!
Rating rating = getRatings(movieId);!
return new Movie(metadata, bookmark, rating);!
}!

75. Function composition
Composing the processing !
of a method into a single control point.
public Observable<Movie> getMovie(String movieId) {!
return Observable.zip(getMovieMetadata(movieId),!
getBookmark(movieId, userId),!
getRatings(movieId),!
(meta,bmark,rating)->new Movie(meta,bmark,rating));!
}!

76. Composing the processing !
of a method into a single control point.
Flow & Lifecycle Control
with

77. What should be async?
Edge Service
Video Metadata Service
getMovieMetadata(movieId)
getBookmark(movieId, userId)
getRatings(movieId)
I/O
I/O
I/O
Application logic
I/O
Network protocol

78. I/O
Edge Service
Server threadpool
Thread
Thread
Thread
Thread
Thread
Client Threadpool
Thread
Thread
Thread
Thread
Thread
getRatings(movieId)

79. I/O
Edge Service
Disclaimer: This is an example and not an exact representation of the processing
Eventloop (Inbound)
Connection
Connection
Connection
Connection
Connection
Eventloops = f (Number of cores)

80. I/O
Edge Service
Disclaimer: This is an example and not an exact representation of the processing
Eventloop (Inbound)
Connection
Connection
Connection
Connection
Connection
Connections multiplexed !
on a !
single eventloop.

81. I/O
Disclaimer: This is an example and not an exact representation of the processing
Edge Service
Eventloop (Inbound)
Connection
Connection
Connection
Connection
Connection
getMovieMetadata(movieId)
Eventloop (Outbound)
Connection
Connection
Connection
Connection
Connection

82. I/O
Edge Service
Disclaimer: This is an example and not an exact representation of the processing
Eventloop (Inbound)
Connection
Connection
Connection
Connection
Connection
getMovieMetadata(movieId)
Eventloop (Outbound)
Connection
Connection
Connection
Connection
Connection
Clients share the eventloops!
with!
the server.

83. I/O
Edge Service
Disclaimer: This is an example and not an exact representation of the processing
Eventloop (Inbound)
Connection
Connection
Connection
Connection
Connection
getMovieMetadata(movieId)
Eventloop (Outbound)
Connection
Connection
Connection
Connection
Connection
All clients share !
the!
same eventloop

84. Composing the processing !
of a service into a single control point.
Flow & Lifecycle Control
with

85. What should be async?
Edge Service
Video Metadata Service
getMovieMetadata(movieId)
getBookmark(movieId, userId)
getRatings(movieId)
I/O
I/O
I/O
Application logic
I/O
Network protocol

86. Network Protocol
HTTP/1.1?

87. HTTP/1.1
GET /movie?id=1 HTTP/1.1

88. HTTP/1.1
GET /movie?id=2 HTTP/1.1
GET /movie?id=1 HTTP/1.1

89. HTTP/1.1
GET /movie?id=3 HTTP/1.1
GET /movie?id=2 HTTP/1.1
GET /movie?id=1 HTTP/1.1

90. HTTP/1.1
GET /movie?id=3 HTTP/1.1
GET /movie?id=2 HTTP/1.1
GET /movie?id=1 HTTP/1.1
HTTP/1.1 200 OK!
ID: 1!
…

91. HTTP/1.1
GET /movie?id=3 HTTP/1.1
GET /movie?id=2 HTTP/1.1
GET /movie?id=1 HTTP/1.1
HTTP/1.1 200 OK!
ID: 1!
… HTTP/1.1 200 OK!
ID: 2!
…

92. HTTP/1.1
GET /movie?id=3 HTTP/1.1
GET /movie?id=2 HTTP/1.1
GET /movie?id=1 HTTP/1.1
HTTP/1.1 200 OK!
ID: 1!
… HTTP/1.1 200 OK!
ID: 2!
…
HTTP/1.1 200 OK!
ID: 3!
…

93. HTTP/1.1
GET /movie?id=3
GET /movie?id=2
GET /movie?id=1
HTTP/1.1 200
ID: 1!
… HTTP/1.1 200
ID: 2!
…
HTTP/1.1 200
ID: 3!
…
Head Of Line Blocking => Synchronous

94. Network Protocol
HTTP/1.1?

95. Network Protocol
We need a multiplexed bi-directional
protocol

96. Multiplexed
GET /movie?id=1 HTTP/1.1
GET /movie?id=2 HTTP/1.1
GET /movie?id=3 HTTP/1.1

97. Bi-directional
GET /movie?id=1 HTTP/1.1
GET /movie?id=2 HTTP/1.1
GET /movie?id=3 HTTP/1.1
CANCEL

98. Composing the processing !
of the entire application into a single control point.
Flow & Lifecycle Control
with

99. Edge Service
Video Metadata Service
Rating service C* store
C* store
/movie?id=123
Disclaimer: This is an example and not an exact representation of the processing

100. Edge Service
Video Metadata Service
Rating service C* store
C* store
/movie?id=123
Disclaimer: This is an example and not an exact representation of the processing
Observable<Movie>

101. Observable<Movie>
Composing the processing !
of the entire application into a single control point.

102. Revisiting the failure modes

103. Latency
Edge Service
Disclaimer: This is an example and not an exact representation of the processing
Eventloop (Inbound)
Connection
Connection
Connection
Connection
Connection
getMovieMetadata(movieId)
Eventloop (Outbound)
Connection
Connection
Connection
Connection
Connection

104. Latency
Edge Service
Disclaimer: This is an example and not an exact representation of the processing
Eventloop (Inbound)
Connection
Connection
Connection
Connection
Connection
getMovieMetadata(movieId)
Eventloop (Outbound)
Connection
Connection
Connection
Connection
Connection
Impact is localized to the
connection.

105. Latency
Edge Service
Disclaimer: This is an example and not an exact representation of the processing
Eventloop (Inbound)
Connection
Connection
Connection
Connection
Connection
getMovieMetadata(movieId)
Eventloop (Outbound)
Connection
Connection
Connection
Connection
Connection
Impact is localized to the
connection.
An outstanding request has
little cost.

106. An outstanding request has little cost.
GET /movie?id=1 HTTP/1.1
HTTP/1.1 200 OK!
…
} Any stored state between!
request - response!
is costly.

107. Outstanding requests have low cost
so
Latency is a lesser evil in asynchronous systems.

108. Overload & Thundering Herds
Edge Service
Eventloop (Inbound)
Connection
Connection
Connection
Connection
Connection
getMovieMetadata(movieId)
Eventloop (Outbound)
Connection
Connection
Connection
Connection
Connection
Disclaimer: This is an example and not an exact representation of the processing
Reduce work done !
when overloaded

109. Overload & Thundering Herds
Edge Service
Eventloop (Inbound)
Connection
Connection
Connection
Connection
Connection
getMovieMetadata(movieId)
Eventloop (Outbound)
Connection
Connection
Connection
Connection
Connection
Disclaimer: This is an example and not an exact representation of the processing
Reduce work done !
when overloaded
Stop accepting!
new requests.

110. Stop accepting new requests
Non-blocking I/O gives better control

111. Stop accepting new requests
But … we are still “throttling”

112. Stop accepting new requests
Are we being empathetic?

113. Request-leasing
http://reactivesocket.io/

114. Request-leasing
Peer 1Peer 2
Network connection

115. Request-leasing
Peer 1Peer 2
“Lease” 5 requests for 1 minute.
Network connection

116. Request-leasing
Peer 1Peer 2
“Lease” 5 requests for 1 minute.
GET /movie?id=1 HTTP/1.1
GET /movie?id=2 HTTP/1.1
Network connection

117. Server
Client 1 Client 2 Client 8

118. ServerCapacity: 100 RPM
Client 1 Client 2 Client 8
“Lease” 10 requests for 1 minute.

119. ServerCapacity: 100 RPM
Client 1 Client 2 Client 8
“Lease” 10 requests for 1 minute.
“Lease” 10 requests for 1 minute.
“Lease” 10 requests for 1 minute.

120. ServerCapacity: 100 RPM
Client 1 Client 2 Client 8
“Lease” 10 requests for 1 minute.
“Lease” 10 requests for 1 minute.
“Lease” 10 requests for 1 minute.
Reserve Capacity: !
20 RPM

121. Time bound lease.
“Lease” 10 requests for 1 minute.

122. Time bound lease.
No extra work for cancelling leases.
“Lease” 10 requests for 1 minute.

123. Time bound lease.
No extra work for cancelling leases.
Receiver controls the flow of requests
“Lease” 10 requests for 1 minute.

124. When things go south

125. ServerCapacity: 20 RPM
Client 1 Client 2 Client 8
“Lease” 5 requests for 1 minute.
“Lease” 2 requests for 1 minute.
X No more “Lease”

126. ServerCapacity: 20 RPM
Client 1 Client 2 Client 8
“Lease” 5 requests for 1 minute.
“Lease” 2 requests for 1 minute.
X No more “Lease”
Prioritization

127. Managing client configs?

128. Threadpools?
Edge Service
Disclaimer: This is an example and not an exact representation of the processing
Eventloop (Inbound)
Connection
Connection
Connection
Connection
Connection
getMovieMetadata(movieId)
Eventloop (Outbound)
Connection
Connection
Connection
Connection
Connection
I/O!
is!
non-blocking.

129. Threadpools?
Edge Service
Disclaimer: This is an example and not an exact representation of the processing
Eventloop (Inbound)
Connection
Connection
Connection
Connection
Connection
getMovieMetadata(movieId)
Eventloop (Outbound)
Connection
Connection
Connection
Connection
Connection
Application code !
is!
non-blocking.

130. Threadpools?
Disclaimer: This is an example and not an exact representation of the processing
Edge Service
Eventloop (Inbound)
Connection
Connection
Connection
Connection
Connection
getMovieMetadata(movieId)
Eventloop (Outbound)
Connection
Connection
Connection
Connection
Connection
No blocking/Waiting => Only CPU work

131. Threadpools?
Disclaimer: This is an example and not an exact representation of the processing
Edge Service
Eventloop (Inbound)
Connection
Connection
Connection
Connection
Connection
getMovieMetadata(movieId)
Eventloop (Outbound)
Connection
Connection
Connection
Connection
Connection
No blocking/Waiting => Only CPU work
So,
Eventloops = # of cores

132. Tuning parameters?

133. Tuning parameters?
X

134. Tuning parameters?
X
X

135. Tuning parameters?
X
X?
?

136. Case for timeouts?

137. Case for timeouts?
Read Timeouts Thread Timeouts
✤ Useful in unblocking threads
on socket reads.
✤ Business level SLA.
✤ Unblock the calling thread.

138. Case for timeouts?
Read Timeouts Thread Timeouts
✤ Useful in unblocking threads
on socket reads.
✤ Business level SLA.
✤ Unblock the calling thread.
X X
As there are no blocking calls.X

139. Case for timeouts?
Read Timeouts Thread Timeouts
✤ Useful in unblocking threads
on socket reads.
✤ Business level SLA.
✤ Unblock the calling thread.

140. Business level SLA
Edge Service
Video Metadata Service
Disclaimer: This is an example and not an exact representation of the processing

141. Business level SLA
Edge Service
Video Metadata Service
Disclaimer: This is an example and not an exact representation of the processing
Rating service C* store
C* store

142. Business level SLA
Edge Service
Video Metadata Service
Disclaimer: This is an example and not an exact representation of the processing
Rating service C* store
C* store

143. Business level SLA
Edge Service
Video Metadata Service
Disclaimer: This is an example and not an exact representation of the processing
Rating service C* store
C* store
Thread timeouts are pretty invasive at every level

144. Business level SLA
Edge Service
Video Metadata Service
Disclaimer: This is an example and not an exact representation of the processing
Rating service C* store
C* store
Thread timeouts are pretty invasive at every level
Do we need them at every step?

145. Edge Service
Video Metadata Service
Rating service C* store
C* store
/movie?id=123
Disclaimer: This is an example and not an exact representation of the processing

146. Edge Service
Video Metadata Service
Rating service C* store
C* store
/movie?id=123
Business timeouts are !
for !
a client request.
Disclaimer: This is an example and not an exact representation of the processing

147. Edge Service
Video Metadata Service
Rating service C* store
C* store
/movie?id=123
Disclaimer: This is an example and not an exact representation of the processing

148. Edge Service
Video Metadata Service
Rating service C* store
C* store
/movie?id=123
X
Disclaimer: This is an example and not an exact representation of the processing

149. Edge Service
Video Metadata Service
Rating service C* store
C* store
/movie?id=123
X
X
X
X
X
Disclaimer: This is an example and not an exact representation of the processing

150. Tuning parameters?
X
X
X
X

151. Less tuning

153. Edge Service
Video Metadata Service
Rating service C* store
C* store
Disclaimer: This is an example and not an exact representation of the processing
Request Leases
Cancellations
Observable<Movie>

154. Edge Service
Video Metadata Service
Rating service C* store
C* store
Disclaimer: This is an example and not an exact representation of the processing
Request Leases
Cancellations
Observable<Movie>

156. !
public Movie getMovie(String movieId) {!
Metadata metadata = getMovieMetadata(movieId);!
Bookmark bookmark = getBookmark(movieId, userId);!
Rating rating = getRatings(movieId);!
return new Movie(metadata, bookmark, rating);!
}!
public Observable<Movie> getMovie(String movieId) {!
return Observable.zip(getMovieMetadata(movieId),!
getBookmark(movieId, userId),!
getRatings(movieId),!
(meta,bmark,rating)->new
Movie(meta,bmark,rating));!
}!

157. Resources
Asynchronous Function composition :
I/O :
Network Protocol :
https://github.com/ReactiveX/RxJava
https://github.com/ReactiveX/RxNetty
http://reactivesocket.io/

158. Nitesh Kant, Engineer, Netflix Edge Gateway @NiteshKant

159. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/netflix-
asynchronous-apps