Slides for the Elasticsearch Meetup in Netflix

1. Elasticsearch In Netflix
Danny Yuan, Jae Bae

2. Welcome
Hashtag: #ES_in_Netflix
@Elasticsearch - Elasticsearch
!
@stonse - Sudhir Tonse
!
@g9yuayon - Danny Yuan
!
@metacret - Jae Bae

3. Who Are We?

4. Who Are We?
Software engineers in Netflix’s
Platform Engineering team,
working on large scale data
infrastructure

5. Who Are We?
Software engineers in Netflix’s
Platform Engineering team,
working on large scale data
infrastructure
Building and operating Netflix’s
cloud real-time query service

6. Why Are We Here?

7. Why Are We Here?
How We Use Elasticsearch

8. Why Are We Here?
How We Use Elasticsearch
Why Elasticsearch

9. Why Are We Here?
How We Use Elasticsearch
Why Elasticsearch
How We Run Elasticsearch

10. Why Are We Here?
How We Use Elasticsearch
Why Elasticsearch
How We Run Elasticsearch
To Seek Your Feedback

11. How We Use Elasticsearch

12. Querying Log Events
Tracking Service Deployments

13. Querying Log Events

14. A Little Historical Perspective

15. Netflix is a log generating company
that also happens to stream movies
- Adrian Cockroft
photo credit: http://www.flickr.com/photos/decade_null/142235888/sizes/o/in/photostream/

18. A Humble Beginning

19. A Humble Beginning

20. A Humble Beginning

21. A Humble Beginning

22. Things Changed

25. Application
Application
Application
Application
Application
Application
Application
Application
Application
Application

26. 70,000,000,000

27. 1,500,000

28. Making Sense of Billions of Events

29. So We Evolved

30. So We Evolved

31. So We Evolved
hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket

32. So We Evolved
hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket

33. So We Evolved
hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket

34. So We Evolved
hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket

35. So We Evolved
hgrep -C 10 -k 5,2,3 'users.*[1-9]{3}' *catalina.out s3//bucket
select * from log_events where dateint=20140101

36. Field Name
Field Value
Client
“API”
Server
“Cryptex”
StatusCode
200
ResponseTime
73

37. Log data
Server Farm
Log data
Server Farm
Log Collectors
Log data
Server Farm

43. What Could Go Wrong?

45. You thought parallelization would save the day?
Think again

46. You thought parallelization would save the day?
Think again

47. What Is Missing?

48. Interactive Exploration

49. Functional Requirements
Arbitrary Boolean Queries
Aggregated Query
- Top N Query
- Trend
- Distribution

50. Non-Functional Requirements
- Interactive (response within seconds)
!
- Quickly locates the right log events
- Minimal programming effort

51. It’s All about Extracting Small Data
Out of Big Data

55. Now Back to the Use Case

56. Intelligent Alerts

57. Guided Debugging in the Right Context

58. Guided Debugging in the Right Context

59. Guided Debugging in the Right Context

60. Guided Debugging in the Right Context

61. Guided Debugging in the Right Context

62. Guided Debugging in the Right Context

63. Guided Debugging in the Right Context

64. Guided Debugging in the Right Context

65. A Useful Pattern

66. Aggregated Query -> Individual Query

67. Examples
- S3 diagnostics
!
- Tracking email campaigns
- Request traces

68. Status:200
RequestId Parent Id Node Id Service Name
Status
4965-4a74
0
123
Edge Service
200
4965-4a74
123
456
Gateway
200
4965-4a74
456
789
Service A
200
4965-4a74e
456
abc
Service B
200

70. Edge Service (456) ---> Gateway (789)
Data Name
Value
Request ID
4965-4a74
Response Time
25 ms
Endpoints
/rest/service
Status Code
200

71. Why Elasticsearch?

72. Automatic Sharding and Replication

74. Flexible Schema

75. Flexible Schema
- Schemaless

76. Flexible Schema
- Schemaless
- Reasonable defaults

78. Nice Extension Model

79. Nice Extension Model
- Customizable REST Actions

80. Nice Extension Model
- Customizable REST Actions
- Site Plugins

81. Nice Extension Model
- Customizable REST Actions
- Site Plugins
- River Plugins

82. Nice Extension Model
- Customizable REST Actions
- Site Plugins
- River Plugins
- Discovery Module

84. Ecosystem - Plugins, Kibana

85. Tracking Service Deployments

86. !
{ edda }

88. Built by Netflix Monitoring Eng Team

89. Built by Netflix Monitoring Eng Team
Tracks History and Changes to Service
Deployments

90. Built by Netflix Monitoring Eng Team
Tracks History and Changes to Service
Deployments
Keeps Many Revisions

91. Built by Netflix Monitoring Eng Team
Tracks History and Changes to Service
Deployments
Keeps Many Revisions
Tracks Dozens of Document Types

94. Why Elasticsearch?

96. Schemas may change at any time

97. Schemas may change at any time
Go schemaless

99. Users may search for any combination of fields

100. Users may search for any combination of fields
This is what search engine is designed for

102. Users often needs only a few fields

103. Users often needs only a few fields
Projection via “fields” query

105. Need range queries on date and revisions

106. Need range queries on date and revisions
Natively supported by Elasticsearch

107. Need range queries on date and revisions
Natively supported by Elasticsearch
Route by document ID

108. Running ES in Netflix

109. Operational Challenges

110. Operational Challenges
Back pressure when indexing

111. Operational Challenges
Back pressure when indexing
Diverse configurations and data

112. Operational Challenges
Back pressure when indexing
Diverse configurations and data
Dynamic flow of log events

113. Operational Challenges
Back pressure when indexing
Diverse configurations and data
Dynamic flow of log events
Needs extensive monitoring and alerting

114. Operational Challenges
Back pressure when indexing
Diverse configurations and data
Dynamic flow of log events
Needs extensive monitoring and alerting
Tolerating outage at different scales

115. Favor Pulling Over Pushing

118. Choose Config with Data

120. Integrating ES

121. AMI for Deployment by Asgard

122. Archaius for Configuration

123. Eureka for Server Discovery

124. Suro for Data Delivery

125. Servo for Monitoring Metrics

126. Zone-aware Replication

127. Multi-region Deployment

128. Multi-region Deployment
Discovery over Cassandra
Region-aware replication

129. Favor Index Rolling Over TTL

130. Favor Index Rolling Over TTL
A dedicated service manages index rolling
Uses index template and routing

131. Worth Trying G1

132. Worth Trying G1
Not recommended by ES team, but

133. Worth Trying G1
Not recommended by ES team, but
Has fewer and shorter GC pauses

134. Worth Trying G1
Not recommended by ES team, but
Has fewer and shorter GC pauses
Occasional SIGSEGV, but it’s okay

135. Simple Majority for Master Election

136. Simple Majority for Master Election
Split-brain problem

137. Simple Majority for Master Election
Split-brain problem
discovery.zen.minimum_master_nodes

138. Simple Majority for Master Election
Split-brain problem
discovery.zen.minimum_master_nodes
Dynamically updated

139. Future Work

140. Future Work
Automatic incremental backup and restore

141. Future Work
Automatic incremental backup and restore
Auto scaling

142. Future Work
Automatic incremental backup and restore
Auto scaling
Fully automated deployment

143. Future Work
Automatic incremental backup and restore
Auto scaling
Fully automated deployment
Support more use cases

144. We’re Hiring

145. Thank You!