Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to @Indeedeng: RAD - How We Replicate Terabytes of Data Around the World Every Day
Similar to @Indeedeng: RAD - How We Replicate Terabytes of Data Around the World Every Day (20)
More from indeedeng
More from indeedeng (20)
Recently uploaded
Recently uploaded (20)
@Indeedeng: RAD - How We Replicate Terabytes of Data Around the World Every Day
- 1. RAD How We Replicate Terabytes of Data Around the World Every Day
- 3. Indeed is the #1 external source of hire 64% of US job searchers search on indeed each month Unique Visitors (millions) Million unique visitors 2009 2011 2012 2013 2014 2015 0 20 40 60 80 100 120 140 160 180 2010 180M 180 million unique users 80.2M unique US visitors per month 16M jobs 50+ countries 28 languages
- 5. How We Build Systems fast simple resilient scalable
- 6. fast
- 7. Fast
- 9. Job Search Browser Rendering median ~0.5 seconds Feb 24 Feb 25 Feb 26 Feb 27 Feb 28 Feb 29 Mar 1 Mar 2 Mar 3 Mar 4 Mar 5 Mar 6 Mar 7 Mar 8 0 100 200 300 400 500 600 700 800 milliseconds
- 10. simple
- 11. 2004 launch: a few servers, 1.8m US jobs
- 12. 2004 Aggregation MySQL Job Search Every job on the web
- 13. relational database, accessed across the network
- 14. NOT fast at full text search NOT a search engine
- 16. LuceneTM a high-performance, full featured text search engine library
- 17. LuceneTM NOT a remote database, files must be on local disk
- 18. MySQL Database Server Lucene Index Server Index Builder /data/jobindex
- 19. Index Builder Index Builder Index Builder Index Builder /data/jobindex /data/jobindex /data/jobindex /data/jobindex MySQL
- 20. MySQL Database Server Indexer Server Index Builder /data/jobindex Search Engine /data/jobindex 4 Search Servers
- 21. any combination of data, not just lucene
- 22. lucene + model
- 26. MySQL Database Server Index Builder Producer Artifact Artifact Consumers Search Engine
- 27. MySQL Database Server Index Builder Producer Artifact Artifact Consumers Search Engine Artifact is read-optimized data stored in a directory on the file system
- 28. Producer creates and updates a data artifact Database Server Index Builder Producer Artifact Artifact Consumers Search Engine MySQL
- 29. Consumer reads a data artifact Database Server Index Builder Producer Artifact Artifact Consumers Search Engine MySQL
- 30. produce once, consume many times
- 31. MySQL Database Server Index Builder Producer Artifact Artifact Consumers Search Engine Benefit: minimize database access
- 32. MySQL Database Server Index Builder Producer Artifact Artifact Consumers Search Engine Benefit: compute artifact once
- 33. MySQL Database Server Index Builder Producer Artifact Artifact Consumers Search Engine Benefit: scale consumers independently
- 34. MySQL Expensive Index Builder Producer Artifact Artifact Commodity Search Engine Benefit: scale consumers independently
- 35. MySQL Database Server Index Builder Producer Artifact Artifact Consumers Search Engine Benefit: separate code deployables
- 39. rsync efficient point-to-point file transfer utility
- 41. 1 consumers should reload data regularly
- 42. 1 consumers should reload data regularly 2 roll back
- 43. consumers should reload data regularly 2 roll back 3 data reload should not interrupt requests 1
- 45. $ ls -d jobindex.* jobindex.1 jobindex.2 jobindex.3 new directory for new version
- 46. $ ls -d jobindex.* jobindex.1 jobindex.2 jobindex.3 jobindex.latest -> jobindex.3 symlink to know current version
- 47. $ ls -d jobindex.* jobindex.1 jobindex.2 jobindex.3 jobindex.4 jobindex.latest -> jobindex.4 load new data
- 48. $ ls -d jobindex.* jobindex.1 jobindex.2 jobindex.3 jobindex.4 jobindex.latest -> jobindex.3 roll back
- 49. each new version takes disk space & time
- 50. versions total bytes on disk normal disk copy
- 51. versions disk latency total bytes on disk normal disk copy
- 52. versions version create time disk latency total bytes on disk normal disk copy
- 53. 1.8m jobs, change <2% per hour
- 55. all jobs 00:00 AM all jobs 04:00 AM new jobs changed jobs
- 56. all jobs 00:00 AM all jobs 04:00 AM new jobs changed jobs unchanged
- 58. save disk space & time
- 59. share data between versions
- 62. file1.bin file2.bin file3.bin 3GB jobindex.1 file1.bin file2.bin file3.bin file4.bin 4GB jobindex.2
- 66. file1.bin file2.bin file3.bin file4.bin jobindex.2 file1.bin file2.bin file3.bin file5.bin jobindex.3 deleted 1GB 1GB = 5GB+ 2GB file4.bin
- 67. remove referenced file of symlink, data is gone
- 68. hardlink additional name for an existing file
- 73. remove last hardlink, data is gone
- 74. artifact versions: symlinks + hardlinks + rsync
- 76. scale: single producer, many consumers
- 77. Job Search Browser Rendering median ~0.5 seconds Feb 24 Feb 25 Feb 26 Feb 27 Feb 28 Feb 29 Mar 1 Mar 2 Mar 3 Mar 4 Mar 5 Mar 6 Mar 7 Mar 8 0 100 200 300 400 500 600 700 800 milliseconds
- 78. fast simple resilient scalable How We Build Systems
- 81. 2004 2008 200920062005 22.5 M5.2 M 7.1 M4.0 M1.8 M jobs added or modified each month
- 85. 2011 52 countries 4 datacenters 2004 Indeed 1999 Lucene 2008 6 countries 2009 23 countries
- 86. 2004 2008 200920062005 22.5 M5.2 M 7.1 M4.0 M1.8 M jobs added or modified each month 2011 32.5 M
- 87. rsync system growing pains
- 88. Simple: serially copy one artifact at a time DC1 Producer Artifacts DC2 Staging Artifacts
- 89. Problem: serially can cause delays Producer Staging New New New Old DC1 DC2
- 90. smalllarge2large1 smalllarge2large1 Workaround: copy separately in “streams” DC1 DC2 Staging Producer
- 91. Simple: point-to-point datacenter rsync paths DC4 DC3 DC2 DC1
- 92. Problem: Internet, why did you do that? Down DC4 DC3 DC2 DC1
- 93. Workaround: shift replication path DC4 DC3 DC2 DC1
- 94. Scale: few consumers with rsync Producer Artifacts Consumers
- 95. Consumers Producer Grow: many consumers with rsync Artifacts Consumers
- 96. Consumers Producer Problem: too many consumers with rsync Artifacts Consumers network 100% used
- 97. Workaround: add more network bandwidth Consumers Producer Artifacts Consumers
- 98. Workaround: add staging tiers Consumers Producer Artifacts Staging Artifacts Artifacts Staging Artifacts Staging Artifacts Consumers Consumers Consumers Consumers Consumers Consumers Consumers Staging
- 99. rsync growth required sysad intervention
- 100. 2011 52 countries 2004 Indeed 1999 Lucene 2008 6 countries 2009 23 countries 2014 rsync growth
- 101. 100 artifacts, adding +1 producer each month
- 102. producing 1,761 TB per month
- 103. over 200 consumers, +2 each month
- 104. replicating over 21,931 TB per month
- 105. staging tiers or network bandwidth, quarterly
- 106. modify replication path, monthly
- 107. requiring too much intervention from system administrators
- 109. 2011 52 countries 2004 Indeed 1999 Lucene 2008 6 countries 2009 23 countries 2014 rsync limits
- 112. Jobsearch backend team produces a lot of data
- 114. Design GoalsDesign Goals Minimize network bottlenecks Loose coupling Automatic recovery Developer empowerment System-wide visibility 1 2 3 4 5
- 115. Design Goals Minimize network bottlenecks Loose coupling Automatic recovery Developer empowerment System-wide visibility 3 4 5 1 2
- 116. Design Goals Minimize network bottlenecks Loose coupling Automatic recovery Developer empowerment System-wide visibility 1 2 5 4 3
- 117. Design Goals Minimize network bottlenecks Loose coupling Automatic recovery Developer empowerment System-wide visibility 1 2 3 5 4
- 118. Design Goals Minimize network bottlenecks Loose coupling Automatic recovery Developer empowerment System-wide visibility 1 2 3 4 5
- 119. Design GoalsDesign Goals Minimize network bottlenecks Loose coupling Automatic recovery Developer empowerment System-wide visibility 1 2 3 4 5
- 122. Measure time and network traffic Bittorrent: Would it work? Sample replication to 3 consumers https://github.com/shevek/ttorrent
- 123. Network Test Total MB received + transmitted for 700MB artifact Producer 2,240 Consumer 1 746 Consumer 2 747 Consumer 3 747 machine RSYNC
- 124. Network Test Total MB received + transmitted for 700MB artifact Producer 2,240 782 Consumer 1 746 1,226 Consumer 2 747 1,225 Consumer 3 747 1,245 machine BITTORRENTRSYNC
- 125. Network Test Total MB received + transmitted for 700MB artifact Producer 2,240 782 Consumer 1 746 1,226 Consumer 2 747 1,225 Consumer 3 747 1,245 Total 4,481 4,480 machine BITTORRENTRSYNC
- 127. How does bittorrent work?
- 128. Data split into small pieces of equal size
- 129. Hash computed for each piece
- 131. File3.bin (50MB) File1.bin (100MB) File2.bin (200MB) jobindex.1 Piece 1: 75 MB Piece 2: 75 MB Piece 3: 75 MB Piece 4: 75 MB Piece 5: 25 MB
- 133. { files:file1.bin,100MB; file2.bin,200MB; file3.bin,50MB } { piecelength:75MB } { infohash:XSDJSK;JDISJLD;DJKJDB;KDJB OP;FJEIODK; } .torrent metadata file:
- 134. { files:file1.bin,100MB; file2.bin,200MB; file3.bin,50MB } { piecelength:75MB } { infohash:XSDJSK;JDISJLD;DJKJDB;KDJB OP;FJEIODK; } .torrent metadata file:
- 135. { files:file1.bin,100MB; file2.bin,200MB; file3.bin,50MB } { piecelength:75MB } { infohash:XSDJSK;JDISJLD;DJKJDB;KDJB OP;FJEIODK; } .torrent metadata file:
- 136. Tracker Coordinator of the download
- 137. Seeder Any client providing data
- 138. Seeder Data I have pieces for info hash Tracker .torrent Info Hash File manifest
- 139. Data .torrent Info Hash File manifest Seeder Tracker Info hash peer Map Ok! I have pieces for info hash
- 140. Consumer Any client downloading data
- 141. Peers for infohash Consumer Tracker .torrent Info Hash File manifest Tracker URL Map Info hash peer How a consumer gets the first piece
- 142. Peers for infohash Peerlist Consumer Tracker .torrent Info Hash File manifest Tracker URL Map Info hash peer How a consumer gets the first piece
- 143. Data .torrent Info Hash File manifest Consumer/ Seeder I have pieces for infohash Tracker Info hash peer Map It is also a seeder
- 144. Consumer 1 Seeding as it downloads Consumer 2 Seeding as it downloads Consumer 3 Seeding as it downloads Seeder SWARM
- 145. Didn’t quite meet our needs
- 146. Piece 1: HASH1 Piece 2: HASH2 Piece 3: HASH3 Piece 4: HASH4 Piece 5: HASH5 File3.bin (50MB) File1.bin (100MB) File2.bin (200MB) jobindex.1
- 148. jobindex.2 File4.bin (50MB) File3.bin (50MB) File1.bin (100MB) File2.bin (200MB) Piece 1: HASH1 Piece 2: HASH2 Piece 3: HASH3 Piece 4: HASH4 Piece 5: HASH6 Piece 1: HASH1 Piece 2: HASH2 Piece 3: HASH3 Piece 4: HASH4 Piece 5: HASH5 Piece 6: HASH7 File3.bin (50MB) File1.bin (100MB) File2.bin (200MB) jobindex.1
- 149. jobindex.2 File4.bin (50MB) File3.bin (50MB) File1.bin (100MB) File2.bin (200MB) Piece 1: HASH1 Piece 2: HASH2 Piece 3: HASH3 Piece 4: HASH4 Piece 5: HASH6 Piece 1: HASH1 Piece 2: HASH2 Piece 3: HASH3 Piece 4: HASH4 Piece 5: HASH5 Piece 6: HASH7 File3.bin (50MB) File1.bin (100MB) File2.bin (200MB) jobindex.1
- 151. Piece 1: HASH6 Piece 2: HASH7 Piece 3: HASH8 Piece 4: HASH9 Piece 5: HASH10 Piece 1: HASH1 Piece 2: HASH2 Piece 3: HASH3 Piece 4: HASH4 Piece 5: HASH5 Piece 6: HASH11 File3.bin (50MB) File1.bin (100MB) File2.bin (200MB) jobindex.1 jobindex.2 File4.bin (50MB) File3.bin (50MB) File1.bin (100MB) File2.bin (200MB) jobindex.2 File0.bin (50MB) File3.bin (50MB) File1.bin (100MB) File2.bin (200MB)
- 152. Control sort order?
- 153. jobindex.2 File3.bin (50MB) File1.bin (150MB) File2.bin (200MB) Piece 1: HASH6 Piece 2: HASH7 Piece 3: HASH8 Piece 4: HASH9 Piece 5: HASH10 Piece 1: HASH1 Piece 2: HASH2 Piece 3: HASH3 Piece 4: HASH4 Piece 5: HASH5 Piece 6: HASH11 File3.bin (50MB) File1.bin (100MB) File2.bin (200MB) jobindex.1
- 154. File3.bin (50MB) File1.bin (100MB) File2.bin (200MB) jobindex.1 Piece 1: HASH6 Piece 2: HASH7 Piece 3: HASH8 Piece 4: HASH9 Piece 5: HASH10 Piece 1: HASH1 Piece 2: HASH2 Piece 3: HASH3 Piece 4: HASH4 Piece 5: HASH5 Piece 6: HASH11 File3.bin (50MB) File1.bin (150MB) File2.bin (200MB) jobindex.2
- 155. hash each file?
- 156. Compare files not pieces
- 157. { files:file1.bin,100MB,DATETIME; file2.bin,200MB,DATETIME; file3.bin,50MB,DATETIME } { piecelength:75MB } ... .torrent metadata file contents:
- 158. File1.bin (100MB) File2.bin (200MB) File3.bin (50MB) jobindex.1 Piece 1: File 0, File1 Piece 2: File 1 Piece 3: File 1, File 2 Piece 4: File 2 Piece 5: File 2, File 3 Piece 6: File 3 File1.bin (100MB) File2.bin (200MB) File3.bin (50MB) jobindex.2 File0.bin (50MB)
- 159. File1.bin (100MB) File2.bin (200MB) File3.bin (50MB) jobindex.1 File1.bin (100MB) File2.bin (200MB) File3.bin (50MB) jobindex.2 File0.bin (50MB) Piece 1: File 0, File1 Piece 2: File 1 Piece 3: File 1, File 2 Piece 4: File 2 Piece 5: File 2, File 3 Piece 6: File 3
- 160. Bittorrent Evaluation Result substantially faster drastically reduces network load on the producer machine horizontally scalable
- 161. Design GoalsDesign Goals Automatic recovery Developer empowerment System-wide visibility 3 4 5 Loose coupling2 Minimize network bottlenecks1
- 164. Headwater The beginning of a river
- 166. Headwater takes ownership of the data (hardlink + read-only)
- 169. create the .torrent metadata file
- 170. Headwater The beginning of a river River Course the water carves across the landscape
- 175. Rhone Headwater Host Data Producer Data Tracker .torrent metadata can be retrieved data.version torrent metadata
- 176. Headwater The beginning of a river River Course the water carves across the landscape Delta The end of the river
- 178. Make all subscribed artifacts available
- 183. It’s at /rad/data Delta Host Data ConsumerData Where’s the latest data? /rad/data
- 185. Keep all subscribed artifacts current
- 187. Rhone Data Host Artifact Availability Flow Delta Headwater Host Data Consumer Data Producer Data
- 188. Design GoalsDesign Goals Automatic recovery Developer empowerment System-wide visibility 4 5 Minimize network bottlenecks1 Loose coupling2 3
- 191. Development philosophy: Make recovery the common case
- 192. Durable state with atomic filesystem operations
- 193. All service calls are idempotent
- 194. RAD handles network recovery
- 197. Down DC1 DC4 DC3 DC2 No problem with bittorrent swarm
- 198. RAD treats artifact independently
- 199. Design GoalsDesign Goals Developer empowerment System-wide visibility5 Minimize network bottlenecks1 Loose coupling2 Automatic recovery3 4
- 200. Adding a new artifact in the rsync system
- 202. Adding a new artifact in the RAD system
- 203. Declare it in the code
- 204. REST API is language agnostic
- 205. Design GoalsDesign Goals System-wide visibility Minimize network bottlenecks1 Loose coupling2 Automatic recovery3 Developer empowerment4 5
- 206. Rhone already knows all artifacts
- 207. Rhone stores list of versions by artifact. version 4 version 5 version 6 artifactA version 221 version 226 version 227 version 228 artifactB version 1artifactC
- 208. Heartbeats from Delta and Headwater
- 209. Rhone has system-wide view
- 210. RADAR: Developers can easily see where their data is
- 211. RADAR: Developers can easily see where their data is
- 212. RADAR: Developers can easily see where their data is
- 213. RADAR: Developers can easily see where their data is
- 214. start simple and iterate
- 215. 2011 52 countries 2004 Indeed 2008 6 countries 2009 23 countries 2014 rsync limits 1st artifact migrated to RAD
- 216. Lesson learned: prevent people from using the system incorrectly
- 217. We made configuration TOO easy
- 218. New Requirement: protect the disks
- 219. Delta Prevent downloading artifacts that will fill the disk (and alarm)
- 220. 2011 52 countries 2004 Indeed 2008 6 countries 2009 23 countries 2014 rsync limits 1st artifact migrated to RAD 2015 critical artifacts migrated
- 221. 2011 52 countries 2004 Indeed 2008 6 countries 2009 23 countries 2014 rsync limits 1st artifact migrated to RAD 2015 critical artifacts migrated 2016 80 RAD artifacts
- 222. 2011 52 countries 2004 Indeed 2008 6 countries 2009 23 countries 2014 rsync limits 1st artifact migrated to RAD 2015 critical artifacts migrated 2016 80 RAD artifacts 100 artifacts in 10 years
- 223. 100 artifacts in 10 years 2011 52 countries 2004 Indeed 2008 6 countries 2009 23 countries 2014 rsync limits 1st artifact migrated to RAD 2015 critical artifacts migrated 2016 80 RAD artifacts 80 new artifacts in 1 year
- 224. 7,666 versions published Producer Consumer 56 unique producers 52,357 versions downloaded 670 unique consumers RAD Stats March 23, 2016
- 225. Duration of JobIndex replication in RAD v. Rsync Jan 18 6 AM 12 PM 6 PM Jan 19 6 AM 12 PM 6 PM 1,000 2,000 3,000 RAD rsync time
- 226. replicating over 65,193 TB per month
- 227. Learn More Engineering blog & talks http://indeed.tech Open Source http://opensource.indeedeng.io Careers http://indeed.jobs Twitter @IndeedEng