Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Taking Elasticsearch From 0 to 88mph
Semelhante a Taking Elasticsearch From 0 to 88mph (20)
Mais de Molly Struve
Mais de Molly Struve (13)
Último
Último (20)
Taking Elasticsearch From 0 to 88mph
- 1. Taking Elasticsearch from 0 to 88 mph By: Molly Struve
- 2. 500 million documents 1 million processed daily
- 3. Over 3 billion documents 200 million processed daily
- 4. The average company has... 60 thousand assets 24 million vulnerabilities?
- 10. In-memory buffer
- 11. Toggle the Refresh Interval curl -XPUT 'localhost:9200/my_index/_settings' -d '{ "index" : { "refresh_interval" : "30s" } }'
- 12. In addition... ● ?refresh=wait_for option ● Manual refresh ○ curl -XPOST 'localhost:9200/my_index/_refresh'
- 13. Speed up Indexing ● Toggle the refresh interval1 2 3 Speed Up Searching 4 5 6 7
- 14. ● 200 thousand assets ● 100 million vulnerabilities
- 17. Bulk Processing
- 18. POST _bulk
- 19. When bulk processing your data... ● Start with batches of 100 and double the size from there until indexing time plateaus ● Too large requests can put memory pressure on Elasticsearch so keep it under a couple tens of megabytes
- 20. Bulk process data Speed up Indexing Toggle the refresh interval1 2 3 Speed Up Searching 4 5 6 7
- 22. 429 Too Many Requests
- 24. 8 Threads 4 Threads Shard 2 Shard 3 Shard 4 4 Threads 4 Threads Shard 1 Shard 2 Shard 3 Shard 4 Shard 1 Shard 1 Shard 2 Shard 2 2 Threads 2 Threads Shard 1 Shard 3 Shard 4 Shard 4 Shard 3
- 25. Routing shard = hash(_routing) % number_of_primary_shards PUT my_index/_doc/1?routing=custom { "title": "This is a document" } document _id custom
- 26. 4 Threads 2 Threads 2 Threads Shard 1 Shard 2 Shard 2 Shard 1 Shard 3 Shard 4 Shard 4 Shard 3 Route 1 Route 2 Route 2 Route 1 Route 3 Route 4 Route 4 Route 3
- 27. Parent -> Child Asset -> Vulnerabilities PUT my_index/_vulnerability/1?parent=2 { "title": "This is a document" }
- 28. Route your documents Bulk process data Speed up Indexing Toggle the refresh interval1 2 3 Speed Up Searching 4 5 6 7
- 31. Group Your Data
- 32. Client 1 Client 2 Client 3 Client 4
- 33. Speed Up Searching Group your data Route your documents Bulk process data Speed up Indexing Toggle the refresh interval1 2 3 4 5 6 7
- 34. Does NOT score documents and only cares if the document matches the search criteria or not Queries Scores documents based on how well they match the search criteria. Filters Easy/Fast Hard/Slow Easy/Fast Hard/Slow
- 35. 2.x 5.x
- 41. GET /_search { "query": { "bool" : { "must" : { "term" : { "user" : "kimchy" } }, "filter": { "term" : { "tag" : "tech" } } } } }
- 42. Use filters whenever possible Speed Up Searching Group your data Route your documents Bulk process data Speed up Indexing Toggle the refresh interval1 2 3 4 5 6 7
- 44. Store IDs as keywords
- 45. Why? ● Numeric mapping types are optimized for RANGE queries ● Keyword mapping types are optimized for TERM queries
- 47. 30% increase in search speed
- 48. Store IDs as keywords Use filters whenever possible Speed Up Searching Group your data Route your documents Bulk process data Speed up Indexing Toggle the refresh interval1 2 3 4 5 6 7
- 51. Don’t let your users slow you down
- 53. Define keywords for searching
- 54. Don’t let your users slow you down Store IDs as keywords Use filters whenever possible Speed Up Searching Group your data Route your documents Bulk process data Speed up Indexing Toggle the refresh interval1 2 3 4 5 6 7
- 55. Questions?