To be presented @ RootConf, 2018

1. Growing with ElasticSearch
Devi A S L @ RootConf
11th
May, 2018

2. About me
● Over a decade of experience in building software
● Lead developer/Architect at PowerToFly

3. Our journey with ElasticSearch
2014: launched with Postgres Full text search
2015: Faceted Search with ES v1.4
2016: Log monitoring system with ELK 2.3
2017: Analytics pipeline with ELK 5.5

4. Search for a search engine
Postgres
v9.3
Sphinx
v2.1
Solr
v4.x
ElasticSearch
v1.4
Full text search ✓ ✓ ✓ ✓
Support for facets ❌ ✓ ✓ ✓
Cluster ready ❌ ❌ Limited ✓
Search in PDFs ❌ ❌ ✓ ✓
REST APIs ❌ ❌ ❌ ✓
Nested docs,
Parent-Child relations
❌ NA Limited ✓
Powerful and Flexible
Query DSL
❌ NA ❌ ✓

5. distributed, multitenant-capable, full-text search engine.
● Built upon battle tested Lucene
● Powerful and flexible Query DSL
● Powerful Aggregations
● REST APIs for everything
● Ease with nested documents and parent-child relationships
● Suitable eco system for data pipelines
The goodness of ElasticSearch

7. What sits where ?
Internet
Search
Service
ES
cluster
Periodic
Indexing
job
Postgres
DB
Primary datastore
for
core data
jobs, candidates
data

8. Log monitoring with ELK

9. Log monitoring: From a third-party solution to ELK based
AWS
S3
ElasticSearch cluster
web & worker nodes
with filebeat
logstash
Dashboards
on
Kibana
Daily indices
logs

11. Analytics pipeline with ELK stack

12. Recommendation
engine
Web Application
ElasticSearch cluster
web nodes
with filebeat
logstash
User activity
Kibana
Dashboards
Daily indices

14. Handling growth

15. ● enable slow query log, customizable per index
Search performance tuning

16. ● Avoid nested documents, if you can
Document modelling

17. ● Deep pagination is costly with search API
Use scroll API where applicable

18. ● POST /unused_index/_close
● POST /index_with_more_segments/_forcemerge
● Use _rollover API to let hot/recent indexes use best servers
Manage your indexes

19. ● Disable indexing, storing, norms, _source when you don’t need
● Use smallest numeric data or make it keyword
● Optimize number of primary shards
● Use bulk requests, optimize their size
Index performance tuning

20. Summary
● Elastic stack is growing and improving - see if it fits your needs
● Defaults are good only to start - know what they are and tune them
● Different indexes for different data
● Understand your needs and model your documents well

21. Thank You!
@asldevi