3. The Problem
â You are selling beer online
â You have a huge database of beers and brewreis ( Approx ~ 2
million )
â You want simple keyword based searching
â You also want structured searching
( All beers > 7% ABV )
â You want some real time analytics on how many beers are being
viewed and bought
4. Enter Elasticsearch
â Lucene Based
â Distributed
â Fast
â RESTful interface
â Document-Based with JSON
â Real Time search and analytics engine
â Open Source - Apache licence
â Platform Independent
5. Why not
Relational Database Management Systems (RDBMS) ?
â Full text search generally slower
â Cannot provide relevancy score for results
â Not suitable for unstructured data
â Limited partition tolerance ( cannot be distributed easily )
8. Real World Use Case 1 - Dell
â Switched to Elasticsearch to index 27 million documents which contained product
information
â Dell uses two Elasticsearch cluster running on Windows server (.NET framework)
â Dell uses one cluster for searching and the other for analytics. The analytics cluster
has 1 billion documents with their site and product analytics
â Dell leveraged Elasticsearchâs real time feature to create a virtual assistant which
gives relevant suggestions based on partial keywords
9. Real World Use Case 1 - The Guardian
â Switched to Elasticsearch for realtime insight into audience engagement
â Every user with access privileges can see realtime traffic and viewership data for
stories on The Guardian which helps them modify content to attract more traffic
and get more exposure during peak rates
â The guardian processes 27 million documents per day with their in house analytics
system which consists of Elasticsearch at the core
â Dell leveraged Elasticsearchâs real time feature to create a virtual assistant which
gives relevant suggestions based on partial keywords
13. Mapping in Elasticsearch
â Each index can have different types
â You can define the datatype of fields in a type with mapping
{
âsample_indexâ : {
âmappingsâ : {
âsample_type1â : {
âpropertiesâ : {
âdate_a_sample_fieldâ : {
âtypeâ : âdateâ,
âformatâ : âdateOptionalTimeâ
}
}
}
}
}
}
14. Data Types available in Elasticsearch
Mapping
â Core Types
â String
â Numeric
â Date
â Boolean
â Arrays
â Multi-fields
â Pre-defined fields ( _timestamp, _uid, _ttle )
16. Update an Index Setting
PUT /my_index_name/_settings
{
"index": {
"refresh_interval": "-1",
"number_of_replicas": 0
}
}
17. Update Index mapping by adding a Field to
a Type
PUT /my_index_name/_mapping/my_type_name
{
"my_type_name": {
"properties": {
"tag": {
"type": "keyword"
}
}
}
}
18. Get Mapping and Settings
GET /my_index_name/_mapping
GET /my_index_name/_settings
19. Create a Document
POST /my_index_name/my_type_name
{
"title": "Elastic is funny",
"tag": [
"lucene"
]
}
20. Update a Document
PUT /my_index_name/my_type_name/12abc <This is the Document ID>
{
"title": "Elastic is funny",
"tag": [
"lucene"
]
}
21. Delete a Document
DELETE /my_index_name/my_type_name/12abc
Open / Close an Index to save
memory/CPU
POST /my_index_name/_close
POST /my_index_name/_open
23. Search Scopes
GET /_search -d â...â ---> Entire cluster
GET /index_name/_search -d â...â ----> Just the index
GET /index_name/type_name/_search -d â...â ----> Just the type in the index
GET /_all/type_name/_search -d â...â ----> All type with name type_name in the cluster
GET /*/type_name/_search -d â...â ----> All type with name type_name in the cluster
GET /index_name_1,index_name_2/type_name_1,type_name_2/_search -d â...â
-----> the types in the indexes
24. Basic components of a Search request
â Query : Configures the best documents to return based on a score
â Size : Amount of documents to return
â From : Used to do pagination. Can be expensive since ES orders results.
Example - A value of 7 will return result from 8th result
â _source : The fields to return with the result
â Sort : Default or customized sorting for results
25. URL based search requests
GET /index_name/_search?from=7&size=5
GET /index_name/_search?sort=date:asc
GET /index_name/_search?sort=date:asc&q=title:elasticsearch
32. bool Query
must Combine clauses with AND
must_not Combine clauses with binary NOT
should Combine clauses with binary OR
33. Aliases in Elasticsearch
â Grouping multiple indexes or a single index and
giving it an alias name for the purpose of
querying / searching
â Aliases can be used with a filter to create
multiple âviewsâ of the same index
34. Create an Alias for single Index
POST /_aliases
{
"actions":
{
"add": {
"index": "my_index_name",
"alias": "alias_name_1",
}
}
}
35. Remove an Alias for single Index
POST /_aliases
{
"actions":
{
"remove": {
"index": "my_index_name",
"alias": "alias_name_1",
}
}
}
36. Create an Alias for multiple Indicies
POST /_aliases
{
"actions":
{
"add": {
"indices": ["my_index_name","my_index_name2"]
"alias": "alias_name_1",
}
}
}
37. Create an Alias for multiple Indicies with
wildcard
POST /_aliases
{
"actions":
{
"add": {
"index": "my_index_name*"
"alias": "alias_name_1",
}
}
}
38. Create an Alias with filter term and routing value
POST /_aliases
{
"actions":
{
"add": {
"index": "my_index_name",
"alias": "bar",
"filter" : { "term" : { "customer_id" : "1" } }
"routing" : 1
}
}
}
Routing values can be used
To avoid unnecessary shard operations
They are added to the document automaticall
And will be added to the search query which is
Using the alias
39. Design Problem
Your application has multiple users and each user
has multiple subscriptions. Assume subscriptions
follow the same format more or less
What is the best way to design an elasticsearch
cluster for this type of problem?
40. Potential Designs
â Have an index per user and each user has subscription documents
Advantages
â Searches are fairly easy
â In line with relational database mentality
â Controlling shards and replica of each index gives us the advantage
of independent scaling
Disadvantages
â Too many shards!
â Waste of space as not all shards will have documents up to their
capacity
â Couple thousand customers can make elasticsearch unresponsive
41. Potential DesignsâŠ(contd)
â Single Index + Aliasing
Advantages
â No extra shards needed
Disadvantages
â Each shard will be hit when querying making queries slower
42. Best Solution
Single Index + Aliasing with Routing enabled
â Create alias for each customer in the single index
â Assign a routing key to each alias which is the same as customer
id
â Now each query hits only specific shards making queries really
fast