Realtime Analytics With Elasticsearch [New Media Inspiration 2013]

Real time analytics
of big data with Elasticsearch

Karel Minařík

cets
Fa

ly tics
SON Ana
J

http://www.youtube.com/watch?v=-GftBySG99Q

http://karmi.cz
http://elasticsearch.com

Realtime Analytics With ElasticSearch

Using a search engine for analytics?

wat?


HOW DOES SEARCH WORK?

A collection of documents

file_1.txt
The ruby is a pink to blood-‐red colored gemstone ...

file_2.txt
Ruby is a dynamic, reflective, general-‐purpose object-‐oriented
programming language ...

file_3.txt
"Ruby" is a song by English rock band Kaiser Chiefs ...


How do you search documents?

File.read('file_1.txt').include?('ruby')
File.read('file_2.txt').include?('ruby')
...


The inverted index

TOKENS POSTINGS

ruby file_1.txt file_2.txt file_3.txt
pink file_1.txt
gemstone file_1.txt

dynamic file_2.txt
reflective file_2.txt
programming file_2.txt

song file_3.txt
english file_3.txt
rock file_3.txt

http://en.wikipedia.org/wiki/Index_(search_engine)#Inverted_indices


The inverted index

search "ruby"

pink file_1.txt
gemstone file_1.txt

dynamic file_2.txt

song file_3.txt
english file_3.txt
rock file_3.txt



The inverted index

search "song"

pink file_1.txt
gemstone file_1.txt

dynamic file_2.txt

song file_3.txt
english file_3.txt
rock file_3.txt



The inverted index

search "ruby AND song"

pink file_1.txt
gemstone file_1.txt

dynamic file_2.txt

song file_3.txt
english file_3.txt
rock file_3.txt



The inverted index

TOKENS POSTINGS
Statistics!

ruby 3 file_1.txt file_2.txt file_3.txt
pink 1 file_1.txt
gemstone file_1.txt

dynamic file_2.txt

song file_3.txt
english file_3.txt
rock file_3.txt


ElasticSearch is an open source, scalable,
distributed, cloud-ready, highly-available full-
text search engine and database with powerful
aggregation features, communicating by JSON
over RESTful HTTP, based on Apache
Lucene.


FACETS

Faceted Navigation

Query

Facets

http://blog.linkedin.com/2009/12/14/linkedin-faceted-search/

FACETS

Faceted Navigation with Elasticsearch

curl "http://localhost:9200/people/_search?pretty=true" -‐d '
{
"query" : {
"match" : { "name" : "John"} User query
},
"filter" : {
"terms" : { "employer" : ["IBM"] } “Checkboxes”
},
"facets" : {
"employer" : {
"terms" : { Facets
"field" : "employer",
"size" : 3
} "facets" : {
} "employer" : {
} "missing" : 0,
}'
"total" : 10,
"other" : 3,
"terms" : [ {
"term" : "ibm",
Response "count" : 3
}, {
"term" : "twitter",
"count" : 2
}, {
"term" : "apple",
"count" : 2
} ]
}
}

http://www.elasticsearch.org/guide/reference/api/search/facets/index.html

FACETS

Visualizing the Facets

"facets" : {
"employer" : {
"missing" : 0,
"total" : 10,
"other" : 3,
"terms" : [ {
"term" : "ibm",
"count" : 3
}, {
"term" : "twitter",
"count" : 2
}, {
"term" : "apple",
"count" : 2
} ] DEMO: http://bl.ocks.org/4571766
}
}

d3.js ~ A Bar Chart, Part 1
http://mbostock.github.com/d3/tutorial/bar-1.html

FACETS


FACETS


http://demo.kibana.org

Important Concepts
‣ No batch orientation
‣ No stats precomputation and caching
‣ No predefined metrics or schemas

‣ Combination of free text search, structured
search, and facets
‣ Scripting for performing ad–hoc analytics
‣ Extendable: write your own facet types


FACETS

Scripting
Extract and aggregate most popular domains from article URLs
curl -X DELETE localhost:9200/demo-articles
curl -X POST localhost:9200/demo-articles -d '{"mappings": { "a": { "properties": {"url": {type: "string", "index": "not_analyzed"}} } } }'

curl -X PUT localhost:9200/demo-articles/a/1 -d '{"title":"...","url":"http://some.blogger.com/2012/09/01/index.html"}'
curl -X PUT localhost:9200/demo-articles/a/3 -d '{"title":"...","url":"http://some.blogger.com/about.html"}'
curl -X PUT localhost:9200/demo-articles/a/5 -d '{"title":"...","url":"https://github.com/user/A"}'
curl -X PUT localhost:9200/demo-articles/a/5 -d '{"title":"...","url":"http://github.com/user/B"}'
curl -X POST localhost:9200/demo-articles/_refresh

curl -X GET 'localhost:9200/demo-articles/_search/?search_type=count&pretty' -d '{
"facets": {
"popular-domains": {
"terms": {
"field" : "url",
"script" : "term.replace(new RegExp("https?://"), "").split("/")[0]",
"lang" : "javascript"
}
}
}
"facets" : {
}'
"popular-‐domains" : {
// ...
"terms" : [ {
Response "term" : "some.blogger.com", "count" : 3
}, {
"term" : "github.com", "count" : 1
} ]
}
}

FACETS

Demonstrations
Extract and aggregate most popular domains from article URLs
curl -X DELETE localhost:9200/demo-articles
curl -X POST localhost:9200/demo-articles -d '{"mappings": { "a": { "properties": {"url": {type: "string", "index": "not_analyzed"}} } } }'

curl -X PUT localhost:9200/demo-articles/a/3 -d '{"title":"...","url":"http://some.blogger.com/about.html"}'
curl -X PUT localhost:9200/demo-articles/a/5 -d '{"title":"...","url":"https://github.com/user/A"}'
curl -X PUT localhost:9200/demo-articles/a/5 -d '{"title":"...","url":"http://github.com/user/B"}'
curl -X POST localhost:9200/demo-articles/_refresh

curl -X GET 'localhost:9200/demo-articles/_search/?search_type=count&pretty' -d '{
"facets": {
"popular-domains": {
"terms": {
"field" : "url",
"script" : "term.replace(new RegExp("https?://"), "").split("/")[0]",
"lang" : "javascript"
}

}'
}
}
Demo
"facets" : {
"popular-‐domains" : {
// ...
"terms" : [ {
Response "term" : "some.blogger.com", "count" : 3
}, {
"term" : "github.com", "count" : 1
} ]
}
}

Realtime Analytics With Elasticsearch [New Media Inspiration 2013]

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (16)

Similar to Realtime Analytics With Elasticsearch [New Media Inspiration 2013]

Similar to Realtime Analytics With Elasticsearch [New Media Inspiration 2013] (20)

More from Karel Minarik

More from Karel Minarik (18)

Recently uploaded

Recently uploaded (20)

Realtime Analytics With Elasticsearch [New Media Inspiration 2013]