3. Keyword1 Keyword2 Website Name
Tag
Keyword1 Keyword2 Website Name
Tag
Unstructured Data
Enrichment
Some Transaction Description Website
Incoming raw data
Keyword1 Keyword2 Website
Structured identified data
Name
Tag
Description
Keyword1
Tag
13. Why it is useful?
• b-tree index sort values inside each node.
• b-tree is balanced
• Same level nodes are connected using doubly linked list.
14. After multicolumn index on country_id
and merchant_type Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
Postgres + multicolumn index
~4mln. Records
15. After multicolumn index on country_id
and merchant_type Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
Postgres + multicolumn index
28.73
9.88
2.1
~4mln. Records
16. After multicolumn index on country_id
and merchant_type Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
Postgres + multicolumn index
1.37
0.99
0.73
28.73
9.88
2.1
~4mln. Records
17. After multicolumn index on country_id
and merchant_type Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
Postgres + multicolumn index
10.19
5.09
2.28
1.37
0.99
0.73
28.73
9.88
2.1
~4mln. Records
18. What is GiST
Generalized Search
Tree
• In GiST each leaf contains
logical expression and
pointer to TID, where
indexed data should
satisfy logical expression.
• Faster on insert, update
What is GIN
Generalized Inverted
Index
• It is b-tree with elements to
which is connected another
b-tree or plain list of TID's.
• Faster and more accurate
on select.
19. Welcome to ruby meditation.
All of us love ruby.
Does everyone love meditation?
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1 0,10,1
2,1
1,51,5 2,1 2,1
Yellow rectangle are TID’s. First number is a page number and second is
position on a page
0,1
1,52,1
1,5
20. Welcome to ruby meditation.
All of us love ruby.
Does everyone love meditation?
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1
1,5
0,1 0,10,1
2,1
1,51,5 2,1 2,1
Yellow rectangle are TID’s. First number is a page number and second is
position on a page
2,1
1,5
ruby
rubylove
love
21. 1,5
1,5
Welcome to ruby meditation.
All of us love ruby.
Does everyone love meditation?
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1 0,1 0,10,1
2,1
1,51,5 2,1 2,1
Yellow rectangle are TID’s. First number is a page number and second is
position on a page
2,1
ruby
rubylove
love
love ruby
22. gin_trgm_ops
A trigram is a group of three consecutive characters
taken from a string.
We can measure the similarity of two strings by counting the
number of trigrams they share.
23. Performance after gin index on websites
Min
Average
Max
Seconds
0 5 10 15 20
Postgres
Elasticsearch
Postgres + multicolumn index
Postgres + gin index with trgm_ops on websites
~4mln. Records
24. Performance after gin index on websites
Min
Average
Max
Seconds
0 5 10 15 20
Postgres
Elasticsearch
Postgres + multicolumn index
Postgres + gin index with trgm_ops on websites
28.73
9.88
2.1
~4mln. Records
25. Performance after gin index on websites
Min
Average
Max
Seconds
0 5 10 15 20
Postgres
Elasticsearch
Postgres + multicolumn index
Postgres + gin index with trgm_ops on websites
1.37
0.99
0.75
28.73
9.88
2.1
~4mln. Records
26. Performance after gin index on websites
Min
Average
Max
Seconds
0 5 10 15 20
Postgres
Elasticsearch
Postgres + multicolumn index
Postgres + gin index with trgm_ops on websites
10.19
5.09
2.28
1.37
0.99
0.75
28.73
9.88
2.1
~4mln. Records
27. Performance after gin index on websites
Min
Average
Max
Seconds
0 5 10 15 20
Postgres
Elasticsearch
Postgres + multicolumn index
Postgres + gin index with trgm_ops on websites
0.55
0.34
0.26
10.19
5.09
2.28
1.37
0.99
0.75
28.73
9.88
2.1
~4mln. Records
28. How elasticsearch works
• It uses analyzers for all incoming data. (it could be custom
or default one)
• Each analyzer has at least one tokenizer
• Zero or more TokenFilters
• Tokenizer may be preceded by one or more CharFilters
37. Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
html_strip
The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
38. Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
html_strip
The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
standart tokenizer
The 2 QUICK Brown jumpedFoxes over
the lazy dog’s bone
39. Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
html_strip
The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
standart tokenizer
The 2 QUICK Brown jumpedFoxes over
the lazy dog’s bone
lowercase
the 2 quick brown jumpedfoxes over
the lazy dog’s bone
40. Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
html_strip
The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
standart tokenizer
The 2 QUICK Brown jumpedFoxes over
the lazy dog’s bone
lowercase
the 2 quick brown jumpedfoxes over
the lazy dog’s bone
stop
2 quick brown jumpedfoxes over lazy dog’s bone
the
the
41. Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
html_strip
The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
standart tokenizer
The 2 QUICK Brown jumpedFoxes over
the lazy dog’s bone
lowercase
the 2 quick brown jumpedfoxes over
the lazy dog’s bone
stop
2 quick brown jumpedfoxes over lazy dog’s bone
snowball
2 quick brown jumpfox over lazi dog bone
the
the
jump lazi dog
42. Postgres full search
implementation
• We can use tsvector type to achieve almost the same
functionality. By using to_tsvector function
• To imporve perfomance we could create separate tsvector
column with to_tsvector values.
• To create a request we should use to_tsquery. & | <->
• plainto_tsquery works with plain text so you don’t need to
insert any special symbols. Inserts &
• phraseto_tsquery also works with plain text but marks that
each token should be close to each other. Inserts <->
43. Rum access method
• Based on GIN access method code
• Solves slow ranking
• Solves slow phrase search (tsquery with <-> operator)
• Supports index on tsquery column
44. 122
1
5
3
2
4
4
3
3
4211
Welcome to ruby meditation.
All of us love ruby.
Does everyone love meditation?
ruby, meditation, love
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1 0,10,1
2,1
1,51,5 2,1 2,1
The number in green rectangle is word position in the document.
0,1
1,52,1
1,5
8,4 8,4 8,4
45. 122
1
5
3
2
4
4
3
3
4211
Welcome to ruby meditation.
All of us love ruby.
Does everyone love meditation?
ruby, meditation, love
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1
1,5
0,1 0,10,1
2,1
1,51,5 2,1 2,1
The number in green rectangle is word position in the document.
2,1
1,5
ruby
rubylove
love
love
ruby
8,4 8,4 8,4
46. 122
1
5
3
2
4
4
3
3
4211
1,5
1,5
Welcome to ruby meditation.
All of us love ruby.
Does everyone love meditation?
ruby, meditation, love
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1 0,1 0,10,1
2,1
1,51,5 2,1 2,1
The number in green rectangle is word position in the document.
2,1
ruby
rubylove
love
love ruby
love
ruby
8,4 8,4 8,4
47. Conclusion
• Postgres can also be fast.
• Multicolumn indexes can improve performance if your search has multicolumn
constraints.
• For fast text search prefer using Gin when table doesn’t update occasionally,
otherwise use GiST
• Use gin with trgm_ops when using full text search. If full text search is still slow
try to use tsvector data type with gin index on it.
• When you have some kind ‘inverse full-text search’ problem. Add tsquery type in
your table as a query and incoming data treat as a document. Add rum access
method on query column with tsquery_ops for fast classification.
• Before moving to other instrument make analysis of current/new instrument and
verify is it worth moving or not.