Presented at RubyConfBR 2017

1. Dealing with a search engine
in your application
a Solr approach for beginners
Elaine Naomi Watanabe

2. Elaine Naomi Watanabe
Full-stack developer
(Playax)
Master's degree in Computer Science
(IME-USP)
Passionate about:
Web Development, Agile,
Cloud Computing, DevOps,
NoSQL and RDBMS

3. ANALYZING BILLIONS OF DATA TO HELP
ARTISTS AND MUSIC PROFESSIONALS TO
DEVELOP THEIR AUDIENCE
BIG DATA + MUSIC + TECH = <3

4. AGENDA

5. Searching
Problem
Introduction
Information
Retrieval
Basic concepts
Apache
Solr
How to configure
Sunspot
Gem
Integrating with
Ruby on Rails
Next
Steps
Including
references
SPOILER ALERT

6. THE SEARCHING
PROBLEM

7. or did you mean...
HOW TO SEARCH LIKE THE
GOOGLE?

8. A LIST OF SONGS…
TITLES, ARTISTS, LYRICS...
IMAGINE IN OUR CONTEXT…

9. "Imagine all the people living life in peace"
(Imagine - John Lennon)
"I'm a radioactive, radioactive"
(Radioactive - Imagine Dragons)
"Welcome to the jungle,
watch it bring you to your knees"
(Welcome to the jungle - Guns N' Roses)
A little little sample of songs...

10. A SQL LIKE statement is enough?
SEARCHING TERM: "IMAGINE"
SELECT *
FROM songs
WHERE
title LIKE '%IMAGINE%' OR
artist LIKE '%IMAGINE%' OR
lyrics LIKE '%IMAGINE%';

11. "Imagine all the people living life in peace"
(Imagine - John Lennon)
"I'm a radioactive, radioactive"
(Radioactive - Imagine Dragons)
"Welcome to the jungle,
watch it bring you to your knees"
(Welcome to the jungle - Guns N' Roses)
Searching for "Imagine"

12. A SQL LIKE statement is really enough??
SEARCHING TERM:
"IMAGINE PEOPLE"

13. "Imagine all the people living life in peace"
(Imagine - John Lennon)
"I'm a radioactive, radioactive"
(Radioactive - Imagine Dragons)
"Welcome to the jungle,
watch it bring you to your knees"
(Welcome to the jungle - Guns N' Roses)
A little little sample of songs...

14. A SQL LIKE statement is really enough??
SEARCHING TERM: "IMAGINE PEOPLE"
SELECT *
FROM songs
WHERE
title LIKE '%IMAGINE%PEOPLE%' OR
artist LIKE '%IMAGINE%PEOPLE%' OR
lyrics LIKE '%IMAGINE%PEOPLE%';

15. USER x YOUR APP
A BUG SEARCH TOOL

16. When LIKE STATEMENT is not enough...
SEARCH TERMS:
"Dragons Imagine"
"Imagine John"
"Imagine JONH" <- TYPO!

17. When only Yahoo! Answers is the solution...
Ueca tudi diango...
tanananananananaann nisss
♬ welcome to the jungle,
watch it bring you to your knees ♬
(╯°▽°)╯ ︵ ┻━┻

18. INFORMATION
RETRIEVAL

19. Unstructured data
Large number of documents

20. IN THE PAST...
List all documents that match a search query
was enough…
However, in a Big Data era…

21. NOWADAYS …
Ranking documents by their relevance
for a search query is the most important goal.

22. Basic concepts

23. TOKENIZATION: Tokens ~> Words
A list wordsof"A list of words!"
Tokens
semantic units

24. DON T PANIC
"DON'T PANIC"
Is it enough to remove punctuation and spaces?
"do not panic" do not panic
How to tokenize contractions?
Are all of them semantic units?
Are they same tokens?
don't = do not
DON'T PANIC

25. Imagine Dragons
"Imagine Dragons"
Imagine Dragons
Is it enough to remove punctuation and spaces?
or

26. you know
"you-know-who" you-know-who
Is it enough to remove punctuation and spaces?
or
who

27. "Fullstack developer"
"Full-stack dev"
"Full stack developer"
Is it enough to remove punctuation and spaces?
Fullstack
Full developerstack
developer
Full-stack developer
For a user, these terms should return the same documents, isn't it?

28. 30 seconds to Mars Thirty seconds to Mars
November, 18th, 2017 2017-11-18
SP São Paulo
How to deal with numbers and abbreviations?

29. Kaminari
Is a gem or thunder in
japanese?
Windows
Is it plural of window or
about the company?
About the semantics of the original term and its normalized token...

30. 音楽
ONGAKU
お​んがく
SAME LANGUAGE, SAME PRONUNCIATION
DIFFERENT ALPHABETS

31. STOP WORDS: extremely common words
In English:
a, an, the, and, or, are, as, at, by,
for, from, of ...
In Portuguese:
um, uma, a, o, as, os, é, são, por, de,
da, do, se …

32. STOP WORDS: extremely common words
A list wordsof list words
meaningful tokens

33. Stop words, diacritics, case folding...
Stop word
removal
Case folding
normalization
Diacritics
removal
HELLO WORLD
Hello World
hello world
hello
world
naive
naïve
naive
roses are red
red roses
roses
red

34. When not to normalize tokens…
The Who (a band)
Se (Brazilian song, from Djavan)
Strings solely composed by stop
words
Different meanings for words
with and without
diacritics
In Spanish:
peña means a "cliff"
pena means "sorrow"
When not to set all characters to
lowercase
General Motors
Windows
Apple

35. LEMMATIZATION / STEMMING
To reduce a token to its base form

36. LEMMATIZATION: based on a vocabulary
am, are, is be
sou, somos, foi, é ser
car, cars, car’s, cars’ car
English
Portuguese
carros, carro carro

37. STEMMING
Heuristic process that chops off the ends of words
cats cat
ponies poni
Increase the number of returned documents.
However, harming precision...

38. STEMMING
Heuristic process that chops off the ends of words
amor
amoramores
amora
operating operat
system
Portuguese
system
English
It means love
It's a Brazilian berry
not so meaningful tokens

39. SYNONYMS
bike bicycle
indivíduo pessoa

40. Bag of words:
List of keywords
Ordering of words is ignored!
e.g.
Imagine Dragons
Dragons Imagine
Phrase queries:
Order matters!
Restrict searches
e.g.
"Imagine Dragons"

41. RELEVANCE
term frequency (tf)
total of occurrences of a term in a document
inverse document frequency (idf)
how rare is a term in all indexed documents

42. RELEVANCE
tf-idf = tf x idf
function that balances the term frequency in a
document within how rare is term in a collection

43. Boolean Model
Probabilistic Model
PageRank
...

44. Evaluating a
searching model

45. Evaluation method
We need a test dataset with:
1. A document collection
2. A collection of queries
3. A set of relevance judgments, for each query, a list of relevant and
non-relevant documents
TP: True Positive
TN: True Negative
FP: False Positive
FN: False Negative

46. ACCURACY
TP + TN
TP + FP + TN + FN

47. PRECISION
TP
TP + FP
# Corrected Matches / # Total Results Returned

48. RECALL
TP
TP + FN
# Corrected Matches /
(# Corrected Matches + # Missed Matches)

49. F1 SCORE
2 * (RECALL + PRECISION)
(RECALL + PRECISION)

50. When a model is good enough for an app?
You can choose the model with the best F1 score, for
example.
However, there is no universal solution
It is an incremental process
You should tune it based on users' information needs
Usability tests is also a good way to evaluate a model

51. Adding a search
engine to my app

52. FULL TEXT SEARCH IN MARIADB...
CREATE TABLE `songs` (
`id` int NOT NULL AUTO_INCREMENT PRIMARY KEY,
`title` varchar(300),
`artist` varchar(255),
`genre` varchar(255),
`lyrics` text
) ENGINE=InnoDB;
CREATE FULLTEXT INDEX songs_title_idx ON songs (title);
CREATE FULLTEXT INDEX songs_artist_idx ON songs (artist);
CREATE FULLTEXT INDEX songs_lyrics_idx ON songs (lyrics);
CREATE FULLTEXT INDEX songs_genre_idx ON songs (genre);
FTS

53. FULL TEXT SEARCH IN MARIADB...
SELECT *
FROM songs
WHERE MATCH (title,artist, lyrics) AGAINST ('imagine' IN NATURAL
LANGUAGE MODE);
SELECT *
FROM songs
WHERE MATCH (title,artist, lyrics) AGAINST ('imagine' IN BOOLEAN MODE);
CREATE FULLTEXT INDEX songs_all_idx ON songs (title,artist,lyrics);
default mode

54. FULL TEXT SEARCH IN MARIADB...
SELECT *
FROM songs
WHERE MATCH (title,artist, lyrics) AGAINST ('imagine dragons');
Returned rows:
Radioactive - Imagine Dragons
Imagine - John Lennon

55. FULL TEXT SEARCH IN MARIADB...
SELECT *
FROM songs
WHERE MATCH (title,artist, lyrics)
AGAINST ('+imagine +dragons') IN BOOLEAN MODE);
Radioactive - Imagine Dragons

56. FULL TEXT SEARCH IN MARIADB...
SELECT *
FROM songs
WHERE MATCH (title,artist, lyrics) AGAINST ('"imagine dragons"');
Radioactive - Imagine Dragons

57. FULL TEXT SEARCH IN MARIADB...
SELECT * FROM songs WHERE MATCH(genre) AGAINST('alternative');
SELECT * FROM songs WHERE MATCH(genre) AGAINST('music');
SELECT * FROM songs WHERE MATCH(genre)
AGAINST('alternative' WITH QUERY EXPANSION);
SELECT * FROM songs WHERE MATCH(genre)
AGAINST('music' WITH QUERY EXPANSION);
Imagine Dragons
John Lennon
Imagine Dragons
John Lennon
Imagine Dragons - Alternative Rock
John Lennon - Rock music, Pop music

58. Why to use an external search engine?
Spell checking!
Spell checking! Or did you mean…
search like Google? ♡

59. Why to use an external search engine?
You can use spell checking!
You can also:
- Add multivalued fields (document oriented database)
- Add new algorithms to the databases
- Customize stop words, stemming analyzers
- Use fuzziness functions
- Boost some documents/fields according to the search

60. Apache Solr and ElasticSearch
Based on Apache Lucene
Document oriented databases (welcome to polyglot persistence!)
It is not a relational database, ok? No ACID, sorry!
Developed to be scalable
Apache Solr has a better documentation +50
ES has native support to Structured Query DSL +1
ES is better for analytic queries

61. ElasticSearch DSL
// artist = John Lennon AND (genres = rock OR genres = pop)
// AND NOT(nome = imagine)
GET /songs/v1/_search
{
"query" : {
"bool": {
"must": {"match": {"artist": "John Lennon" }},
"should": [
{"match": {"genres": "rock" }},
{"match": {"genres": "pop" }}
],
"must_not": {"match": {"nome": "imagine"}}
}
}
}

62. Our choice: Apache Solr
Apache Solr is Open Source and Open Development +1000
Latest release: 7.1.0 (October 17th, 2017)

63. Apache Solr

64. Installing for development environment...
docker run --name my_solr -p 8983:8983 -d solr
https://hub.docker.com/r/risdenk/docker-solr/

65. localhost:8983

66. Creating a core
docker exec -it my_solr solr create_core -c development
core ~> database or table
document ~> a row from a table
schemaless!!
core name

67. List of all cores

68. Menu options for each core

69. Creating a document...
curl -X POST -H 'Content-Type: application/json'
'http://localhost:8983/solr/development/update/json/docs' --data-binary
'
{
"id": "1",
"title": "Song 1"
}'

70. Zero documents??
Check!

71. Commit!!
curl -X POST -H 'Content-Type: application/json'
'http://localhost:8983/solr/development/update?commit=true'
--data-binary '
{
"commit": {}
}'

72. Creating a document...
curl -X POST -H 'Content-Type: application/json'
'http://localhost:8983/solr/development/update?commit=true'
--data-binary '
[{
"id": "1",
"title": "Song 1"
},{
"title": "Song 2"
}]'
Optional in insert

73. Our new documents!

74. Our new documents!
title and title_str?
dynamic fields
*_str, *_i, ...

75. Updating a document...
curl -X POST -H 'Content-Type: application/json'
'http://localhost:8983/solr/development/update?commit=true' --data-binary '
[{
"id": "1",
"title": "Song 3"
},
{
"title": "Song 3"
}]'

76. id: 1
new doc

77. Documents menu - JSON

78. Deleting a document...
curl -X POST -H 'Content-Type: application/json'
'http://localhost:8983/solr/development/update' --data-binary '
{
"delete": { "id":"1" },
"commit": {},
}'

79. Deleting ALL documents
curl -X POST -H 'Content-Type: application/json'
'http://localhost:8983/solr/development/update?commit=true' --data-binary '
{
"delete": {
"query": "*:*"
}
}'

80. Documents menu - Solr command

81. Searching

82. q (query) main query parameter
fq (filter query) filter query
(to reduce the dataset)
fl (filter list) list of fields to return
sort list of fields to sort the
dataset
Results are paginated
QUERY

83. Basic queries
List all documents (with pagination)
curl 'http://localhost:8983/solr/development/select?q=*:*'

84. Basic queries
List all documents (with pagination)
curl http://localhost:8983/solr/development/select -d '
{
query:"*:*"
}'

85. My documents
{
"docs": [
{
"title": ["Song 1"],
"genre": "Rock",
"year": 2010
},
{
"title": ["Song 2"],
"genre": "MPB",
"year": 1990
}, {
"title": ["Other music Rock"],
"genre": "Pop",
"year": 1970
},
{
"title": ["My favorite songs"],
"genre": "Rock Music",
"year": 2011
}
]
}

86. Fuzzy matching
title:Song* 3 documents
title:Song? 1 document
title:Sonjs 0 documents
title:Sonjs~1 1 document
title:Sonjs~2 3 documents
title:(my songs) 1 document
title:"my songs" 0 documents
title:"my songs"~2 1 document
title:(-favorite +song*) 2 documents
*:* 4 documents
Wildcards:
? one letter
* any number letter
~ query slop
( ) keyword search
" " phrase query

87. Fuzzy matching
title:"song" AND genre:"rock" 1 document
(title:"song" AND genre:"rock") OR title:"track" 2 documents
year: [1980 TO *] 3 documents
genre:[Pop TO *] 3 documents
Boosting:
(title:music OR title:Rock)^1.5 (genre:music OR genre: Rock) 3 documents
1st: "Other music rock"
(title:music OR title:Rock) (genre:music OR genre: Rock)^1.5 3 documents
1st: "My favorite songs"

88. Searching in all fields
In your schema.xml, add:
<copyField source="*_txt" dest="_text_" />
<copyField source="*_text" dest="_text_" />
You can add but it is not recommended:
<copyField source="*" dest="_text_" />
Then, you can search without defining the default field

89. Analysis: list all indexing and querying transformations
Indexing
Transform.
Querying
Transform.

90. Customizing fields and their analyzers (schema.xml)
<fieldtype name="phonetic" stored="false" indexed="true"
class="solr.TextField" >
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.DoubleMetaphoneFilterFactory" inject="false"/>
<filter class="solr.StopFilterFactory" format="snowball"
words="lang/stopwords_pt.txt" />
</analyzer>
</fieldtype>

91. Spell Checking

92. Building the spell checking indexing
curl --request GET --url
'http://localhost:8983/solr/development/select?q=*:*&spellcheck.build=t
rue&spellcheck=true'

93. Suggestion:
IMAGINE
Searching: IMAGINA

94. Searching: DRAGOONS
Suggestion:
DRAGONS

95. Searching:
IMAGINA DRAGOONS
Suggestion:
IMAGINE DRAGONS

96. Integrating with
Ruby on Rails

97. Connecting through a REST Client …
params = {q: 'title:song' }
response = RestClient.get
"http://localhost:8983/solr/development/select?#{params.to_param}"
response_json = JSON.parse(response.body)
items = response_json["response"]["docs"]
[{"title"=>["Song 1"], "id"=>"eeb507c6-461f-4219-9f5a-50528340d84d",
"_version_"=>1584234836063682560, "title_str"=>["Song 1"]},
{"title"=>["Song 2"], "id"=>"1b8bacc1-9ed9-4c85-922d-71b3472f9d44",
"_version_"=>1584234836065779712, "title_str"=>["Song 2"]}]
ヽ(•́o•̀)ノ

98. Sunspot Gem
V. 2.2.7

99. Installing...
gem 'sunspot_rails'
rails generate sunspot_rails:install
development:
solr:
hostname: solr
port: 8983
path: /solr/playax
log_level: INFO
auto_index_callback: after_commit
auto_remove_callback: after_commit
config/sunspot.yml

100. Sunspot needs its own schema.xml.
Follow this example in:
elainenaomi/search_engine

101. Sunspot DSL - Defining the indexed fields
class Song < ActiveRecord::Base
searchable do
text :title, stored: true
text :lyrics, stored: false
text :artist, stored: true
string :genre, multiple: true, stored: true do
genre.split(',')
end
end
end
Sunspot.index! Song.all

102. Bag of words:
search = Song.search do
fulltext 'imagine dragons'
with :genre, 'Rock'
without :genre, 'Pop'
with(:year).less_than 2014
field_list :title, :artist
order_by :title, :asc
end
songs = search.results
Imagine
(John Lennon)
Radioactive
( Imagine Dragons)

103. Phrase queries:
search = Song.search do
fulltext ""imagine dragons""
with :genre, 'Rock'
without :genre, 'Pop'
with(:year).less_than 2014
field_list :title, :artist
order_by :title, :asc
end
songs = search.results
Radioactive
( Imagine Dragons)

104. Query Phrase Slop
# Two words can appear between the words in the phrase, so
# "imagine all the people" also matches, in addition to "imagine people"
Song.search do
fulltext '"imagine people"' do
fields :lyrics
query_phrase_slop 2
end
end

105. Minimum Match
Song.search do
fulltext "dragons imagine test" do
fields :artist, :title
minimum_match '70%'
end
end
Song.search do
fulltext 'dragons imagine test' do
fields :artist, :title
boost_fields title: 2.0
minimum_match '60%'
end
end
1 document:
Radioactive
( Imagine Dragons)
2 documents:
1st: Imagine
(John Lennon)
2nd: Radioactive
(Imagine Dragons)
boost
rounded down

106. Spell checking
search = Sunspot.search(Song) do
keywords 'Imagina Dragoons'
spellcheck :count => 3
end
search.spellcheck_suggestion_for('imagina') # => 'imagine'
search.spellcheck_suggestions # => [{"word"=>"imagine",
"freq"=>3}, {"word"=>"dragons", "freq"=>1}]

107. To test or not to test?

108. To test or not to test?
Unit tests? No.
Integration tests? Maybe…
Search engines depends on terms frequency to ranking docs
You will need all your dataset to compute precision, recall..
You can test only filter queries, indexing callbacks…

109. Summary

110. Summary
The searching
problem
● User: a bug search tool
Adding a search
engine to my app
● Full text search in MariaDB
● Apache Solr x ElasticSearch
Apache Solr
● How to create cores
● CRUD operations
Integrating with Rails
● Sunspot gem
● How to index, search and test

111. Keep in mind
Always verify the user's information needs from your app
E.g.: check if removing stop words, synonymous should be applied
"No" Meghan Trainor
"I am" - P.O.D
E.g: which transformations your search engine should apply
- Phonetic transformations? Custom language analyzers?

112. Keep in mind
The information is not only on text files but also in
audios, videos, images, etc.

113. Suggested topics for studying
- Evaluation of available analyzers for FTS
- Optimization of Performance (such as soft commit, lazy build indexes)
- Distribution and replication through SolrCloud
- Using of Machine Learning algorithms
- Creation of custom function queries
- Authentication
- Integrating with Logstash and Kibana
- Geospatial searches

114. Thank you! <3
github.com/elainenaomi
slideshare.net/elainenaomi
@elaine_nw

115. References
Introduction to Information Retrieval
Manning, Christopher D., Prabhakar Raghavan, and Hinrich Schütze (2008)
Solr in action
Grainger, Trey, Timothy Potter, and Yonik Seeley (2014)
Sunspot gem
http://sunspot.github.io/
Uma introdução ao tema recuperação de informações textuais.
Barth, F. J. (2013)
10 Reasons to Choose Apache Solr Over Elasticsearch (2016)
https://dzone.com/articles/10-reasons-to-choose-apache-solr-over-elasticsearc

116. References
Apache Solr vs Elasticsearch
http://solr-vs-elasticsearch.com/
When to consider Solr
https://stackoverflow.com/questions/4960952/when-to-consider-solr
Indexing for full text search in PostgreSQL
https://www.compose.com/articles/indexing-for-full-text-search-in-postgresql/
PolyglotPersistence
https://martinfowler.com/bliki/PolyglotPersistence.html
Yahoo! Answers: Qual o nome desta Música?
https://br.answers.yahoo.com/question/index?qid=20080627085726AAJM9Wa

117. References
Full-Text Index in MariaDB
https://mariadb.com/kb/en/library/full-text-index-overview/
Natural Language Full-Text Searches (MySQL)
https://dev.mysql.com/doc/refman/5.7/en/fulltext-natural-language.html
Postgres full-text search is Good Enough! (2015)
http://rachbelaid.com/postgres-full-text-search-is-good-enough/
Text Indexes in MongoDB
https://docs.mongodb.com/manual/core/index-text/
Full-Text Index Stopwords for MariaDB
https://mariadb.com/kb/en/library/full-text-index-stopwords/