6. What s NLP4L?
• GOAL
• Improve Lucene users search experience
• FEATURES
• Use of Lucene index as a Corpus Database
• Lucene API Front-end written in Scala
• NLP4L provides
• Preprocessors for existing ML tools
• Provision of ML algorithms and Applications (e.g. Transliteration)
6
7. What s NLP4L?
• GOAL
• Improve Lucene users search experience
• FEATURES
• Use of Lucene index as a Corpus Database
• Lucene API Front-end written in Scala
• NLP4L provides
• Preprocessors for existing ML tools
• Provision of ML algorithms and Applications (e.g. Transliteration)
7
8. What s NLP4L?
• GOAL
• Improve Lucene users search experience
• FEATURES
• Use of Lucene index as a Corpus Database
• Lucene API Front-end written in Scala
• NLP4L provides
• Preprocessors for existing ML tools
• Provision of ML algorithms and Applications (e.g. Transliteration)
8
9. Agenda
• What s NLP4L?
• How NLP improves search experience
• Calculate probabilities using ShingleFilter
• Transliteration (Application for HMM)
• NLP4L Framework (coming soon)
9
20. Structured Documents
ID product price gender
1
CURREN New Men s Date Stainless
Steel Military Sport Quartz Wrist Watch
8.92 Men s
2 Suiksilver The Gamer Watch 87.99 Men s
20
21. Unstructured Documents
ID article
1
David Cameron says he has a mandate to pursue EU reform following the
Conservatives' general election victory. The Prime Minister will be hoping his
majority government will give him extra leverage in Brussels.
2
He wants to renegotiate the terms of the UK's membership ahead of a
referendum by the end of 2017. He has said he will campaign for Britain to
remain in the EU if he gets the reforms he wants.
21
22. Make them Structured
I
D
article person org loc
1
David Cameron says he has a mandate to pursue EU reform following
the Conservatives' general election victory. The Prime Minister will be
hoping his majority government will give him extra leverage in Brussels.
David
Cameron
EU
Bruss
els
2
He wants to renegotiate the terms of the UK's membership ahead of a
referendum by the end of 2017. He has said he will campaign for
Britain to remain in the EU if he gets the reforms he wants.
EU
UK
Britai
n
NEE[1] extracts interesting words.
[1] Named Entity Extraction
22
24. Agenda
• What s NLP4L?
• How NLP improves search experience
• Calculate probabilities using ShingleFilter
• Transliteration (Application for HMM)
• NLP4L Framework (coming soon)
24
25. Language Model
• LM represents the fluency of language
• N-gram model is the LM which is most widely
used
• Calculation example for 2-gram
totalTermFreq(”word2g”,”an apple”)
totalTermFreq(”word”,”an”)
25
26. Alice/NNP ate/VB an/AT apple/NNP ./.
Mike/NNP likes/VB an/AT orange/NNP ./.
An/AT apple/NNP is/VB red/JJ ./.
NNP Proper noun, singular
VB Verb
AT Article
JJ Adjective
. period
Part-of-Speech Tagging
Our Corpus for training
26
33. Agenda
• What s NLP4L?
• How NLP improves search experience
• Calculate probabilities using ShingleFilter
• Transliteration (Application for HMM)
• NLP4L Framework (coming soon)
33
34. Transliteration
Transliteration is a process of transcribing letters or words from one
alphabet to another one to facilitate comprehension and pronunciation
for non-native speakers.
computer コンピューター
server サーバー
internet インターネット
mouse マウス
information インフォメーション
examples of transliteration from English to Japanese
34
38. Demo: Transliteration
Input Prediction Right Answer
アルゴリズム algorism algorithm
プログラム program (OK)
ケミカル chaemmical chemical
ダイニング dining (OK)
コミッター committer (OK)
エントリー entree entry
nlp4l> :load examples/trans_katakana_alpha.scala
38
39. Gathering loan words
① crawl
gathering
Katakana-Alphabet
string pairs
アルゴリズム, algorithm
Transliteration
アルゴリズム
algorism
calculate
edit distance
synonyms.txt
store pair of strings
if edit distance
is small enough
②
③
④
⑤
⑥
39
40. Gathering loan words
① crawl
gathering
Katakana-Alphabet
string pairs
アルゴリズム, algorithm
Transliteration
アルゴリズム
algorism
calculate
edit distance
synonyms.txt
store pair of strings
if edit distance
is small enough
②
③
④
⑤
⑥
Got 1,800+ records of
synonym knowledge
from jawiki
40
41. Agenda
• What s NLP4L?
• How NLP improves search experience
• Calculate probabilities using ShingleFilter
• Transliteration (Application for HMM)
• NLP4L Framework (coming soon)
41
42. NLP4L Framework
• A framework that improves search experience (for mainly
Lucene-based search system). Pluggable.
• Reference implementation of plug-ins and corpora
provided.
• Uses NLP/ML technologies to output models, dictionaries
and indexes.
• Since NLP/ML are not perfect, an interface that enables
users to personally examine output dictionaries is
provided as well.
42
43. NLP4L Framework
• A framework that improves search experience (for mainly
Lucene-based search system). Pluggable.
• Reference implementation of plug-ins and corpora
provided.
• Uses NLP/ML technologies to output models, dictionaries
and indexes.
• Since NLP/ML are not perfect, an interface that enables
users to personally examine output dictionaries is
provided as well.
43
44. NLP4L Framework
• A framework that improves search experience (for mainly
Lucene-based search system). Pluggable.
• Reference implementation of plug-ins and corpora
provided.
• Uses NLP/ML technologies to output models, dictionaries
and indexes.
• Since NLP/ML are not perfect, an interface that enables
users to personally examine output dictionaries is
provided as well.
44
45. NLP4L Framework
• A framework that improves search experience (for mainly
Lucene-based search system). Pluggable.
• Reference implementation of plug-ins and corpora
provided.
• Uses NLP/ML technologies to output models, dictionaries
and indexes.
• Since NLP/ML are not perfect, an interface that enables
users to personally examine output dictionaries is
provided as well.
45
46. Solr
ES
Mahout Spark
Data Source
・Corpus (Text data, Lucene index)
・Query Log
・Access Log
Dictionaries
・Suggestion
(auto complete)
・Did you mean?
・synonyms.txt
・userdic.txt
・keyword attachment
maintenance
Model files
Tagged
Corpus
Document
Vectors
・TermExtractor
・Transliteration
・NEE
・Classification
・Document Vectors
・Language Detection
・Learning to Rank
・Personalized Search
46
47. Keyword Attachment
• Keyword attachment is a general format that enables the
following functions.
• Learning to Rank
• Personalized Search
• Named Entity Extraction
• Document Classification
Lucene
doc
Lucene
doc
keyword
↑
Increase boost
47
50. Learning to Rank
• Program learns, from access log and other
sources, that the score of document d for a
query q should be larger than the normal
score(q,d)
Lucene
doc d
q, q, …
https://en.wikipedia.org/wiki/Learning_to_rank
50
53. Personalized Search
• Program learns, from access log and other sources, that
the score of document d for a query q by user u should
be larger than the normal score(q,d)
• Since you cannot specify score(q,d,u) as Lucene restricts
doing so, you have to specify score(qu,d).
• Limit the data to high-order queries or divide fields
depending on a user as the number of q-u combinations
can be enormous.
Lucene
doc d1
q1u1, q2u2
Lucene
doc d2
q2u1, q1u2
53
54. Join and Code with Us!
Contact us at
koji at apache dot org
for the details.
54