Generative AI - Gitex v1Generative AI - Gitex v1.pptx
An Introduction to NLP4L - Natural Language Processing Tool for Apache Lucene: Presented by Tomoko Uchida, Rondhuit Co. Ltd.
1. O C T O B E R 1 3 - 1 6 , 2 0 1 6 • A U S T I N , T X
2. An Introduction to NLP4L:
Natural Language Processing Tool for Apache Lucene
Tomoko Uchida
Consultant, Rondhuit Co. Ltd.
3. 3
Who am I
• Tomoko Uchida (@moco_beta)
• Luke (Lucene Toolbox) collaborator (2015 ~)
• https://github.com/DmitryKey/luke
• The best-known tool for debugging and
learning Lucene/Solr, Elasticsearch index :-)
4. 4
Agenda
• What’s NLP4L?
• How NLP improves search experience
• Calculate probabilities using ShingleFilter
• Transliteration (Application for HMM)
• NLP4L Framework (coming soon)
5. 5
Agenda
• What’s NLP4L?
• How NLP improves search experience
• Calculate probabilities using ShingleFilter
• Transliteration (Application for HMM)
• NLP4L Framework (coming soon)
6. 6
What’s NLP4L?
• GOAL
• Improve Lucene users’ search experience
• FEATURES
• Use of Lucene index as a Corpus Database
• Lucene API Front-end written in Scala
• NLP4L provides
• Preprocessors for existing ML tools
• Provision of ML algorithms and Apprications (e.g. Transliteration)
7. 7
What’s NLP4L?
• GOAL
• Improve Lucene users’ search experience
• FEATURES
• Use of Lucene index as a Corpus Database
• Lucene API Front-end written in Scala
• NLP4L provides
• Preprocessors for existing ML tools
• Provision of ML algorithms and Apprications (e.g. Transliteration)
8. 8
What’s NLP4L?
• GOAL
• Improve Lucene users’ search experience
• FEATURES
• Use of Lucene index as a Corpus Database
• Lucene API Front-end written in Scala
• NLP4L provides
• Preprocessors for existing ML tools
• Provision of ML algorithms and Applications (e.g. Transliteration)
9. 9
Agenda
• What’s NLP4L?
• How NLP improves search experience
• Calculate probabilities using ShingleFilter
• Transliteration (Application for HMM)
• NLP4L Framework (coming soon)
23. ID product price gender
1
CURREN New Men s Date Stainless Steel Military Sport
Quartz Wrist Watch
8.92 Men s
2 Suiksilver The Gamer Watch 87.99 Men s
23
Structured Documents
24. ID article
1
David Cameron says he has a mandate to pursue EU reform following the Conservatives' general election victory.
The Prime Minister will be hoping his majority government will give him extra leverage in Brussels.
2
He wants to renegotiate the terms of the UK's membership ahead of a referendum by the end of 2017. He has said
he will campaign for Britain to remain in the EU if he gets the reforms he wants.
24
Unstructured Documents
25. I
D
article person org loc
1
David Cameron says he has a mandate to pursue EU reform following the Conservatives' general
election victory. The Prime Minister will be hoping his majority government will give him extra
leverage in Brussels.
David Cameron EU Brussels
2
He wants to renegotiate the terms of the UK's membership ahead of a referendum by the end of
2017. He has said he will campaign for Britain to remain in the EU if he gets the reforms he wants.
EU
UK
Britain
NEE[1] extracts interesting words.
[1] Named Entity Extraction
25
Make them Structured
26. 26
Agenda
• What’s NLP4L?
• How NLP improves search experience
• Calculate probabilities using ShingleFilter
• Transliteration (Application for HMM)
• NLP4L Framework (coming soon)
29. • LM represents the fluency of language
• N-gram model is the LM which is most widely
used
29
Language Model
30. • LM represents the fluency of language
• N-gram model is the LM which is most widely
used
• Calculation example for 2-gram
30
totalTermFreq(”word2g”,”an apple”)
totalTermFreq(”word”,”an”)
Language Model
31. Alice/NNP ate/VB an/AT apple/NNP ./.
Mike/NNP likes/VB an/AT orange/NNP ./.
An/AT apple/NNP is/VB red/JJ ./.
NNP Proper noun, singular
VB Verb
AT Article
JJ Adjective
. period
31
Our Corpus for training
Part-of-Speech Tagging
38. 38
Agenda
• What’s NLP4L?
• How NLP improves search experience
• Calculate probabilities using ShingleFilter
• Transliteration (Application for HMM)
• NLP4L Framework (coming soon)
39. 39
Transliteration
Transliteration is a process of transcribing letters
or words from one alphabet to another one to
facilitate comprehension and pronunciation for
non-native speakers.
40. computer コンピューター
server サーバー
internet インターネット
mouse マウス
information インフォメーション
examples of transliteration from English to Japanese
40
Transliteration
45. nlp4l> :load examples/trans_katakana_alpha.scala
45
Demo: Transliteration
val indexer = new HmmModelIndexer(index)
val file = Source.fromFile("train_data/alpha_katakana_aligned.txt", "UTF-8")
val pattern: Regex = """([u30A0-u30FF]+)([a-zA-Z]+)(.*)""".r
def align(result: List[(String, String)], str: String): List[(String, String)] = {
str match {
case pattern(a, b, c) =>
align(result :+ (a, b), c)
case _ =>
result
}
}
// create hmm model index
file.getLines.foreach{ line: String =>
val doc = align(List.empty[(String, String)], line)
indexer.addDocument(doc)
}
46. Input Prediction Right Answer
アルゴリズム algorism algorithm
プログラム program (OK)
ケミカル chaemmical chemical
ダイニング dining (OK)
コミッター committer (OK)
エントリー entree entry
46
Demo: Transliteration
47. ① crawl
gathering
Katakana-Alphabet
string pairs
アルゴリズム, algorithm
Transliteration
アルゴリズム
algorism
calculate
edit distance
synonyms.txt
47
store pair of strings
if edit distance
is small enough
②
③
④
⑤
⑥
Gathering loan words
48. 48
Agenda
• What’s NLP4L?
• How NLP improves search experience
• Calculate probabilities using ShingleFilter
• Transliteration (Application for HMM)
• NLP4L Framework (coming soon)
49. 49
NLP4L Framework
• A framework that improves search experience (for mainly Lucene-
based search system). Pluggable.
• Reference implementation of plug-ins and corpora provided.
• Uses NLP/ML technologies to output models, dictionaries and
indexes.
• Since NLP/ML are not perfect, an interface that enables users to
personally examine output dictionaries is provided as well.
50. 50
NLP4L Framework
• A framework that improves search experience (for mainly Lucene-
based search system). Pluggable.
• Reference implementation of plug-ins and corpora provided.
• Uses NLP/ML technologies to output models, dictionaries and
indexes.
• Since NLP/ML are not perfect, an interface that enables users to
personally examine output dictionaries is provided as well.
51. 51
NLP4L Framework
• A framework that improves search experience (for mainly Lucene-
based search system). Pluggable.
• Reference implementation of plug-ins and corpora provided.
• Uses NLP/ML technologies to output models, dictionaries and
indexes.
• Since NLP/ML are not perfect, an interface that enables users to
personally examine output dictionaries is provided as well.
52. 52
NLP4L Framework
• A framework that improves search experience (for mainly Lucene-
based search system). Pluggable.
• Reference implementation of plug-ins and corpora provided.
• Uses NLP/ML technologies to output models, dictionaries and
indexes.
• Since NLP/ML are not perfect, an interface that enables users to
personally examine output dictionaries is provided as well.
53. 53
Solr
ES
Mahout Spark
Data Source
・Corpus (Text data, Lucene index)
・Query Log
・Access Log
Dictionaries
・Suggestion
(auto complete)
・Did you mean?
・synonyms.txt
・userdic.txt
・keyword attachment
maintenance
Model files
Tagged
Corpus
Document
Vectors
・TermExtractor
・Transliteration
・NEE
・Classification
・Document Vectors
・Language Detection
・Learning to Rank
・Personalized Search
54. 54
Solr
ES
Mahout Spark
Data Source
・Corpus (Text data, Lucene index)
・Query Log
・Access Log
Dictionaries
・Suggestion
(auto complete)
・Did you mean?
・synonyms.txt
・userdic.txt
・keyword attachment
maintenance
Model files
Tagged
Corpus
Document
Vectors
・TermExtractor
・Transliteration
・NEE
・Classification
・Document Vectors
・Language Detection
・Learning to Rank
・Personalized Search
55. 55
Solr
ES
Data Source
・Corpus (Text data, Lucene index)
・Query Log
・Access Log
Dictionaries
・Suggestion
(auto complete)
・Did you mean?
・synonyms.txt
・userdic.txt
・keyword attachment
maintenance
・TermExtractor
・Transliteration
・NEE
・Classification
・Document Vectors
・Language Detection
・Learning to Rank
・Personalized Search
Mahout Spark
Model files
Document
Vectors
Tagged
Corpus
56. 56
Mahout Spark
Data Source
・Corpus (Text data, Lucene index)
・Query Log
・Access Log
Dictionaries
・Suggestion
(auto complete)
・Did you mean?
・synonyms.txt
・userdic.txt
・keyword attachment
maintenance
Model files
Tagged
Corpus
Document
Vectors
・TermExtractor
・Transliteration
・NEE
・Classification
・Document Vectors
・Language Detection
・Learning to Rank
・Personalized Search
Solr
ES
57. 57
example: Keyword Attachment
Information about associated
Solr collection (core)
NLP/ML task (processor) chain
described by HOCON (Human-
Optimized Config Object Notation)
UI prototype for NLP4L Framework (Lucia)
https://github.com/NLP4L/lucia
64. 64
example: Keyword Attachment
If you delete keyword(s) already have been
attached to solr documents,
the keyword(s) also will be removed from solr
index when next “synch” action executed.
68. • Program learns, from access log and other
sources, that the score of document d for a
query q should be larger than the normal
score(q,d)
68
Lucene
doc d
q, q, …
https://en.wikipedia.org/wiki/Learning_to_rank
Learning to Rank
71. 71
Lucene
doc d1
q1u1, q2u2
Lucene
doc d2
q2u1, q1u2
Personalized Search
• Program learns, from access log and other sources, that the score of
document d for a query q by user u should be larger than the
normal score(q,d)
• Since you cannot specify score(q,d,u) as Lucene restricts doing so,
you have to specify score(qu,d).
• Limit the data to high-order queries or divide fields depending on a
user as the number of q-u combinations can be enormous.
72. 72
example: Generating Synonyms (loanwords)
Execute job that generate pairs of Katakana and
corresponding English words from corpus