1. Full Text Search
David LeBer
Align Software Inc.

2. What is full text search?

4. How?
• Wild card database queries
• Database implementations
• Third party search engines
• Text indexing libraries

5. Wild Card Queries
SELECT FROM 'SOME_TABLE' WHERE 'SOME_COLUMN' LIKE '%Some String%'

6. Wild Card Queries
• Easy

7. Wild Card Queries
• Slow
• Hard to optimize
• Difficult to rank

8. Database Implementations
• MySQL FULLTEXT index and MATCH queries
• PostgreSQL tsvector & tsquery

9. Database Implementations
• Fairly Easy

10. Database Implementations
• Database specific SQL
• May include additional limitations
(i.e: MySQL - MyISAM tables only)
• Functionality define by the DB engine

11. Third Party Search Engines
• Google indexing / searching of your content

12. Third Party Search Engines
• Easy
• Matches user expectations

13. Third Party Search Engines
• Content must be available for indexing
• Loss of control
• Enhances the Google hegemony

14. Text Indexing Library
• Lucene

15. Text Indexing Library
• Complete control
• Database independent
• Flexible search behaviour
• Ranked results

16. Text Indexing Library
• Adds complexity
• Additional query language
• Parallel index

17. Lucene Overview
• Open Source - part of the Apache Project
• Very flexible
• Wickedly fast
• Index based

18. Lucene : Installing
• Add the Lucene jars to your classpath
• Use ERIndexing

19. Lucene : Tasks
• Indexing
• Searching

20. Indexing

21. What is Indexing?

22. Indexing : Steps
• Conversion (to plain text)
• Analysis (clean and convert the text to tokens)
• Index (save the tokens to the index)

23. Indexing : Parts
• Index - either file or memory based
• Document - represents a unique object added to the index
• Field - identifies a chunk of data in the document

24. Indexing : Classes
• IndexWriter
• Directory
• Analyzer
• Document
• Field

25. Creating an Index
URL indexDirectoryURL = ... // assume exists
File indexFile = new File(indexDirectoryURL.getPath());
FSDirectory indexDirectory = FSDirectory.open(indexFile);
StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);
IndexWriter indexWriter = new IndexWriter(index, analyzer, true,
IndexWriter.MaxFieldLength.UNLIMITED);

26. Indexing : Field Parameters
• Stored or not
• Analyzed or not, with and without norms
• Include position, offset, and term frequency

27. Indexing : Analyzers
• SimpleAnalyzer
• StopAnalyzer
• StandardAnalyzer
• ...

28. Adding a Document
String value = ... // assume exists
Document doc = new Document();
Field docField = new Field("title", value,
Field.Store.YES, Field.Index.ANALYZED);
doc.add(docField);
...
indexWriter.addDocument(doc);

29. Indexing : Fun with indexes
• Multiple Access

30. Searching

31. What is Searching

32. Searching : Steps
• Clean the user input
• Create a Query
• Query the Index
• Return the results

33. Searching : Search Classes
• IndexReader
• IndexSearcher
• Query
• QueryParser
• TopDocs/ScoreDocs
• Document

34. Searching : QueryTypes
• TermQuery
• RangeQuery
• PrefixQuery
• BooleanQuery
• PhraseQuery
• WildCardQuery
• FuzzyQuery

35. Searching : QueryParser
• 'webobjects' - contains an exact match - TermQuery
• 'webobjects apple', 'webobjects OR apple' - an OR Query
• +webobjects +apple / webobjects AND apple - an AND Query
• title:webobjects - Contains the term in title field
• title:webobjects -subject:iTunes / title:webobjects AND NOT
subject:iTunes
• (webobjects OR apple) AND iTunes

36. Searching : QueryParser
• title:"apple webobjects" - Phrase Query
• title:"apple webobjects"~5 - slop of 5
• webobj* - Prefix Query
• webobjicts~ - Fuzzy Query
• lastmodified:[1/1/10 TO 1/1/11] - Range Query

37. Performing a Search
Query q = ... // assume exists
IndexSearcher searcher = new IndexSearcher(index, true);
TopScoreDocCollector collector = TopScoreDocCollector.create(10, true);
searcher.search(query, collector);
ScoreDoc[] hits = collector.topDocs().scoreDocs;

38. Using a QueryParser
QueryParser queryParser = new QueryParser(Version.LUCENE_2.9,
"content", analyzer);
Query query = queryParser.parse(queryString);

39. Demo

40. Scoring

41. “The more times a query term appears in a
document relative to the number of times the term
appears in all the documents in the collection, the
more relevant that document is to the query”

42. Boost
• While Indexing
• Document
• Field
• While Searching
• Query

43. Luke

44. Demo

45. ERIndexing

46. ERIndexing : Strengths
• Hides some of the complexity of integrating Lucene with WO
• Offers lots of utility and helper methods
• Speaks WebObjects collection classes
• Simplifies index creation

47. ERIndexing : Weaknesses
• Hides some of the complexity of integrating Lucene with WO
• Not fully baked
• Auto indexing may be dangerous

48. Demo

49. Beyond Lucene
• Solr
• Compass
• ElasticSearch

50. Q&A
Lucene: http://lucene.apache.org
Luke: http://code.google.com/p/luke/
Solr: http://lucene.apache.org/solr/
Compass: http://www.compass-project.org/overview.html
ElasticSearch: http://www.elasticsearch.com/