1. Lucenefromtheory to real world Information retrieval Indexing Cluster Apache Performance tuning Parser Dictionary IndexReader Solr Real world Java Analysis Troubleshooting Vector Relevance Query Server Design Fields Document Probabilistic Production Model Search application Open Source Inverted index Doug Cutting Library Architecture

2. Agenda Introduction to Information Retrieval Luceneoverview Lucene in details Search applications design Performance tuning www.xebia.fr / blog.xebia.fr 2

3. www.xebia.fr / blog.xebia.fr 3 Information Retrieval

4. www.xebia.fr / blog.xebia.fr 4 Information Retrieval “ Information Retrieval (IR) is the science of searching for document ”

5. 5

6. www.xebia.fr / blog.xebia.fr 6 Inverted Index

9. Similaritycanbecomputedwith :dj = (w1,j,w2,j,...,wt,j) q = (w1,q,w2,q,...,wt,q)

10. www.xebia.fr / blog.xebia.fr 9 Lucene

11. www.xebia.fr / blog.xebia.fr 10 Lucene : where do we come from ?

12. www.xebia.fr / blog.xebia.fr 11 Lucene documentation

13. www.xebia.fr / blog.xebia.fr 12 Lucene : Simple indexingexample Directorydirectory= new RAMDirectory(); IndexWriter writer = new IndexWriter(directory, new WhitespaceAnalyzer(), IndexWriter.MaxFieldLength.UNLIMITED); Documentdoc = new Document(); doc.add(new Field(“company”, “Xebia”, Field.Store.YES, Field.Index.NOT_ANALYZED)); doc.add(new Field(“country”, “France”, Field.Store.YES, Field.Index.NO)); writer.addDocument(doc); writer.close();

14. www.xebia.fr / blog.xebia.fr 13 Lucene : Simplesearchexample IndexSearcher searcher = new IndexSearcher(dir, true); Termt = new Term(“country”, “France”); Queryquery = new TermQuery(t); TopDocs docs = searcher.search(query, 10); assertEquals(1, docs.totalHits); searcher.close();

15. www.xebia.fr / blog.xebia.fr 14 Lucene - indexing

16. www.xebia.fr / blog.xebia.fr 15 Lucene - analyzers

18. Index : NO / ANALYZED / NOT_ANALYZED / ANALYZED_NO_NORMS / NOT_ANALYZED_NO_NORMS

22. Or automaticallymergeddepending on : (int) log(max(minMergeMB, size))/log(mergeFactor)

23. www.xebia.fr / blog.xebia.fr 20 Lucene - search

25. TermQuery

26. PhraseQuery

27. WildcardQuery

28. RangeQuery

29. FuzzyQuery

31. 3 layers :

32. QueryParser : Transforms a query string into an Abstract SyntaxTreerepresentation

33. QueryNodeProcessor : Processesnodes of the tree to move, remove or modifythem

36. PhraseQuery : SpanQuerywith a slop value of 0

38. www.xebia.fr / blog.xebia.fr 27 Inverted Index

42. Can befullycustomized

43. By default limited to 50 KB characters

45. Usedinternally by Sortobjects

46. Can beused to manuallyload values of a single field :float[] weights = FieldCache.DEFAULT.getFloats(reader, “weight”);

48. Produces a query to besearchedMoreLikeThismlt = new MoreLikeThis(reader); mlt.setFieldNames(new String[] {"title", "author"}); mlt.setMinTermFreq(1); mlt.setMinDocFreq(1); Queryquery = mlt.like(docId); indexSearcher.search(query, 10);

50. ConsiderusingFieldCaches to ReducefetchingcostFieldScoreQueryscoreQuery = new FieldScoreQuery("score", FieldScoreQuery.Type.BYTE); CustomScoreQuerycustomQ = new CustomScoreQuery(q, scoreQuery ) { public floatcustomScore(int doc, floatsubQueryScore, floatvalSrcScore) { return (float) (Math.sqrt(subQueryScore) * valSrcScore); } };

51. www.xebia.fr / blog.xebia.fr 34 Lucene – Luke

53. ConsiderusingRAMDirectory / InstanciatedIndex

54. Uses latest version of Lucene

55. Uses NIODirectory for Unix and MMAPDirectory for Windows

58. WarmupFieldCache to ensureimmediateaccesswhensorting

59. Limit use of TermVector

61. www.xebia.fr / blog.xebia.fr 39 Architecture withSolr

62. www.xebia.fr / blog.xebia.fr 40 Architecture withInfinispan

65. Apache Compass : Search engine framework

66. Apache Solr : Lucenestandalonesearch server

67. Apache Mahout : Distributed machine learning

68. HibernateSearch : Hibernate + Lucene