Search Quality Evaluation: a Developer Perspective
âą
0 gostouâą332 visualizaçÔes
Search quality evaluation is an ever-green topic every search engineer ordinarily struggles with. Improving the correctness and effectiveness of a search system requires a set of tools which help measuring the direction where the system is going. The slides will focus on how a search quality evaluation tool can be seen under a practical developer perspective, how it could be used for producing a deliverable artifact and how it could be integrated within a continuous integration infrastructure.
Recomendados
Recomendados
Mais conteĂșdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a Search Quality Evaluation: a Developer Perspective
Semelhante a Search Quality Evaluation: a Developer Perspective (20)
Mais de Sease
Mais de Sease (20)
Ăltimo
Ăltimo (20)
Search Quality Evaluation: a Developer Perspective
- 1. Search Quality Evaluation a Developer Perspective Andrea Gazzarini - Software Engineer, Sease Ltd 26th June 2018 Apache Lucene/Solr London
- 2. Apache Lucene/Solr London Who I am âȘ SOFTWARE ENGINEER SINCE 1999 âȘ âHERMITâ SOFTWARE ENGINEER SINCE 2010 âȘ JAVA PASSIONATE âȘ INFORMATION RETRIEVAL PASSIONATE âȘ APACHE SOLR / LUCENE ENTHUSIAST âȘ APACHE QPID (PAST) COMMITTER âȘ HUSBAND & FATHER âȘ BASS PLAYER! Andrea Gazzarini VIGNANELLO
- 3. Apache Lucene/Solr London Sease Search Services â Open Source Enthusiasts â Apache Lucene/Solr experts ! Community Contributors â Active Researchers â Hot Trends : Learning To Rank, Document Similarity, Measuring Search Quality, Relevancy Tuning
- 4. Agenda Apache Lucene/Solr London â Search Quality Evaluation ⣠Context overview ⣠Correctness ⣠Evaluation Measures âą Rated Ranking Evaluator (RRE) âą Future Works âą Q&A
- 5. Apache Lucene/Solr London Search Quality Evaluation / Context Overview Search engineering is the production of quality search systems. Search quality (and in general software quality) is a huge topic which can be described using internal and external factors. In the end, only external factors matter, those that can be perceived by users and customers. But the key for getting optimal levels of those external factors are the internal ones. One of the main differences between search and software quality (especially from a correctness perspective) is in the ok / ko judgment, which is more âdeterministicâ in case of software development. Context Overview Search Quality Internal Factors External Factors Correctness Robustness Extendibility Reusability Efficiency Timeliness Modularity Readability Maintainability Testability Maintainability Understandability Reusability âŠ. Focused on Primarily focused on
- 6. Apache Lucene/Solr London Search Quality Evaluation / Correctness Correctness is the ability of a system to perform its exact task, as defined by its specification. Search domain is critical from this perspective because correctness depends on arbitrary user judgments. For each internal (gray) and external (red) iteration we need to find a way to measure the correctness. Evaluation measures for an information retrieval system are used to assert how well the search results satisfied the user's query intent. Correctness Swimlane A: a new system Swimlane B: an existing system Here are the requirements Ok V1.0 has been released Cool! a month later⊠We have a change request. We found a bug We need to improve our search system, users are complaining about junk in search results. Ok v0.1 ⊠v0.9 v1.1 v1.2 v1.3 ⊠v2.0 v2.0 How can we know where our system is going between versions, in terms of correctness, relevancy?
- 7. Apache Lucene/Solr London Search Quality Evaluation / Measures Evaluation measures for an information retrieval system try to formalise how well a search system satisfies its user information needs. Measures are generally split into two categories: online and offline measures. In this context we will focus on offline measures. We will talk about something that can help a search engineer during his ordinary day (i.e. in those phases previously called âinternal iterationsâ) We will also see how the same tool can be used for a broader usage, like contributing in the continuous integration pipeline or even for delivering value to functional stakeholders (i.e. external iterations). Evaluation Measures Evaluation Measures Online Measures Offline Measures Average Precision Mean Reciprocal Rank Recall NDCG Precision Click-through rate F-Measure Zero result rate Session abandonment rate Session success rate âŠ. âŠ. We are mainly focused here, in this talk
- 8. Agenda Apache Lucene/Solr London âą Search Quality Evaluation âRated Ranking Evaluator (RRE) ⣠What is it? ⣠How does it work? ⣠Domain Model ⣠Apache Maven binding ⣠RRE Server âą Future Works âą Q&A
- 9. Apache Lucene/Solr London RRE / What is it? âą A set of search quality evaluation tools âą A search quality evaluation framework âą Multi (search) platform âą Written in Java âą It can be used also in non-Java projects âą Licensed under Apache 2.0 âą Open to contributions âą Extremely dynamic! RRE: What is it? https://github.com/SeaseLtd/rated-ranking-evaluator
- 10. Apache Lucene/Solr London RRE / at a glance 2 ____________________________________________________________________________________________________________________________________________________________________________________________________________________________ Months 2 ____________________________________________________________________________________________________________________________________________________________________________________________________________________________ People 10 ____________________________________________________________________________________________________________________________________________________________________________________________________________________________ modules 48950 ____________________________________________________________________________________________________________________________________________________________________________________________________________________________ lines of code
- 11. Apache Lucene/Solr London RRE / ecosystem The picture illustrates the main modules composing the RRE ecosystem. All modules with a dashed border are planned for a future release. RRE CLI has a double border because although the rre-cli module hasnât been developed, you can run RRE from a command line using RRE Maven archetype, which is part of the current release. As you can see, the system development takes in account two target search platforms: Apache Solr and Elasticsearch. The Search Platform API module provide a search platform abstraction for plugging-in additional search systems. RRE Ecosystem CORE Plugin Plugin Reporting Plugin Search Platform API RequestHandler RRE Server RRE CLI Plugin Plugin Plugin Archetypes
- 12. Apache Lucene/Solr London RRE / Domain Model RRE Domain Model is organized into a composite / tree-like structure where the relationships between entities are always 1 to many. The top level entity is a placeholder representing an evaluation execution. Versioned metrics are computed at query level and then reported, using an aggregation function, at upper levels. The benefit of having a composite structure is clear: we can see a metric value at different levels (e.g. a query, all queries belonging to a query group, all queries belonging to a topic or at corpus level) RRE Domain Model Evaluation Corpus 1..* v1.0 P@10 NDCG AP F-MEASURE âŠ. v1.1 P@10 NDCG AP F-MEASURE âŠ. v1.2 P@10 NDCG AP F-MEASURE âŠ. v1.n P@10 NDCG AP F-MEASURE âŠ. Topic Query Group Query 1..* 1..* 1..* ⊠Top level domain entity Test dataset / collection Information need Query variants Queries
- 13. Apache Lucene/Solr London RRE / Domain Model, an example The domain model provides all the abstractions needed for articulating a complex judgment sets. If, from one side, it is able to capture a complicated and composite ratings model, on the other side there are some cases where such complexity is not needed. Variants (i.e. queries which belong to the same group) could be automatically generated starting from one query. This query could be manually entered, retrieved from the query logs or generated in some other way. Domain Model Evaluation Corpus 1..* Topic Query Group Query 1..* 1..* 1..* âRanking Evaluation Report - created on ⊠bfa_2018_15_5.json Coloured Mini Bar Fridges Black Fridges âą Black mini fridges âą black mini fridge âą black minibar fridge âą mini bar fridge black
- 14. Apache Lucene/Solr London RRE / Process overview Runtime Container RRE Core For each ratings set For each dataset For each topic For each query group For each query Starts the search platform Stops the search platform Creates & configure the index Indexes data For each version Executes query Computes metric 2 3 4 5 6 7 8 9 12 13 1 11 outputs the evaluation data 14 uses the evaluation data 15
- 15. Apache Lucene/Solr London RRE / Output The RRE Core itself is a library, so it outputs its result as a Plain Java object that must be programmatically used. However when wrapped within a runtime container, like the RRE Maven Plugin, the Evaluation object is marshalled in JSON format. Being interoperable, the JSON format can be used by some other component for producing a different kind of output. An example of such usage is the RRE Apache Maven Reporting Plugin which can âą output a spreadsheet âą send the evaluation data to a running RRE Server Evaluation output
- 16. Apache Lucene/Solr London RRE / Available Metrics These are the RRE built-in metrics which can be used out of the box. The most part of them are computed at query level and then aggregated at upper levels. However, compound metrics (e.g. MAP, or GMAP) are not explicitly declared or defined, because the computation doesnât happen at query level. The result of the aggregation executed on the upper levels will automatically produce these metric. For example, the Average Precision computed for Q1, Q2, Q3, Qn becomes the Mean Average Precision at Query Group or Topic levels. Available Metrics Precision Recall Precision at 1 (P@1) Precision at 2 (P@2) Precision at 3 (P@3) Precision at 10 (P@10) Average Precision (AP) Reciprocal Rank Mean Reciprocal Rank Mean Average Precision (MAP) Normalised Discounted Cumulative Gain
- 17. Apache Lucene/Solr London RRE / What we need to provide âȘ Dataset / collection which consists of representative data in your domain. âȘ In general, we could say it should have a reasonable size. âȘ For those functional scenarios where we are managing different entity kinds, RRE allows to provide more than one dataset. âȘ Although formally a dataset is provided in JSON files, the actual content depends on the target search platform. âȘ It CONFIGURATION SETS PROCESSING RATINGS CORPUS âȘ a structured set of judgements (i.e. relevant documents for a given query) âȘ Itâs not a plain list because it is structured on top of the composite RRE domain model âȘ Here we can define all things that compose the RRE domain model: topics, query groups, and queries. âȘ At query group level we can list all documents which are relevant to all queries belonging to that group âȘ For each relevant document we can express a gain, which indicates how much a document is relevant. âȘ In the current implementation we are using a three-level judgement, but this is one thing that most probably will be generalised in future versions: âȘ 1 => marginally relevant âȘ 2 => relevant âȘ 3 => very relevant âȘ configuration instances at a given time. This concept is often captured by introducing âversionsâ âȘ For each version of our system, we assume thereâs a different configuration set. âȘ The actual content of each configuration set depends on the target platform âȘ For example, if we are using Solr, each version would contain one or more core definitions.
- 18. Apache Lucene/Solr London RRE / What we need to provide: corpora An evaluation execution can involve more than one dataset targeting a given search platform. Within RRE, corpus, dataset, collection are synonyms. Each corpus must be located under the corpora configuration folder. It is then referenced in one or more ratings file. The internal format depends on the target search platform. Solr datasets are provided using a plain JSON format (no JSON Update Commands!). Elasticsearch is instead using the pseudo-json bulk format. Corpora
- 19. Apache Lucene/Solr London RRE / What we need to provide: configuration sets RRE encourages a configuration immutability approach. Even for internal iterations, each time we make a relevant change to the current configuration, itâs better to clone it and move forward with a new version. In this way weâll end up having the historical progression of our system, and RRE will be able to make comparisons. The actual content of the configuration sets actually depends on the target search platform. Configuration Sets
- 20. Apache Lucene/Solr London RRE / What we need to provide: ratings Ratings files (i.e. judgments sets) must be located under the âratingsâ configured folder. There must be at least one ratings file (otherwise no evaluation happens) You can define here all the compounding blocks of the RRE domain model: reference dataset, topics, query groups, queries and judgements. Judgments, the most fundamental part of this input, consist of a list of all relevant documents for the owning query group, with a corresponding âgainâ which is the actual relevancy judgment. If a document is in this list, that means it is relevant for the current query. Ratings
- 21. Apache Lucene/Solr London RRE / Where we need to provide At configuration level, RRE needs to know where the following folders are located: âą configuration sets âą corpora âą ratings âą query templates Although the RRE core itself requires these information, when itâs wrapped within a container defaults values are provided. For example, the Maven Plugin assumes the $ {project.dir}/src/etc folder as the parent of all folders above. Folders
- 22. Apache Lucene/Solr London RRE / Query templates For each query (or for each query group) itâs possible to define a query template, which is a kind of query shape containing one or more placeholders. Then, in the ratings file you can reference one of those defined templates and you can provide a value for each placeholder. Templates have been introduced in order to: âą allow a common query management between search platforms âą define complex queries âą define runtime parameters that cannot be statically determined (e.g. filters) Query templates only_q.json filter_by_language.json
- 23. Apache Lucene/Solr London RRE / Maven (Solr | Elasticsearch) Plugin The RRE Apache Maven Plugin is a runtime container which is able to execute the evaluation process within a (Maven) build cycle. As you can see from the picture on the left, all things described in the previous slides can be configured. You donât need to provide all listed parameters, this is just a demonstration example. They have a good defaults that can work in many cases. The plugin can be attached to whatever phase, although usually it is executed in the test, integration-test or install phase. In this way, if you have some custom code (e.g an UpdateRequestProcessor), it will be correctly put in the runtime classpath. Maven Plugin
- 24. Apache Lucene/Solr London RRE / Maven Reporting Plugin The RRE Maven Plugin produces its output in JSON format, which is interoperable but not so human- readable. The RRE Maven Reporting Plugin can be used for transforming such output in different ways. The plugin is configured within the pom.xml following the standard procedures. The output formats can be configured. Allowed values, at time of writing, are: âą rre-server: sends the evaluation data to a running RRE server instance âą spreadsheet: produces an xls files Reporting Plugin
- 25. Apache Lucene/Solr London RRE / Maven (Solr | Elasticsearch) Archetype Very useful if âą youâre starting from scratch âą you donât use Java as main programming language. The RRE Maven archetype generates a Maven project skeleton with all required folders and configuration. In each folder thereâs a README and a sample content. The skeleton can be used as a basis for your Java project. The skeleton can be used as it is, just for running the quality evaluation (and in this case your main project could be somewhere else) Maven Archetype > mvn archetype:generate ⊠[INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Maven Stub Project (No POM) 1 [INFO] ------------------------------------------------------------------------ [INFO] âŠâŠâŠ [INFO] BUILD SUCCESS
- 26. Apache Lucene/Solr London RRE / Output, the âspreadsheetâ format After the build cycle, a rre-report.xls is generated under the target/rre folder. Spreadsheet format
- 27. Apache Lucene/Solr London RRE / Output, RRE Server (1/2) The RRE console is a simple SpringBoot application which starts a web server. It provides real-time information about evaluation results. Each time a build happens, the RRE reporting plugin sends the evaluation result to a RESTFul endpoint provided by RRE Server. The web console is an AngularJS app which gets refreshed with that incoming data. Useful during the development / tuning phase iterations (as you donât have to open and open again an excel file) RRE Server
- 28. Apache Lucene/Solr London RRE / Output, RRE Server (2/2) The evaluation data, at query / version level, collects the top n search results. In the web console, under each query, thereâs a little arrow which allows to open / hide the section which contains those results. In this way you can get immediately the meaning of each metric and its values between different versions. In the example above, you can immediately see why thereâs a loss of precision (first metric) between v1.0, v1.1, which got fixed in v1.2
- 29. Agenda Apache Lucene/Solr London âą Search Quality Evaluation âą Rated Ranking Evaluator â Future Works âą Q&A
- 30. Apache Lucene/Solr London Future Works / Building the Input The main input for RRE is the Ratings file, in JSON format. Writing a comprehensive JSON to detail the ratings sets for your Search ecosystem can be expensive! 1. Explicit feedback from users judgements 2. An intuitive UI allow judges to run queries, see documents and rate them 3. Relevance label is explicitly assigned by domain experts 1. Implicit feedback from users interactions (Clicks, Sales âŠ) 2. Log to disk / internal Solr instance for analytics 3. Estimate <q,d> relevance label based on Click Through Rate, Sales Rate Users Interactions Logger Judgement Collector UI Quality Metrics Ratings Set Interactions Logger Judgements Collectorâš Explicit Feedback Implicit Feedback RRE
- 31. Apache Lucene/Solr London Future Works / Jenkins Plugin RRE Maven plugin already produces the evaluation data in a machine-readable format (JSON) which can be consumed by another component. The Maven RRE Report plugin or the RRE Server are just two examples of such consumers. RRE can be already integrated in a Jenkins CI build cycle. By means of a dedicated Jenkins plugin, the evaluation data will be graphically displayed in the Jenkins dashboard. Jenkins Plugin
- 32. Apache Lucene/Solr London Future Works / Solr Rank Eval API The RRE core will be used for implementing a RequestHandler which will be able to expose a Ranking Evaluation endpoint. That would result in the same functionality introduced in Elasticsearch 6.2 [1] with some differences. âą rich tree data model âą metrics framework Here it doesnât make so much sense to provide comparisons between versions. As part of the same module we will have a SearchComponent, for evaluating a single query interaction. [1] https://www.elastic.co/guide/en/elasticsearch/reference/6.2/search-rank-eval.html Rank Eval API /rank_eval ?q=something&evaluate=true + RRE RequestHandler + RRE SearchComponent
- 33. Agenda Apache Lucene/Solr London âą Search Quality Evaluation âą Rated Ranking Evaluator âą Future Works â Q&A
- 34. Search Quality Evaluation a Developer Perspective Andrea Gazzarini - Software Engineer, Sease Ltd 26th June 2018 Apache Lucene/Solr London Thank you!