O slideshow foi denunciado.

Rated Ranking Evaluator Enterprise: the next generation of free Search Quality Evaluation Tools

0

Compartilhar

Carregando em…3
×
1 de 48
1 de 48

Rated Ranking Evaluator Enterprise: the next generation of free Search Quality Evaluation Tools

0

Compartilhar

Baixar para ler offline

RRE is an open-source search quality evaluation tool that can be used to produce a set of reports about the quality of a system, iteration after iteration, and that can be integrated within a continuous integration infrastructure to monitor quality metrics after each release.

Many aspects remained problematic though:

– how to directly evaluate a middle layer search-API that communicates with Apache Solr or Elasticsearch?
– how to easily generate explicit and implicit ratings without spending hours on tedious json files?
– how to better explore the evaluation results? with nice widgets and interesting insights?

Rated Ranking Evaluator Enterprise solves these problems and much more.

Join us as we introduce the next generation of open-source search quality evaluation tools, exploring the internals and real-world scenarios!

RRE is an open-source search quality evaluation tool that can be used to produce a set of reports about the quality of a system, iteration after iteration, and that can be integrated within a continuous integration infrastructure to monitor quality metrics after each release.

Many aspects remained problematic though:

– how to directly evaluate a middle layer search-API that communicates with Apache Solr or Elasticsearch?
– how to easily generate explicit and implicit ratings without spending hours on tedious json files?
– how to better explore the evaluation results? with nice widgets and interesting insights?

Rated Ranking Evaluator Enterprise solves these problems and much more.

Join us as we introduce the next generation of open-source search quality evaluation tools, exploring the internals and real-world scenarios!

Mais Conteúdo rRelacionado

Audiolivros relacionados

Gratuito durante 14 dias do Scribd

Ver tudo

Rated Ranking Evaluator Enterprise: the next generation of free Search Quality Evaluation Tools

  1. 1. Rated Ranking Evaluator Enterprise: the Next Generation of Free Search Quality Evaluation Tools
 
 Alessandro Benedetti, Director Andrea Gazzarini, Co-Founder 15th September 2021
  2. 2. ‣ Born in Tarquinia(ancient Etruscan city) ‣ R&D Software Engineer ‣ Director ‣ Master in Computer Science ‣ Apache Lucene/Solr PMC member/committer ‣ Elasticsearch expert ‣ Semantic, NLP, Machine Learning technologies passionate ‣ Beach Volleyball player and Snowboarder Who We Are Alessandro Benedetti
  3. 3. ‣ Born in Viterbo ‣ Hermit Software Engineer ‣ Master in Economy ‣ Programming Passionate ‣ RRE Creator ‣ Apache Lucene/Solr Expert ‣ Elasticsearch Expert ‣ Apache Qpid Committer ‣ Father, Husband ‣ Bass player, aspiring (still frustrated at the moment) Chapman Stick player Who We Are Andrea Gazzarini
  4. 4. ‣ Headquarter in London/distributed ‣ Open Source Enthusiasts ‣ Apache Lucene/Solr experts ‣ Elasticsearch experts ‣ Community Contributors ‣ Active Researchers ‣ Hot Trends : Learning To Rank, Document Similarity, Search Quality Evaluation, Relevancy Tuning Search Services www.sease.io
  5. 5. Clients
  6. 6. Agenda RRE Open Source RRE Enterprise (RREE) Query Discovery Rating Generation Explore Evaluation Results
  7. 7. Agenda RRE Open Source RRE Enterprise (RREE) Query Discovery Rating Generation Explore Evaluation Results
  8. 8. ‣ Open Source library for Search Quality Evaluation ‣ Ratings are expected in input (Json supported) ‣ Many offline metrics available out-of-the-box (Precision@k, NDCG@k, F-Measure…) ‣ Apache Solr and Elasticsearch support ‣ Development-centric approach ‣ Evaluation on the fly and results in various formats ‣ Community building up! ‣ RRE-User mailing list: https://groups.google.com/g/rre-user ‣ https://github.com/SeaseLtd/rated-ranking-evaluator Rated Ranking Evaluator : RRE Open Source What is it ?
  9. 9. Rated Ranking Evaluator : The Genesis 2018 Search Consultancy Project A customer explicitly asked for a rudimental search quality evaluation tool while we were working on its search infrastructure. Jun Search Quality Evaluation A Developer Perspective 0.9 Search Quality Evaluation Tools And Techniques Oct 1.0 to be continued... mumble mumble
  10. 10. Rated Ranking Evaluator: The Idea RRE has been thought as a development tool that executes search quality evaluations as part of a project build process. It’s like a read–eval–print loop (REPL) tied on top of an Information Retrieval subsystem that encourages an incremental / iterative approach. The underlying idea New system Existing system Here are the requirements Ok V1.0 has been released Cool! a month later… We have a change request. We found a bug We need to improve our search system, users are complaining about junk in search results. Ok v0.1 … v0.9 v1.1 v1.2 v1.3 … v2.0 v2.0 In terms of retrieval effectiveness, how can we know the system performance across various versions?
  11. 11. Rated Ranking Evaluator: Domain Model RRE Domain Model is organized into a composite / tree-like structure where the relationships between entities are always 1 to many. The top level entity is a placeholder representing an evaluation execution. Versioned metrics are computed at query level and then reported, using an aggregation function, at upper levels. The benefit of having a composite structure is clear: we can see a metric value at different levels (e.g. a query, all queries belonging to a query group, all queries belonging to a topic or at corpus level) Domain Model Evaluation Corpus 1..* v1.0 P@10 NDCG AP F-MEASURE …. v1.1 P@10 NDCG AP F-MEASURE …. v1.2 P@10 NDCG AP F-MEASURE …. v1.n P@10 NDCG AP F-MEASURE …. Topic Query Group Query 1..* 1..* 1..* … Top level domain entity Test dataset / collection Information need Query variants Queries
  12. 12. Rated Ranking Evaluator : How it works Data Configuration Ratings Search Platform uses a produces Evaluation Data INPUT LAYER EVALUATION LAYER OUTPUT LAYER JSON RRE Console … used for generating
  13. 13. Explainability - Why is important in Information Retrieval? Dev, tune & Build Check evaluation results We are thinking about how to fill a third monitor
  14. 14. Agenda RRE Open Source RRE Enterprise (RREE) Query Discovery Rating Generation Explore Evaluation Results
  15. 15. RRE Enterprise : The Genesis 2018 Search Consultancy Project A customer explicitly asked for a rudimental search quality evaluation tool while we were working on its search infrastructure. 2019 Rated Ranking Evaluator An Open Source Approach to be continued... The first sketches depicting an idea about an enterprise-level version of RRE. The development starts few months later.
  16. 16. Rated Ranking Evaluator : The Genesis 2018 Search Consultancy Project A customer explicitly asked for a rudimental search quality evaluation tool while we were working on its search infrastructure. 2019 to be continued... 2020 2021 JBCP ID Discovery Query Discovery
  17. 17. Agenda RRE Open Source RRE Enterprise (RREE) Query Discovery Rating Generation Explore Evaluation Results
  18. 18. RRE Open Source Recap: How it works (½) Data Configuration Ratings targets produces Evaluation Data INPUT LAYER EVALUATION LAYER OUTPUT LAYER JSON RRE Console … used for generating OR
  19. 19. RRE Open Source Recap: How it works (2/2) Data Configuration Ratings targets produces Evaluation Data INPUT LAYER EVALUATION LAYER OUTPUT LAYER JSON RRE Console … used for generating OR
  20. 20. RRE Enterprise: Query Discovery? API Problem: I have an intermediate Search-API that builds complex Apache Solr/Elasticsearch queries
  21. 21. RRE Open Source: No Query Discovery targets OR 1 Evaluation Data produces 2
  22. 22. Query Discovery RRE Enterprise: Query Discovery & Evaluation 2 Evaluation Data produces 3 targets OR {SEARCH API} 1 correlates on targets
  23. 23. Input Rating: RRE Open Source vs RREE RRE Enterprise RRE Open Source Topic Query Group Query Information need Query variant (Search Engine) Query + 3 Rated Document Topic Query Group API Request Information need Query variant (Search API) Request + 3 Rated Document Query (Search Engine) Query
  24. 24. Agenda RRE Open Source RRE Enterprise (RREE) Query Discovery Rating Generation Explore Evaluation Results
  25. 25. RRE Enterprise: Rating Generation A fundamental requirement of offline search quality evaluation is to gather <query,document,rating> triples that represent the relevance(rating) of a document given a user information need(query). Before assessing the retrieval effectiveness of a system it is necessary to associate a relevance rating to each pair <query, document> involved in our evaluation.
  26. 26. RRE Enterprise: Explicit Ratings (1/2) • Explicitly provided by domain experts • High accuracy • High effort / time / resources • RRE Open Source accepts only explicit ratings Explicit Ratings RRE Rating Structure Topic Query Group Query Information need Query variant Query + 3 Rated Document Question: how to minimize the relevant effort required to provide explicit ratings?
  27. 27. RRE Enterprise: Explicit Ratings (2/2) • Chrome plugin which applies an evaluation layer on top of an arbitrary website • Ratings are generated using directly the customer website • Users Lowest learning curve • Generated ratings are sent to RREE through a specific endpoint • An “ID Discovery” component translates the received data (rated web items) in RRE Ratings (rated Solr/Elasticsearch documents) Judgment Collector
  28. 28. Implicit Feedback - Click { "collection": "papers", "query": "interleaving", "blackBoxQueryRequest": "GET /query?q=interleaving HTTP/1.1rnHost: localhost:5063", "documentId": "1", "click": 1, "timestamp": "2021-06-23T16:10:49Z", "queryDocumentPosition": 0 } Click http://vps-933d20b7.vps.ovh.net:8080/1.0 /rre-enterprise-api/input-api/interaction Search Result Page
  29. 29. Implicit Feedback - Add To Cart { "collection": "papers", "query": "interleaving", "blackBoxQueryRequest": "GET /query?q=interleaving HTTP/1.1rnHost: localhost:5063", "documentId": "1", "addToCart": 1, "timestamp": "2021-06-23T16:10:49Z", "queryDocumentPosition": 0 } Add To Cart http://vps-933d20b7.vps.ovh.net:8080/1.0 /rre-enterprise-api/input-api/interaction Search Result Page
  30. 30. Implicit Feedback - Sale { "collection": "papers", "query": "interleaving", "blackBoxQueryRequest": "GET /query?q=interleaving HTTP/1.1rnHost: localhost:5063", "documentId": "1", "sale": 1, "timestamp": "2021-06-23T16:10:49Z", "queryDocumentPosition": 0 } Sale http://vps-933d20b7.vps.ovh.net:8080/1.0 /rre-enterprise-api/input-api/interaction Search Result Page
  31. 31. Implicit Feedback - Revenue { "collection": "papers", "query": "interleaving", "blackBoxQueryRequest": "GET /query?q=interleaving HTTP/1.1rnHost: localhost:5063", "documentId": "1", "revenue": 100, "timestamp": "2021-06-23T16:10:49Z", "queryDocumentPosition": 0 } Revenue http://vps-933d20b7.vps.ovh.net:8080/1.0 /rre-enterprise-api/input-api/interaction Search Result Page
  32. 32. Implicit Feedback - Storage { "collection": "papers", "query": "interleaving", "blackBoxQueryRequest": "GET /query?q=interleaving HTTP/1.1rnHost: localhost:5063", "documentId": "1", "revenue": 100, "timestamp": "2021-06-23T16:10:49Z", "queryDocumentPosition": 0 } http://vps-933d20b7.vps.ovh.net:8080/1.0 /rre-enterprise-api/input-api/interaction Interactions
  33. 33. Implicit Feedback - Calculate Online Metrics Interactions <query,document>
  34. 34. Implicit Feedback - Estimate Relevance Global Max: 1 Local Max: [0...1] Global Min: 0 Local Min: [0...1] Simple Click Model 0 1 2 3 4 Rating Metric Score 0 1 0.5
  35. 35. Agenda RRE Open Source RRE Enterprise (RREE) Query Discovery Rating Generation Explore Evaluation Results
  36. 36. ‣ UI using React library ‣ Configuration support ‣ Navigation of Evaluation Results ‣ Overview - quick view for Business Stakeholders ‣ Explore (an evaluation) - deep view for Software Engineers ‣ Compare (two evaluations) - deep comparison for Software Engineers Explore Evaluation Results: UI
  37. 37. Evaluation Results - Overview https://rre-enterprise.netlify.app/overview
  38. 38. Evaluation Results - Overview Expanded
  39. 39. Evaluation Results - Overview Zoom
  40. 40. Evaluation Results - Explore
  41. 41. Evaluation Results - Explore Query Info
  42. 42. Evaluation Results - Explore Query Info 2
  43. 43. Evaluation Results - Compare (1/3)
  44. 44. Evaluation Results - Compare (2/3)
  45. 45. Evaluation Results - Compare (3/3)
  46. 46. Evaluation Results - Compare Query Info
  47. 47. ‣ Release with free usage plan ‣ Configuration support ‣ Support for multimedia document properties ‣ Intelligent insights on weak performing queries, groups, topics ‣ Improvements on Click Modelling for Implicit Relevance estimation RRE Enterprise: Future Work
  48. 48. Thank You!

×