Relevance denotes how well a search result satisfies the user information need. In addition to the search engine components (i.e., indexer and query parser), there are many other components that impact relevance. e.g., user understanding , data optimization, domain knowledge, etc. Improving relevance remains the main and most challenging goal of each search engine. Indeed, relevance can be subjective, therefore hard to measure and to improve. In this talk, Saïd will demystify the concept of relevance by defining its main components. For each component, he will present the technology enablers, the data, and processes that are required in order to measure and improve relevance. In this talk, attendees will learn how to provide a relevant user experience and track it over time.
Call Girls Begur Just Call 👗 7737669865 👗 Top Class Call Girl Service Bangalore
Anatomy of Search Relevance: From Data To Action
1. Anatomy Of Relevance: From Data To Action
Saïd Radhouani, Ph.D.
radhouani
Yellow Pages, Canada
2. Agenda
• Yellow Pages & Myself
• The importance of relevance for our business
• Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion
3. Who we are
• Leading media and marketing solutions company in Canada
• Champion the neighbourhood economy by connecting businesses
and consumers
300 000
Local and national advertisers
7.3 million
Unduplicated unique visitors to our network of websites
6.5 million
Downloads of our mobile apps
26%
Internet users reached in Canada each month
4. Who I am
• Entrepreneur
• Ph.D. in Search and Knowledge Management
• IT Director – Content, Search and Relevance
• Noble cause: leverage data to help build smarter organizations
5. Agenda
• Yellow Pages & Myself
• The importance of relevance for our business
• Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion
8. Agenda
• Yellow Pages & Myself
• The importance of relevance for our business
• Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion
9. User
Objective
Understand the user intent to maximize their satisfaction
Data
• Who the user is (individual or group)
• Where they come from (direct, SEs, …)
• Their context (Location, TOD/DOW/Season, Device, …)
• Their interactions history
KPIs
• Coverage: %tracked user, %tracked user interactions
• Freshness: Daily updates of User profiles & interactions
User
10. Content
Objective
Have accurate and “fresh” content to satisfy your audience needs
Data
• Enriched Directory Content (Merchants, Products, Deals, etc.)
• Rich Editorial Content (Neighborhoods, Smart Tips, etc.)
• Third-party content (social networks, etc. )
• User-Generated Content
KPIs
• Coverage: %Canadian businesses, %Neighborhoods
• Completeness: e.g., for each merchant, have all required info
• Freshness: Frequency of validation & updates Content
User
11. Knowledge
Objective
Help to bridge the gap between user queries and content data
Data
• Linguistic (synonyms, acronyms, etc.)
• Rich and “fresh” geo data (location names, polygons, etc.)
• Multilingual
KPIs
• Coverage: %searched entities (keywords, location names, etc.)
• Completeness: e.g., for each location: centroid, polygon,
synonyms, relationships, etc.
• Freshness: Frequency of validation & updates
Content
Knowledge
User
12. Search
Objective
Action-driven, personalized and contextualized
relevant search experience
KPIs
Offline
• Recall & Precision
Online
• #returned results
• Mean Reciprocal Rank (MRR) – the multiplicative inverse of
the rank of the first correct answer. Perfect SE should have a
MRR of 1
Content
Knowledge
Search
User
13. Presentation
Objective
Allow users to access quickly to actionable content through
optimal & dynamic search results presentation
e.g., “Movie Theater” vs. “Restaurant”
KPIs [that really matter]
• Page Views
• Unique Visitors
• Visit Duration
• Bounce Rate
• Return Visits Frequency
• (Useful) Interactions per Visits
Search
Presenta4on
Content
Knowledge
User
15. Agenda
• Yellow Pages & Myself
• The importance of relevance for our business
• Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion
17. Data-Driven Actions
Measure Act
• Implicit User Feedback (analytics & logs)
• Stable Test Collection
• Side-By-Side Comparison
• A/B Testing
• Explicit User Feedback (Survey)
• Find gaps in data and improve them
• Improve processes
• Improve presentation
• Improve algorithms
• Confirm hypothesis
KPIs
18. Know Your Data
• Web & Mobile Analytics
• Search logs Analysis
• Build KPIs
19. From Data To Action
Examples:
• Missing geo-location
• Missing businesses
• Seasonal trends (adapt search algo)
Manual
or
automa8c
ac8ons
20. Test Collection
KPIs: Recall & Precision
• Predefined test collection built manually
• Variety of use cases that cover most users needs
• Evaluate new updates before going to production
21. Side-By-Side Comparison
• Comparison of two (versions of the)
Search Engines
• Pre-defined set of real searches selected
from logs
• Pre-classified (keywords, business
names, language, geo location, etc.)
• “Blind” comparison
• Significant number of testers per search
22. A/B Testing
Principle
• Experiment one or many ingredients at a
time with real users
• Some non-trivial amount of traffic is
always being tested
KPIs
• Page Views
• Unique Visitors
• Visit Duration
• Bounce Rate
• Return Visits Frequency
• Useful Interactions per Visits
Simulation
• Run new versions on a significant
search logs sample
• Estimate the impact prior
deployment to prod and avoid bad
surprises
23. Relevance In Action
Valuable signals, NOT only for marketing!
e.g., Bounce Rate – Why the user left quickly?
• Who is the user?
• Where they come from (direct vs. SE)
• Their devices
• Which content was provided?
• Which page?
• etc.
24. Agenda
• Yellow Pages & Myself
• The importance of relevance for our business
• Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion