Evaluating Heterogeneous Information Access (Position Paper)

•

1 gostou•1,118 visualizações

This document discusses challenges in evaluating heterogeneous information access systems and proposes areas for future research. It notes that traditional IR evaluation focuses on system-oriented metrics using test collections, while heterogeneous search involves more complex user behaviors like non-linear browsing. Key challenges include accounting for diverse search tasks, coherence, diversity, personalization and different result presentation strategies. The document advocates better understanding user behavior through models and applying this understanding to improve evaluation metrics for heterogeneous search systems. It identifies areas for future work such as modeling task complexity, coherence and click patterns to develop more powerful evaluation frameworks.

Tecnologia Negócios

Evalua&ng
Heterogeneous

Informa&on
Access
(Posi&on
Paper)
Ke
Zhou1,
Tetsuya
Sakai2,
Mounia
Lalmas3,

Zhicheng
Dou2
and
Joemon
M.
Jose1

1University
of
Glasgow

2MicrosoN
Research
Asia

3Yahoo!
Labs
Barcelona
SIGIR
2013

MUBE
workshop

IR
Evalua&on
•  System-‐oriented
Evalua&on
(test
collec&on
+

metrics)

•  User-‐oriented
Evalua&on
(interac&ve
user

study)

•  Current
endeavor
to
incorporate
user
into

system-‐oriented
metrics

– Time-‐Biased
Gain
(Smucker,
Clarke.)

– U-‐measure
(Sakai,
Dou)

– etc.

Increasing
Heterogeneous
Nature

on
Search

Posi&on
•  Compared
with
tradi&onal
homogeneous

search,
evalua&on
in
the
context
of

heterogeneous
informa&on
is
more

challenging
and
requires
taking
into
account

more
complex
user
behaviors
and

interac4ons.

Challenges
•  Non-‐linear
Traversal
Browsing

•  Diverse
Search
Tasks

•  Coherence

•  Diversity

•  Personaliza&on

•  etc.

Various
Presenta&on
Strategies
Non-‐linear
BlendedBlended
……

Tabbed

User
Browsing
Pa^ern
“E”
Browsing
Pa?ern

on
Aggregated
Search
Page
“F”
Browsing
Pa?ern

on
Organic
Search
Page
h^p://searchengineland.com/eye-‐tracking-‐on-‐universal-‐and-‐personalized-‐search-‐12233

Non-‐linear
Traversal
Browsing

h^p://searchengineland.com/eye-‐tracking-‐on-‐universal-‐and-‐personalized-‐search-‐12233

Search
Tasks:
Ver&cal
Orienta&on
CIKM’10
(Sushmita
et
al.),
WWW’13
(Zhou
et
al.)

Search
Tasks:
Complexity
SIGIR’12
(Arguello
et
al.)

Coherence
Car
Animal
Car
Car
Car
Car
Car
Car
Car
Car
Car
Car
vs.
CIKM’12
(Arguello
et
al.)

Coherence
Car
Animal
Animal
Car
Animal
Car
Car
Car
Animal
Car
Animal
Car
vs.
CIKM’12
(Arguello
et
al.)

Diversity
vs.
Image News+Map+Image
SIGIR’12
(Zhou
et
al.)

Avenues
of
Research
•  Be^er
understanding
of
users

–  Click
models:
WSDM’12
(Chen
et
al.),
SIGIR’13
(Wang

et
al.)

–  Ver&cal
orienta&on:
CIKM’10
(Sushmita
et
al.),

WWW’13
(Zhou
et
al.)

–  Task
complexity:
SIGIR’12
(Arguello
et
al.)

–  Task
coherence:
CIKM’12
(Arguello
et
al.)

–  Diversity:
SIGIR’12
(Zhou
et
al.)

–  Personaliza&on

–  Non-‐linear
presenta&on
strategies

Avenues
of
Research
•  Be^er
incorpora&on
of
learned
user
behavior

into
evalua&on
metrics

– follow
SIGIR’13
(Chuklin
et
al)
and
convert

obtained
aggregated
search
click
models
into

system-‐oriented
evalua&on
metrics.

– model
addi&onal
features
into
powerful

evalua&on
framework
(e.g.
TBG,
U-‐measure,
AS-‐
metric).

Thank
you!
Ques&ons?
Ke
Zhou,
zhouke@dcs.gla.ac.uk
TREC
2013
FedWeb
track:

h^ps://sites.google.com/site/trecfedweb/

Mais conteúdo relacionado

Semelhante a Evaluating Heterogeneous Information Access (Position Paper)

Collaborative information retrieval systems often rely on division of labor policies. Such policies allow work to be divided among collaborators with the aim of preventing redundancy and optimizing the synergic effects of collaboration. Most of the underlying methods achieve these goals by the means of explicit vs. implicit role-based mediation. In this paper, we investigate whether and how different factors, such as users' behavior, search strategies, and effectiveness, are related to role assignment within a collaborative exploratory search. Our main findings suggest that: (1) spontaneous and cohesive implicit roles might emerge during the collaborative search session implying users with no prior roles, and that these implicit roles favor the search precision, (2) role drift might occur alongside the search session performed by users with prior-assigned roles.

Understanding the Impact of the Role Factor in Collaborative Information Retr...

UPMC - Sorbonne Universities

Learning about Information Searchers from Eye-Tracking by Jacek Gwizdka

jacekg

The research topic of this document deals with a particular setting of information retrieval (IR), referred to as collaborative information retrieval (CIR), in which a set of multiple collaborators share the same information need. Collaboration is particularly used in case of complex tasks in which an individual user may have insufficient knowledge and may benefit from the expertise/knowledge or complementarity of other collaborators. This multi-user context rises several challenges in terms of search interfaces as well as ranking models, since new paradigms must be considered, namely division of labor, sharing of knowledge and awareness. These paradigms aim at avoiding redundancy between collaborators in order to reach a synergic effect within the collaboration process. Several approaches have been proposed in the literature. First, search interfaces have been oriented towards a user mediation in order to support collaborators’ actions through information storage or communication tools. Second, more close to our contributions, previous work focus on the information access issue by designing ranking models adapted to collaborative environments dealing with the challenges of (1) personalizing result set to collaborators, (2) favoring the sharing of knowledge, (3) dividing the labor among collaborators and/or (4) considering particular roles of collaborators within the information seeking process. In this thesis, we focus, more particularly, on two main aspects of the collaboration : — The expertise of collaborators by proposing retrieval models adapted to the domain expertise level of collaborators. The expertise levels might be vertical, in the case of domain expert and novice, or horizontal when collaborators have different subdomain expertise. We, therefore, propose two CIR models on two steps including a document relevance scoring with respect to each role and a document allocation to user roles through the Expectation–Maximization (EM) learning method applied on the document relevance scoring in order to assign documents to the most likely suited user. — The complementarity of collaborators throughout the information seeking process by mining their roles on the assumptions that collaborators might be different and complementary in some skills. We propose two algorithms based either on predefined roles or latent roles which (1) learns about the roles of the collaborators using various search-related features for each individual involved in the search session, and (2) adapts the document ranking to the mined roles of collaborators.

Thesis slides - Definition and evluation of collaborative information retriev...

UPMC - Sorbonne Universities

Measurment and Modeling of Eue-mouse Behavior in the Presence of Nonlinear Pa...

Ouzza Brahim

The User is the Query - The Rise of Predictive Proactive Search

Dawn Anderson MSc DigM

user behaviour analysis

vaibhavtyagi111

A variety of query approaches have been proposed by the semantic web community to explore and query semantic data. Each was developed for a specific task and employed its own interaction mechanism; each query mechanism has its own set of advantages and drawbacks. Most semantic web search systems employ only one approach, thus being unable to exploit the benefits of alternative approaches. Motivated by a usability and interactivity perspective, we propose to combine two query approaches (graph-based and natural language) as a hybrid query approach. In this paper, we present NL-Graphs which aims to exploit the strengths of both approaches, while ameliorating their weaknesses. NL-Graphs was conceptualised and developed from observations, and lessons learned, in several evaluations with expert and casual users. The results of evaluating our approach with expert and casual users on a large semantic dataset are very encouraging; both types of users were highly satisfied and could effortlessly use the hybrid approach to formulate and answer queries. Indeed, success rates showed they were able to successfully answer all the evaluation questions.

NL-Graphs: A Hybrid Approach toward Interactively Querying Semantic Data

Suvodeep Mazumdar

Optimizing Search User Interfaces and Interactions within Professional Social...

Nik Spirin

The Search for Truth in Objective & Subject Crowdsourcing

Matthew Lease

Learning to recommend with user generated content

Yueshen Xu

Umap15fernandeztobias slides

Universidad Autonoma de Madrid

Learning To Rank User Queries to Detect Search Tasks

Franco Maria Nardini

These slides depict the work carried out on the COPE project from 2011 to 2013. We propose a method to identify problems and coping tactics in populations that are constrained. Then we generalise the problems encountered and the strategies employed to broader audiences. We conclude that there is an overlap between the problems and strategies of blind and sighted users. This provides more support to the hypothesis of the accessibility continuum.

Detecting Problematic Interactions on the Web. The COPE Project: Coping strat...

Markel Vigo

People search is an active research topic in recent years. Related works includes expert finding, collaborator recommendation, link prediction and social matching. However, the diverse objectives and exploratory nature of those tasks make it difficult to develop a flexible method for people search that works for every task. In this project, we developed PeopleExplorer, an interactive people search system to support exploratory search tasks when looking for people. In the system, users could specify their task objectives by selecting and adjusting key criteria. Three criteria were considered: the content relevance, the candidate authoritativeness and the social similarity between the user and the candidates. This project represents a first attempt to add transparency to exploratory people search, and to give users full control over the search process. The system was evaluated through an experiment with 24 participants undertaking four different tasks. The results show that with comparable time and effort, users of our system performed significantly better in their people search tasks than those using the baseline system. Users of our system also exhibited many unique behaviors in query reformulation and candidate selection. We found that users’ general perceptions about three criteria varied during different tasks, which confirms our assumptions regarding modeling task difference and user variance in people search systems.

Supporting Exploratory People Search: A Study of Factor Transparency and User...

Shuguang Han

Proposed search interfaces have significant effect on eye gaze behavior in terms of fixations; MeSH terms received more attention when displayed alongside each document for experienced searchers; MeSH terms attracted to domain experts and analytic users when displayed under a search box; Perceived search task difficulty has moderating effect on eye gaze patterns; Eye gaze behavior could be used to infer individual differences and user perceptions.

The Eyes Have It: Individual Differences and Eye Gaze Behaviour in Biomedical...

Ying-Hsang Liu

Invited Lecture on Interactive Information Retrieval

DavidMaxwell77

Interfaces for User-Controlled and Transparent Recommendations

Peter Brusilovsky

Analytic innovation transforming instagram data into predicitive analytics wi...

suresh sood

Week 11 Collection of Data – questionnaire and Instruments & sampling 1 RES 500 Academic Writing and Research Skills 2 Overall Flowchart for Instrument Design (Source: Cooper & Schindler, 2013, Exhibit 13-1, p. 296) 3 Flowchart for Instrument Design Phase 1 (Source: Cooper & Schindler, 2013, Exhibit 13-2, p. 297) 4 Disguising Study Objectives Situations where disguise is unnecessary Willingly shared, Conscious-level information Reluctantly shared, Conscious-level information Knowable, Limited-conscious-level information Subconscious-level information 5 Flowchart for Instrument Design Phase 2 (Source: Cooper & Schindler, 2013, Exhibit 13-5, p. 302) 6 Question Content Should this question be asked? Is the question of proper scope and coverage? Can the participant adequately answer this question as asked? Will the participant willingly answer this question as asked? 7 Multiple Choice Response Strategy Which one of the following factors was most influential in your decision to attend Metro U? Good academic standing Specific program of study desired Enjoyable campus life Many friends from home High quality of faculty 8 Flowchart for Instrument Design Phase 3 (Source: Cooper & Schindler, 2013, Exhibit 13-10, p. 315) 9 Guidelines for Question Sequencing Interesting topics early Simple topics early Sensitive questions later Classification questions later Transition between topics Reference changes limited Sampling 10 Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our results back to the population from which they were chosen. 11 The Nature of Sampling Population Population element Census Sample Sampling frame 12 Why Sample? Greater accuracy Availability of elements Greater speed Sampling provides Lower cost 13 Sampling Design within the Research Process (Source: Cooper & Schindler, 2013, Exhibit 14-1, p. 342) 14 Types of Sampling DesignsElement SelectionProbabilityNonprobabilityUnrestrictedSimple randomConvenienceRestrictedComplex randomPurposiveSystematicJudgmentClusterQuotaStratifiedSnowballDouble (Adapted from Cooper & Schindler, 2013, Exhibit 14-2, p. 343) 15 Steps in Sampling Design What is the target population? What are the parameters of interest? What is the sampling frame? What is the appropriate sampling method? What size sample is needed? References 16 Cooper, D. R., & Schindler, P. S. (2013). Questionnaires and instruments. In Business research methods (pp. 294-335). New York, NY: McGraw-Hill. Cooper, D. R., & Schindler, P. S. (2013). Sampling. In Business research methods (pp. 336-372). New York, NY: McGraw-Hill. 16 .

Week 11Collection of Data – questionnaire and Instruments & .docx

melbruce90096

Tutorial at UMAP 2022: In recent years, the use of Artificial Intelligence (AI) technologies expanded to many areas where they directly affect the lives of many people. AI-based approaches advise human decision-makers who should be released on bail, whether it is a good time to discharge a patient from a hospital and whether a specific student is at risk to fail a course. Such an extensive use in AI in decision making came with a range of protentional problems that have been extensively studied over the last few years. Recognition of these problems motivated a rapid rise of research on “human-centered AI”, which attempted to address and minimize the negative effects of using AI technologies. Among the ideas of human-centered AI is user control - engaging users in affecting AI decision making to prevent possible errors and biases. In my talk, I will focus on the application of user control in one popular area of AI application, adaptive information access. Adaptive information access systems such as personalized search and recommender systems attempt to model their users to help them in finding the most relevant information. Yet, user modeling and personalization mechanisms might not always work as expected resulting in errors, biases, and suboptimal behavior. Combining the decision power or AI with the ability of the user to guide and control it brings together the strong sides of artificial and human intelligence and could lead to better results. This tutorial will provide a systematic review of approaches focused on adding various kinds of user control to adaptive information access systems and discuss lessons learned, prospects, and challenges of this direction of research.

User Control in Adaptive Information Access

Peter Brusilovsky

Semelhante a Evaluating Heterogeneous Information Access (Position Paper) (20)

Understanding the Impact of the Role Factor in Collaborative Information Retr...

Learning about Information Searchers from Eye-Tracking by Jacek Gwizdka

Thesis slides - Definition and evluation of collaborative information retriev...

Measurment and Modeling of Eue-mouse Behavior in the Presence of Nonlinear Pa...

The User is the Query - The Rise of Predictive Proactive Search

user behaviour analysis

NL-Graphs: A Hybrid Approach toward Interactively Querying Semantic Data

Optimizing Search User Interfaces and Interactions within Professional Social...

The Search for Truth in Objective & Subject Crowdsourcing

Learning to recommend with user generated content

Umap15fernandeztobias slides

Learning To Rank User Queries to Detect Search Tasks

Detecting Problematic Interactions on the Web. The COPE Project: Coping strat...

Supporting Exploratory People Search: A Study of Factor Transparency and User...

The Eyes Have It: Individual Differences and Eye Gaze Behaviour in Biomedical...

Invited Lecture on Interactive Information Retrieval

Interfaces for User-Controlled and Transparent Recommendations

Analytic innovation transforming instagram data into predicitive analytics wi...

Week 11Collection of Data – questionnaire and Instruments & .docx

User Control in Adaptive Information Access

Mais de Mounia Lalmas-Roelleke

Today, I had the big honor to give the opening keynote at the 8th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2020), being held virtually. HCOMP is the home of the human computation and crowdsourcing community working on frameworks, methods and systems that bring together people and machine intelligence to achieve better results. I decided to totally revamp a previous talk to focus on so-called "human in the loop" and showed how we incorporate human in the loop to personalise at scale, with some of the research at Spotify. Sharing the slides for general interests.

Engagement, Metrics & Personalisation at Scale

Mounia Lalmas-Roelleke

These are the slides of my invited talk at the REVEAL workshop at RecSys 2019. The workshop focuses on the offline evaluation for recommender systems, and this year’s focus was on Reinforcement Learning. Although not directly related to reinforcement learning, it is clear that there are connections to what research in reinforcement learning is attempting to achieve (defining the rewards) and metrics that are optimized by recommender systems. I presented various works and personal thoughts on how to develop metrics of user engagement, which recommender systems can optimize for. An important message was that, for recommender systems to work both in the short and the long-term, it is important to consider the heterogeneity of both user and content to formalise the notion of engagement, and in turn design the appropriate metrics to capture these and optimize for. One way to achieve this is to follow these four steps: 1) Understanding intents; 2) Optimizing for the right metric; 3) Acting on segmentation; and 4) Thinking about diversity. An previous version of this talk was given to UMAP 2019. See https://www.slideshare.net/mounialalmas/metrics-engagement-personalization

Engagement, metrics and "recommenders"

Mounia Lalmas-Roelleke

These are the slides of the keynote I gave at UMAP 2019 (User Modeling, Adaptation and Personalization) held in Larnaca, June 2019. The theme of the conference this year was "Making Personalization Transparent: Giving Control Back To The User". I looked at the 1st part for my talk. When users interact with the recommendations served to them, they leave behind fine-grained traces of interaction patterns, which can be leveraged to predict how satisfying their experience was. This talk will present various works and personal thoughts on how to measure user engagement. It will discuss the definition and development of metrics of user satisfaction that can be used as proxy of user engagement, and will include cases of good, bad and ugly scenarios. An important message will be to show that, to make personalization transparent, it is important to consider the heterogeneity of both user and content to formalise the notion of satisfaction, and in turn design the appropriate satisfaction metrics to capture these. One way to do this is to consider the following angles: 1) Understanding intents; 2) Optimizing for the right metric; 3) Acting on segmentation; and 4) Thinking about diversity.

Metrics, Engagement & Personalization

Mounia Lalmas-Roelleke

These are the slides of the tutorial Liangjie Hong and I gave at The Web Conference in San Francisco, 2019. Full details of the tutorial and previous instances can be found at https://onlineuserengagement.github.io/. Tutorial abstract: User engagement plays a central role in companies operating online services, such as search engines, news portals, e commerce sites, entertainment services, and social networks. A main challenge is to leverage collected knowledge about the daily online behavior of millions of users to understand what engage them short-term and more importantly long-term. Two critical steps of improving user engagement are metrics and their optimization. The most common way that engagement is measured is through various online metrics, acting as proxy measures of user engagement. This tutorial will review these metrics, their advantages and drawbacks, and their appropriateness to various types of online services. Once metrics are defined, how to optimize them will become the key issue. We will survey methodologies including machine learning models and experimental designs that are utilized to optimize these metrics via directly or indirect ways. As case studies, we will focus on four types of services, news, search, entertainment, and e-commerce. We will end with lessons learned and a discussion on the most promising research directions. Presenters: Liangjie Hong, Director of Engineering, Data Science and Machine Learning at Etsy Inc. Mounia Lalmas, Director of Research at Spotify, and Head of Tech Research in Personalization.

Tutorial on Online User Engagement: Metrics and Optimization

Mounia Lalmas-Roelleke

Recommending and searching @ Spotify

Mounia Lalmas-Roelleke

These are the slides of my talk at the 2019 Netflix Workshop on Personalization, Recommendation and Search (PRS). This talk is based on previous talks on research we are doing at Spotify, but here I focus on the work we do on personalizing Spotify Home, with respect to success, intent & diversity. The link to the workshop is https://prs2019.splashthat.com/. This is research from various people at Spotify, and has been published at RecSys 2018, CIKM 2018 and WWW (The Web Conference) 2019.

Personalizing the listening experience

Mounia Lalmas-Roelleke

These are the slides of a talk about some of our research at Spotify, as part of the celebration kickoff of Chalmers AI Research Centre in Gothenburg. I always like to make a story in my talk, and this time I wanted to reflect on the "push" (think recommender system) and "pull" (think search) paradigms. I am using this quote from Nicholas Belkin and Bruce Croft from their Communications of the ACM article published in 1992 to frame my story: "We conclude that information retrieval and information filtering are indeed two sides of the same coin. They work together to help people get the information needed to perform their tasks."

Recommending and Searching (Research @ Spotify)

Mounia Lalmas-Roelleke

At the BCS Search Solutions 2018, I gave a talk about work on search we are doing at Spotify. The talk described what search means in the context of Spotify, how it differs what we know about search, and the challenges associated with understanding user intents and mindsets in an "entertainment" context. The talk also discussed various efforts at Spotify to understand why users submit search queries, what they expect, how they assess their search experience, and how Spotify responds to these search queries. This is work done with many colleagues at Spotify in Boston, London, New York and Stockholm, and our wonderful summer interns.

Search @ Spotify

Mounia Lalmas-Roelleke

User engagement plays a central role in companies operating online services, such as search engines, news portals, e-commerce sites, and social networks. A main challenge is to leverage collected knowledge about the daily online behavior of millions of users to understand what engage them short-term and more importantly long-term. The most common way that engagement is measured is through various online metrics, acting as proxy measures of user engagement. This tutorial reviews these metrics and proposes a taxonomy of metrics. As case studies, it focuses on two types of services, search and e-commerce. The tutorial also discusses how to develop better machine learning models to optimize online metrics, and design experiments to test these models. This tutorial was given by Mounia Lalmas from Spotify and Liangjie Long from Etsy Inc. This tutorial was presented at WSDM 2018 (11th ACM International Conference on Web Search and Data Mining). It is the first delivery of this tutorial, so feedbacks and comments are welcome. We intend to continue working on this material.

Tutorial on metrics of user engagement -- Applications to Search & E- commerce

Mounia Lalmas-Roelleke

Slides for my lecture on IR evaluation, presented at 11th European Summer School in Information Retrieval (ESSIR 2017) at Universitat Pompeu Fabra, Barcelona. These slides were based on 1. Evaluation lecture @ QMUL; Thomas Roelleke & Mounia Lalmas 3. Lecture 8: Evaluation @ Stanford University; Pandu Nayak & Prabhakar Raghavan 4. Retrieval Evaluation @ University of Virginia; Hongnig Wang 5. Lectures 11 and 12 on Evaluation @ Berkeley; Ray Larson 6. Evaluation of Information Retrieval Systems @ Penn State University; Lee Giles Textbooks: 1. Information Retrieval, 2nd edition, C.J. van Rijsbergen (1979) 2. Introduction to Information Retrieval, C.D. Manning, P. Raghavan & H. Schuetze (2008) 3. Modern Information Retrieval: The Concepts and Technology behind Search, 2nd ed; R. Baeza-Yates & B. Ribeiro-Neto (2011)

An introduction to system-oriented evaluation in Information Retrieval

Mounia Lalmas-Roelleke

Many of today’s websites have recognised the importance of mobile friendly pages to keep users engaged and to provide a satisfying user experience. However, next to the experience provided by the sites themselves, advertisements, when clicked, present users with landing pages that are not necessarily mobile friendly. We explore what type of features are able to characterise the mobile friendliness of sponsored search ad landing pages. To have a complete understanding of the mobile ad experience in terms of layout and visual appearance, we also explore the notion of the ad page aesthetic appeal. We design and collect annotations for both dimensions on a large set of ads, and find that mobile friendliness and aesthetics represent different notions. We perform a comprehensive study of the effectiveness of over 120 features on the tasks of friendliness and aesthetics prediction. We find that next to general page size, HTML, and resource usage based features, several features based on the visual composition of landing pages are important to determine mobile friendliness and aesthetics. We demonstrate the additional benefit of these various types of features by comparing against the mobile friendliness guidelines provided by W3C. Finally, we use our models to determine the state of landing page mobile friendliness and aesthetics on a large sample of advertisements of a major internet company. These are the slides of work presented at WWW 2017 in Perth: M. Bron, M. Redi, F. Silvestri, H. Evans, M. Chute and M. Lalmas. Friendly, Appealing or Both? Characterising User Experience in Sponsored Search Landing Pages, 26th International World Wide Web Conference (WWW 2017), Industrial Track, Perth, Australia, 3-7 April, 2017.

Friendly, Appealing or Both? Characterising User Experience in Sponsored Sear...

Mounia Lalmas-Roelleke

Slides for keynote "Social Media and AI: Don’t forget the users" at WWW 2017 workshop "International Workshop on Modeling Social Media: Machine Learning and AI for Modeling and Analyzing Social Media". I am arguing that we need consider two things: the source of what we use to make good algorithms and whether users are impacted the way we want to impact them. The talk is based on two uses cases around providing diversity (something many of us believe is good) to users: 1. Engaging through diversity: serendipity (same algorithm, different sources) 2. Engaging through diversity: awareness (effective algorithm, perception) My goal is to say, we may have the best AI, but we may get it wrong if we forget the users. I don't have answers, but it is important that we ask the right questions in today's world.

Social Media and AI: Don’t forget the users

Mounia Lalmas-Roelleke

Native advertising is a specific form of online advertising where ads replicate the look-and-feel of their serving platform. In such context, providing a good user experience with the served ads is crucial to ensure a positive user experience and hence long-term user engagement. In this talk, I will describe work at Yahoo aiming at understanding the user experience on ads in the mobile context and building learning frameworks to identify and account for ads of low quality while ensuring a return of investment to advertisers. Slides for the Invited Talk at BigData Innovators Gathering (BIG), co-located with WWW 2017, Perth 2017 (https://big2017.org). Earlier versions of this talk were given at various venues in London.

Advertising Quality Science

Mounia Lalmas-Roelleke

The advertising industry is seeking to use the unique data provided by the increasing usage of mobile devices and mobile applications (apps) to improve targeting and the experience with apps. As a consequence, understanding user behaviours with apps has gained increased interests from both academia and industry. In this paper we study user app engagement patterns and disruptions of those patterns in a data set unique in its scale and coverage of user activity. First, we provide a detailed account of temporal user activity patterns with apps and compare these to previous studies on app usage behavior. Then, in the second part, and the main contribution of this work, we take advantage of the scale and coverage of our sample and show how app usage behavior is disrupted through major political, social, and sports events. Slides for paper presented at TempWeb 2017: S. Van Canneyt, M. Bron, A. Haines and M. Lalmas. Describing Patterns and Disruptions in Large Scale Mobile App Usage Data, 7th Temporal Web Analytics Workshop (TempWeb), International World Wide Web Conference (WWW 2017), Industrial Track, Perth, Australia, 3-7 April, 2017.

Describing Patterns and Disruptions in Large Scale Mobile App Usage Data

Mounia Lalmas-Roelleke

We study the news reading behaviour of several hundred thousand users on 65 highly-visited news sites. We focus on a specific phenomenon: users reading several articles related to a particular news development, which we call story-focused reading. Our goal is to understand the effect of story-focused reading on user engagement and how news sites can support this phenomenon. We found that most users focus on stories that interest them and that even casual news readers engage in story-focused reading. During story-focused reading, users spend more time reading and a larger number of news sites are involved. In addition, readers employ different strategies to find articles related to a story. We also analyse how news sites promote story-focused reading, by looking at how they link their articles to related content published by them, or by other sources. The results show that providing links to related content leads to a higher engagement of the users, and that this is the case even for links to external sites. We also show that the performance of links can be affected by their type, their position, and how many of them are present within an article. This work co-authored with J. Lehmann, C. Castillo and R. Baeza-Yates has been published in the Journal of The Association For Information Science And Technology (JASIST), available online in May 2016. The work was presented at the Yahoo TechPulse Annual conference in December 2016.

Story-focused Reading in Online News and its Potential for User Engagement

Mounia Lalmas-Roelleke

Native advertising is a specific form of online advertising where ads replicate the look-and-feel of their serving platform. In such context, providing a good user experience with the served ads is crucial to ensure long-term user engagement. In this work, we explore the notion of ad quality, namely the effectiveness of advertising from a user experience perspective. We design a learning framework to predict the pre-click quality of native ads. More specifically, we look at detecting offensive native ads, showing that, to quantify ad quality, ad offensive user feedback rates are more reliable than the commonly used click-through rate metrics. We then conduct a crowd-sourcing study to identify which criteria drive user preferences in native advertising. We translate these criteria into a set of ad quality features that we extract from the ad text, image and advertiser, and then use them to train a model able to identify offensive ads. We show that our model is very effective in detecting offensive ads, and provide in-depth insights on how different features affect ad quality. Finally, we deploy a preliminary version of such model and show its effectiveness in the reduction of the offensive ad feedback rate. There are the slides of our WWW 2016 paper. This is work with Ke (Adam) Zhou, Miriam Redi and Andy Haines.

Predicting Pre-click Quality for Native Advertisements

Mounia Lalmas-Roelleke

In this paper we focus on estimating the post-click engagement on native ads by predicting the dwell time on the corresponding ad landing pages. To infer relationships between feature of the ads and dwell time we resort to the application of survival analysis techniques, which allow us to estimate the distribution of the length of time that the user will spend on the ad. This information is then integrated into the ad ranking function with the goal of promoting the rank of ads that are likely to be clicked and consumed by users (dwell time greater than a given threshold). The online evaluation over live traffic shows that considering post-click engagement has a consistent positive effect on both CTR, decreases the number of bounces and increases the average dwell time, hence leading to a better user post-click experience. There are the slides of our WWW 2016 paper. This is work with Nicola Barbieri and Fabrizio Silvestri.

Improving Post-Click User Engagement on Native Ads via Survival Analysis

Mounia Lalmas-Roelleke

Evaluating the search experience: from Retrieval Effectiveness to User Engage...

Mounia Lalmas-Roelleke

Slides of my presentation at SPIRE 2015 at King's College London. The talk builds on my work on user engagement, and proposes to move beyond clicks and relevance in information retrieval evaluation, towards user engagement. The talk presents some of my work to show how this could be done. There are two focus: moving from intra- to inter-session evaluation, and moving from small- to large-scale evaluation. These focuses are there to acknoweldge that (1) happy users come back, and (2) we need to properly identify who are the happy users. I hope that this talk will bring new perspectives to those building search systems and wanting to evaluate them, beyond their retrieval effectiveness.

A Journey into Evaluation: from Retrieval Effectiveness to User Engagement

Mounia Lalmas-Roelleke

Click-through rate (CTR) is the most common metric used to assess the performance of an online advert; another performance of an online advert is the user post-click experience. In this paper, we describe the method we have implemented in Yahoo Gemini to measure the post-click experience on Yahoo mobile news streams via an automatic analysis of advert landing pages. We measure the post-click experience by means of two well-known metrics, dwell time and bounce rate. We show that these metrics can be used as proxy of an advert post-click experience, and that a negative post-click experience has a negative effect on user engagement and future ad clicks. We then put forward an approach that analyses advert landing pages, and show how these can affect dwell time and bounce rate. Finally, we develop a prediction model for advert quality based on dwell time, which was deployed on Yahoo mobile news stream app running on iOS. The results show that, using dwell time as a proxy of post- click experience, we can prioritise higher quality ads. We demonstrate the impact of this on users via A/B testing.

Promoting Positive Post-click Experience for In-Stream Yahoo Gemini Users

Mounia Lalmas-Roelleke

Mais de Mounia Lalmas-Roelleke (20)

Engagement, Metrics & Personalisation at Scale

Engagement, metrics and "recommenders"

Metrics, Engagement & Personalization

Tutorial on Online User Engagement: Metrics and Optimization

Recommending and searching @ Spotify

Personalizing the listening experience

Recommending and Searching (Research @ Spotify)

Search @ Spotify

Tutorial on metrics of user engagement -- Applications to Search & E- commerce

An introduction to system-oriented evaluation in Information Retrieval

Friendly, Appealing or Both? Characterising User Experience in Sponsored Sear...

Social Media and AI: Don’t forget the users

Advertising Quality Science

Describing Patterns and Disruptions in Large Scale Mobile App Usage Data

Story-focused Reading in Online News and its Potential for User Engagement

Predicting Pre-click Quality for Native Advertisements

Improving Post-Click User Engagement on Native Ads via Survival Analysis

Evaluating the search experience: from Retrieval Effectiveness to User Engage...

A Journey into Evaluation: from Retrieval Effectiveness to User Engagement

Promoting Positive Post-click Experience for In-Stream Yahoo Gemini Users

Último

What are drone anti-jamming systems? The drone anti-jamming systems and anti-spoof technology protect against interference, jamming, and spoofing of the UAVs. To protect their security, countries are beginning to research drone anti-jamming systems, also known as drone strike weapons. The anti-jam and anti-spoof technology protects against interference, jamming and spoofing. A drone strike weapon is a drone attack weapon that can attack and destroy enemy drones. So what is so unique about this amazing system?

What Are The Drone Anti-jamming Systems Technology?

Antenna Manufacturer Coco

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

GenCyber Cyber Security Day Presentation

Michael W. Hawkins

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Advantages of Hiring UIUX Design Service Providers for Your Business

Pixlogix Infotech

Developing An App To Navigate The Roads of Brazil

V3cube

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Scaling API-first – The story of a global engineering organization

Radu Cotescu

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

HampshireHUG

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

Igalia

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

[2024]Digital Global Overview Report 2024 Meltwater.pdf

hans926745

Evaluating Heterogeneous Information Access (Position Paper)

1. Evalua&ng Heterogeneous Informa&on Access (Posi&on Paper) Ke Zhou1, Tetsuya Sakai2, Mounia Lalmas3, Zhicheng Dou2 and Joemon M. Jose1 1University of Glasgow 2MicrosoN Research Asia 3Yahoo! Labs Barcelona SIGIR 2013 MUBE workshop

2. IR Evalua&on •  System-‐oriented Evalua&on (test collec&on + metrics) •  User-‐oriented Evalua&on (interac&ve user study) •  Current endeavor to incorporate user into system-‐oriented metrics – Time-‐Biased Gain (Smucker, Clarke.) – U-‐measure (Sakai, Dou) – etc.

3. Increasing Heterogeneous Nature on Search

4. Increasing Heterogeneous Nature on Search ……

5. Posi&on •  Compared with tradi&onal homogeneous search, evalua&on in the context of heterogeneous informa&on is more challenging and requires taking into account more complex user behaviors and interac4ons.

6. Challenges •  Non-‐linear Traversal Browsing •  Diverse Search Tasks •  Coherence •  Diversity •  Personaliza&on •  etc.

7. Various Presenta&on Strategies Non-‐linear BlendedBlended …… Tabbed

8. User Browsing Pa^ern “E” Browsing Pa?ern on Aggregated Search Page “F” Browsing Pa?ern on Organic Search Page h^p://searchengineland.com/eye-‐tracking-‐on-‐universal-‐and-‐personalized-‐search-‐12233

9. Non-‐linear Traversal Browsing h^p://searchengineland.com/eye-‐tracking-‐on-‐universal-‐and-‐personalized-‐search-‐12233

10. Search Tasks: Ver&cal Orienta&on CIKM’10 (Sushmita et al.), WWW’13 (Zhou et al.)

11. Search Tasks: Complexity SIGIR’12 (Arguello et al.)

12. Coherence Car Animal Car Car Car Car Car Car Car Car Car Car vs. CIKM’12 (Arguello et al.)

13. Coherence Car Animal Animal Car Animal Car Car Car Animal Car Animal Car vs. CIKM’12 (Arguello et al.)

14. Diversity vs. Image News+Map+Image SIGIR’12 (Zhou et al.)

15. Personaliza&on vs. SIGIRerAverage User

16. Avenues of Research •  Be^er understanding of users –  Click models: WSDM’12 (Chen et al.), SIGIR’13 (Wang et al.) –  Ver&cal orienta&on: CIKM’10 (Sushmita et al.), WWW’13 (Zhou et al.) –  Task complexity: SIGIR’12 (Arguello et al.) –  Task coherence: CIKM’12 (Arguello et al.) –  Diversity: SIGIR’12 (Zhou et al.) –  Personaliza&on –  Non-‐linear presenta&on strategies

17. Avenues of Research •  Be^er incorpora&on of learned user behavior into evalua&on metrics – follow SIGIR’13 (Chuklin et al) and convert obtained aggregated search click models into system-‐oriented evalua&on metrics. – model addi&onal features into powerful evalua&on framework (e.g. TBG, U-‐measure, AS-‐ metric).

18. Thank you! Ques&ons? Ke Zhou, zhouke@dcs.gla.ac.uk TREC 2013 FedWeb track: h^ps://sites.google.com/site/trecfedweb/

Evaluating Heterogeneous Information Access (Position Paper)

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Evaluating Heterogeneous Information Access (Position Paper)

Semelhante a Evaluating Heterogeneous Information Access (Position Paper) (20)

Mais de Mounia Lalmas-Roelleke

Mais de Mounia Lalmas-Roelleke (20)

Último

Último (20)

Evaluating Heterogeneous Information Access (Position Paper)