Query Expansion with Locally-Trained Word Embeddings (Neu-IR 2016)

•

1 gostou•1,450 visualizações

Bhaskar Mitra

Slides presented at the Neu-IR workshop in SIGIR 2016.

Tecnologia

Query Expansion with Locally-Trained
Word Embeddings
Fernando Diaz Bhaskar Mitra Nick Craswell
Microsoft
July 21, 2016
1 / 22

word embedding: discriminatively trained vector
representation
2 / 22

L =
T∑
t=1
ωxt
term weight


∑
y∈Vt
c
log σ(ϕ(xt) · ϕ(y))
observed context
+
∑
y∈Vt
n
log σ(−ϕ(xt) · ϕ(y))
negative context


3 / 22

ωxt needs to reﬂect the importance of the term at evaluation
time.
4 / 22

what terms are important at query time?
6 / 22

p(w|R) probability of the term in the relevant documents.
7 / 22

how different is p(w|R) from p(w|C)?
8 / 22

KL(R, C)w = p(w|R) log
p(w|R)
p(w|C)
9 / 22

KL
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
rank
10 / 22

how much better can we do if we train with∑T
t=1 ωxt ∝ p(w|R)?
11 / 22

Language Model Scoring
score(d, q) = KL(θq, θd)
θq maximum likelihood query language model
θd document language model
12 / 22

Query Expansion with Word Embeddings
˜θq = UUT
θq
U |V| × k term embedding matrix
13 / 22

Query Expansion with Word Embeddings
Uglobal embedding trained with p(w|C)
Ulocal embedding trained with p(w|R)
14 / 22

Getting p(w|R)
p(d) =
exp(−KL(θq, θd))
∑
d′ exp(−KL(θq, θd′ ))
15 / 22

Getting p(w|R)
p(d) =
exp(−KL(θq, θd))
∑
d′ exp(−KL(θq, θd′ ))
˜p(w|R) =
∑
d
p(w|θd)p(d)
15 / 22

Data
docs words queries
trec12 469,949 438,338 150
robust 528,155 665,128 250
web 50,220,423 90,411,624 200
giga 9,875,524 2,645,367 -
wiki 3,225,743 4,726,862 -
17 / 22

Embeddings
• global
• public embeddings (GloVe, word2vec)
• word2vec on target corpus
• local: word2vec with documents sampled by p(d)
18 / 22

• ten-fold cross-validation
• metric: NDCG@10
19 / 22

Results
global local
wiki+giga gnews target target giga wiki
QL 50 100 200 300 300 400 400 400 400
trec12 0.514 0.518 0.518 0.530 0.531 0.530 0.545 0.535 0.563*
0.523
robust 0.467 0.470 0.463 0.469 0.468 0.472 0.465 0.475 0.517*
0.476
web 0.216 0.227 0.229 0.230 0.232 0.218 0.216 0.234 0.236 0.258*
20 / 22

global
local
top words by ˜p(w|R) (blue: query; red: top words by p(w|R))
21 / 22

Summary
• local embedding provides a stronger representation
than global embedding
• potential impact for other topic-speciﬁc natural
language processing tasks
• future work
• effectiveness improvements
• efﬁciency improvements
22 / 22

Mais conteúdo relacionado

Mais de Bhaskar Mitra

(Slides from my talk at SEA: Search Engines Amsterdam) Online information access systems, like recommender systems and search, mediate what information gets exposure and thereby influence their consumption at scale. There is a growing body of evidence that information retrieval (IR) algorithms that narrowly focus on maximizing ranking utility of retrieved items may disparately expose items of similar relevance from the collection. Such disparities in exposure outcome raise concerns of algorithmic fairness and bias of moral import, and may contribute to both representational harms—by reinforcing negative stereotypes and perpetuating inequities in representation of women and other historically marginalized peoples—and allocative harms, from disparate exposure to economic opportunities. In this talk, we present a framework of exposure fairness metrics that model the problem jointly from the perspective of both the consumers and producers. Specifically, we consider group attributes for both types of stakeholders to identify and mitigate fairness concerns that go beyond individual users and items towards more systemic biases in retrieval.

Joint Multisided Exposure Fairness for Search and Recommendation

Bhaskar Mitra

In this talk, I will share some of my personal reflections on the progress in the field of neural IR and some of the ongoing and future research directions that I am personally excited about. This talk will be informed by my own research in this area as well as my experience both as a developer/organizer of the MS MARCO benchmark and the TREC Deep Learning Track and as an applied researcher previously working on web scale search systems at Bing. My goal in this talk would be to move the conversation beyond neural reranking models towards a richer and bolder vision of search powered by deep learning.

What’s next for deep learning for Search?

Bhaskar Mitra

In this talk, I share some of my personal reflections and learnings on benchmark development and community building for making robust scientific progress. This talk is informed by my experience as a developer of the MS MARCO benchmark and as an organizer of the TREC Deep Learning Track. My goal in this talk is to situate the act of releasing a dataset in the context of broader research visions and to draw due attention to considerations of scientific and social outcomes that are invariably salient in the acts of dataset creation and distribution.

So, You Want to Release a Dataset? Reflections on Benchmark Development, Comm...

Bhaskar Mitra

Emerging machine learning approaches, including deep learning methods, for information retrieval (IR) have recently demonstrated significant improvements in accuracy of relevance estimation at the cost of increasing model complexity and corresponding rise in computational and environmental costs of training and inference. In web search, these costs are further compounded by the necessity to train on large-scale datasets, consume long documents as inputs, and retrieve relevant documents from web-scale collections within milliseconds in response to high volume query traffic. A typical playbook for developing deep learning models for IR involves largely ignoring efficiency concerns during model development and then later scaling these methods by either finding faster approximations of the same models or employing heuristics to reduce the input space over which these models operate. Domain knowledge about the specific IR task and deeper understanding of system design and data structures in whose context these models are deployed can significantly help with not only model simplification but also to inform data-structure specific machine learning model design. Alternatively, predictive machine learning can also be employed specifically to improve efficiency in large scale IR settings. In this talk, I will cover several case studies for both improving efficiency of machine learning models for IR as well as direct application of machine learning to improve retrieval efficiency, and conclude with a brief discussion on potential future directions for efficiency-sensitive benchmarking of machine learning models for IR.

Efficient Machine Learning and Machine Learning for Efficiency in Information...

Bhaskar Mitra

Online information access systems, like recommender systems and search, mediate what information gets exposure and thereby influence their consumption at scale. There is a growing body of evidence that information retrieval (IR) algorithms that narrowly focus on maximizing ranking utility of retrieved items may disparately expose items of similar relevance from the collection. Such disparities in exposure outcome raise concerns of algorithmic fairness and bias of moral import, and may contribute to both representational harms—by reinforcing negative stereotypes and perpetuating inequities in representation of women and other historically marginalized peoples—and allocative harms, from disparate exposure to economic opportunities. In this talk, we present a framework of exposure fairness metrics that model the problem jointly from the perspective of both the consumers and producers. Specifically, we consider group attributes for both types of stakeholders to identify and mitigate fairness concerns that go beyond individual users and items towards more systemic biases in retrieval. The development of expected exposure based metrics also opens up new opportunities and challenges for model optimization. We demonstrate how stochastic ranking policies can be optimized towards target expected exposure and highlight the trade-offs that may exist in optimizing for different fairness dimensions.

Multisided Exposure Fairness for Search and Recommendation

Bhaskar Mitra

Learning to rank (LTR) for information retrieval (IR) involves the application of machine learning models to rank artifacts, such as webpages, in response to user's need, which may be expressed as a query. LTR models typically employ training data, such as human relevance labels and click data, to discriminatively train towards an IR objective. The focus of this lecture will be on the fundamentals of neural networks and their applications to learning to rank.

Neural Learning to Rank

Bhaskar Mitra

The emergence of deep learning based methods for search poses several challenges and opportunities not just for modeling, but also for benchmarking and measuring progress in the field. Some of these challenges are new, while others have evolved from existing challenges in IR benchmarking exacerbated by the scale at which deep learning models operate. Evaluation efforts such as the TREC Deep Learning track and the MS MARCO public leaderboard are intended to encourage research and track our progress, addressing big questions in our field. The goal is not simply to identify which run is "best" but to move the field forward by developing new robust techniques, that work in many different settings, and are adopted in research and practice. This entails a wider conversation in the IR community about what constitutes meaningful progress, how benchmark design can encourage or discourage certain outcomes, and about the validity of our findings. In this talk, I will present a brief overview of what we have learned from our work on MS MARCO and the TREC Deep Learning track--and reflect on the state of the field and the road ahead.

Neural Information Retrieval: In search of meaningful progress

Bhaskar Mitra

We benchmark Conformer-Kernel models under the strict blind evaluation setting of the TREC 2020 Deep Learning track. In particular, we study the impact of incorporating: (i) Explicit term matching to complement matching based on learned representations (i.e., the “Duet principle”), (ii) query term independence (i.e., the “QTI assumption”) to scale the model to the full retrieval setting, and (iii) the ORCAS click data as an additional document description field. We find evidence which supports that all three aforementioned strategies can lead to improved retrieval quality.

Conformer-Kernel with Query Term Independence @ TREC 2020 Deep Learning Track

Bhaskar Mitra

Lecture slides presented at Northeastern University (December, 2020). Learning to rank (LTR) for information retrieval (IR) involves the application of machine learning models to rank artifacts, such as webpages, in response to user's need, which may be expressed as a query. LTR models typically employ training data, such as human relevance labels and click data, to discriminatively train towards an IR objective. The focus of this lecture will be on the fundamentals of neural networks and their applications to learning to rank.

Neural Learning to Rank

Bhaskar Mitra

This report discusses three submissions based on the Duet architecture to the Deep Learning track at TREC 2019. For the document retrieval task, we adapt the Duet model to ingest a "multiple field" view of documents—we refer to the new architecture as Duet with Multiple Fields (DuetMF). A second submission combines the DuetMF model with other neural and traditional relevance estimators in a learning-to-rank framework and achieves improved performance over the DuetMF baseline. For the passage retrieval task, we submit a single run based on an ensemble of eight Duet models.

Duet @ TREC 2019 Deep Learning Track

Bhaskar Mitra

The emergence of deep learning-based methods for information retrieval (IR) poses several challenges and opportunities for benchmarking. Some of these are new, while others have evolved from existing challenges in IR exacerbated by the scale at which deep learning models operate. In this talk, I will present a brief overview of what we have learned from our work on MS MARCO and the TREC Deep Learning track, and reflect on the road ahead.

Benchmarking for Neural Information Retrieval: MS MARCO, TREC, and Beyond

Bhaskar Mitra

Deep Neural Methods for Retrieval

Bhaskar Mitra

Learning to rank (LTR) for information retrieval (IR) involves the application of machine learning models to rank artifacts, such as items to be recommended, in response to user's need. LTR models typically employ training data, such as human relevance labels and click data, to discriminatively train towards an IR objective. The focus of this tutorial will be on the fundamentals of neural networks and their applications to learning to rank.

Neural Learning to Rank

Bhaskar Mitra

Learning to Rank with Neural Networks

Bhaskar Mitra

Deep Learning for Search

Bhaskar Mitra

Deep Learning for Search

Bhaskar Mitra

Neural Learning to Rank

Bhaskar Mitra

Deep Learning for Search

Bhaskar Mitra

A fundamental goal of search engines is to identify, given a query, documents that have relevant text. This is intrinsically difficult because the query and the document may use different vocabulary, or the document may contain query words without being relevant. We investigate neural word embeddings as a source of evidence in document ranking. We train a word2vec embedding model on a large unlabelled query corpus, but in contrast to how the model is commonly used, we retain both the input and the output projections, allowing us to leverage both the embedding spaces to derive richer distributional relationships. During ranking we map the query words into the input space and the document words into the output space, and compute a query-document relevance score by aggregating the cosine similarities across all the query-document word pairs. We postulate that the proposed Dual Embedding Space Model (DESM) captures evidence on whether a document is about a query term in addition to what is modelled by traditional term-frequency based approaches. Our experiments show that the DESM can re-rank top documents returned by a commercial Web search engine, like Bing, better than a term-matching based signal like TF-IDF. However, when ranking a larger set of candidate documents, we find the embeddings-based approach is prone to false positives, retrieving documents that are only loosely related to the query. We demonstrate that this problem can be solved effectively by ranking based on a linear mixture of the DESM and the word counting features.

Dual Embedding Space Model (DESM)

Bhaskar Mitra

Traditionally, machine learning based approaches to information retrieval have taken the form of supervised learning-to-rank models. Recently, other machine learning approaches—such as adversarial learning and reinforcement learning—have started to find interesting applications in retrieval systems. At Bing, we have been exploring some of these methods in the context of web search. In this talk, I will share couple of our recent work in this area that we presented at SIGIR 2018.

Adversarial and reinforcement learning-based approaches to information retrieval

Bhaskar Mitra

Mais de Bhaskar Mitra (20)

Joint Multisided Exposure Fairness for Search and Recommendation

What’s next for deep learning for Search?

So, You Want to Release a Dataset? Reflections on Benchmark Development, Comm...

Efficient Machine Learning and Machine Learning for Efficiency in Information...

Multisided Exposure Fairness for Search and Recommendation

Neural Learning to Rank

Neural Information Retrieval: In search of meaningful progress

Conformer-Kernel with Query Term Independence @ TREC 2020 Deep Learning Track

Neural Learning to Rank

Duet @ TREC 2019 Deep Learning Track

Benchmarking for Neural Information Retrieval: MS MARCO, TREC, and Beyond

Deep Neural Methods for Retrieval

Neural Learning to Rank

Learning to Rank with Neural Networks

Deep Learning for Search

Neural Learning to Rank

Deep Learning for Search

Dual Embedding Space Model (DESM)

Adversarial and reinforcement learning-based approaches to information retrieval

Último

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

💥 You’re lucky! We’ve found two different (lead) developers that are willing to share their valuable lessons learned about using UiPath Document Understanding! Based on recent implementations in appealing use cases at Partou and SPIE. Don’t expect fancy videos or slide decks, but real and practical experiences that will help you with your own implementations. 📕 Topics that will be addressed: • Training the ML-model by humans: do or don't? • Rule-based versus AI extractors • Tips for finding use cases • How to start 👨‍🏫👨‍💻 Speakers: o Dion Morskieft, RPA Product Owner @Partou o Jack Klein-Schiphorst, Automation Developer @Tacstone Technology

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

UiPathCommunity

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

DBX First Quarter 2024 Investor Presentation

Dropbox

MINDCTI Revenue Release Quarter One 2024

MIND CTI

Whatsapp Number Escorts Call girls 8617370543 Available 24x7 Mcleodganj Call Girls Service Offer Genuine VIP Model Escorts Call Girls in Your Budget. Mcleodganj Call Girls Service Provide Real Call Girls Number. Make Your Sexual Pleasure Memorable with Our Mcleodganj Call Girls at Affordable Price. Top VIP Escorts Call Girls, High Profile Independent Escorts Call Girls, Housewife Women Escorts Call Girl, College Girls Escorts Call Girls, Russian Escorts Call girls Service in Your Budget.

Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model

Deepika Singh

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

apidays

Keynote 2: APIs in 2030: The Risk of Technological Sleepwalk Paolo Malinverno, Growth Advisor - The Business of Technology Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...

apidays

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Understanding the FAA Part 107 License ..

Christopher Logan Kennedy

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

Vector Search -An Introduction in Oracle Database 23ai.pptx

Remote DBA Services

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Exploring Multimodal Embeddings with Milvus

Zilliz

Retrieval augmented generation (RAG) is the most popular style of large language model application to emerge from 2023. The most basic style of RAG works by vectorizing your data and injecting it into a vector database like Milvus for retrieval to augment the text output generated by an LLM. This is just the beginning. One of the ways that we can extend RAG, and extend AI, is through multilingual use cases. Typical RAG is done in English using embedding models that are trained in English. In this talk, we’ll explore how RAG could work in languages other than English. We’ll explore French, Chinese, and Polish.

Introduction to Multilingual Retrieval Augmented Generation (RAG)

Zilliz

Passkeys: Developing APIs to enable passwordless authentication Cody Salas, Sr Developer Advocate | Solutions Architect - Yubico Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...

apidays

CNIC Information System with Pakdata Cf In Pakistan

danishmna97