Pickens, J., Cooper, M., and Golovchinsky, G. Reverted Indexing for Expansion and Feedback. In Proc. CIKM 2010, Toronto, Canada, ACM Press. See http://fxpal.com/?p=abstract&abstractID=581
3. Query-Document Duality has long history
• Using queries to label documents
• Queries and documents as bipartite graph
– Used for random walks
– Used for partitioning
• Reverse Querying
4. Motivation – Three R’s
Retrievability
Reuse (Algorithmic)
Recall-Oriented
Tasks
5. Our Key Contribution
We treat query result sets as unstructured
text “documents” -- and index them
8. Basis Query
(Reverted Document ID)
Query
Expression
Ranking
Algorithm
giraffe BM25
cheetah BM25
gazelle BM25
gazelle Language Model
gazelle PL2 (Divergence from Randomness)
gazelle Y
gazelle B
gazelle G
fast cheetah BM25
cheetah AND NOT gazelle Boolean
Latitude+Longitude of Zanzibar Euclidean distance
16. Reverted Indexing
1. Choose a set of basis queries
2. For each basis query:
1. Execute each query, producing results up to
cutoff depth k
2. Use results to create a “reverted document”
3. Add the reverted document to the index
How basis queries are chosen (in these experiments):
All singleton terms (unigrams) with df ≥ 2. Ranking
algorithm for all basis queries is PL2.
20. Reverted Index Statistics
Retrieval Score of docid Term Frequency
Sum of Retrieval Scores
of all docids retrieved by
a Basis Query
Document Length
Number of Basis
Queries that docid was
retrieved by
Document Frequency
22. Experiment: Relevance Feedback
1. Run initial query using PL2 (Terrier platform)
[poaching wildlife preserves]
2. Judge top k documents for relevance
3.
4. Expand using top 500 terms (strongest baseline @ 500)
5. Run expanded query using PL2
6. Evaluate
Use KL Divergence
to select and weight
query expansion
terms
Use Bo1 to select
and weight query
expansion terms
Use PL2 retrieval on
the Reverted Index
to select and weight
query expansion
terms
37. Related Work
Inspiration:
“Retrievability: An Evaluation Measure for
Higher Order Information Access Tasks” --
Azzopardi and Vinay, CIKM 2008
Azzopardi & Vinay take a document centric
approach, examining whether documents
(n)ever appear among top k results to any query
38. Related Work
Query-Document Duality has long history
– S. E. Robertson. “Query-Document Symmetry
and Dual models.” Journal of Documentation,
50(3),1994
– B. Billerbeck, F. Scholer, H. E. Williams, and
J. Zobel. Query Expansion Using Associated
Queries. CIKM '03
– N. Craswell and M. Szummer. Random walks
on the Query-Click Graph. SIGIR 2007
– Reverse Querying / alerting (various)
39. Future Extensions
Basis queries
– Query expression may be arbitrarily complex
– Ranking function may be arbitrarily complex
(remember: ranking function is a part of the
basis query)
Reverted queries
– Best Match: [#415 #56 #42 #85]
– Boolean: (#415 AND #56) OR (#42 AND #85)
– Other query operators:
[SYNONYM(#415 #56) #42 #85]
[ORDERED(#415 #56) #42 #85]
40. Motivation – Three R’s
Retrievability
Reuse (Algorithmic)
Recall-Oriented
Tasks
My main difference: TF (=original basis query retrieval score, i.e. it’s tied to the actual performance of the system) and IDF (=just how many basis queries a docid was retrieved by).
Other notes:
Craswell:
bi-partite click-thru graphs for random walks (manual selection no automatic retrievability)
model aggregate behavior using random walks from single starting document (no notion of indexing collection)
Billerbeck et al
Build pseudo-documents comprised of previous queries (text only)
Limited to user queries
Truncates result sets; degenerate DL and IDF statistics
No relevance scores
Standard text search and query expansion in these pseudo-documents
Just under 25% of documents in the collection had zero associations
“Reverse Querying” / alerting
Given a document(s?), find the queries that match it
In implementations I’ve seen, this is just a Boolean proposition (matches/doesn’t match), either as a whole or in the top-k.
Even if it’s in the top-k, it’s a boolean presence/absense
No sense of “tf”, or of “idf”
Docids become “terms”. Score becomes “term frequency”.
And that’s it. We’re done. In IR, we know how to go forward from here!
Add each reverted document to the index.. All-the-while calculating global and local statistics, e.g. tf and idf, etc.
Why we call this a “reverted” index.
PL2 as the “forward” ranking algorithm, because we determined a priori that yielded the best MAP…and we want as many relevant docs in the top k as possible.
Note that everything else is held constant, except for the term expansion and weighting
My main difference: TF (=original basis query retrieval score, i.e. it’s tied to the actual performance of the system) and IDF (=just how many basis queries a docid was retrieved by).
Other notes:
Craswell:
bi-partite click-thru graphs for random walks (manual selection no automatic retrievability)
model aggregate behavior using random walks from single starting document (no notion of indexing collection)
Billerbeck et al
Build pseudo-documents comprised of previous queries (text only)
Limited to user queries
Truncates result sets; degenerate DL and IDF statistics
No relevance scores
Standard text search and query expansion in these pseudo-documents
Just under 25% of documents in the collection had zero associations
“Reverse Querying” / alerting
Given a document(s?), find the queries that match it
In implementations I’ve seen, this is just a Boolean proposition (matches/doesn’t match), either as a whole or in the top-k.
Even if it’s in the top-k, it’s a boolean presence/absense
No sense of “tf”, or of “idf”
There are many similarities with running docids as queries, to running terms as queries. But they’re not completely similar! (phrase operators, for example?)
(Retrievability): The basis queries that we retrieve already have shown themselves as capable of retrieving (at least some) relevant documents at a high rank! Don’t need to build fancy probabilistic models to know what the “best” terms are, because the basis queries have already “recorded” it.
…
(Reuse): Did not have to invent any new relevance feedback models. Simply used PL2 (or BM25 or tf.idf or LM or DFRee) to do the retrieval of basis queries. Combining terms, synonym operators, etc. all possible
…
(Recall): Evaluated by applying it to relevance-feedback
And that’s it. We’re done. In IR, we know how to go forward from here!