Slides

Algorithms for Large Data Sets Ziv Bar-Yossef Lecture 2 March 26, 2006 http://www.ee.technion.ac.il/courses/049011

Information Retrieval Setting I want information about Michael Jordan , the machine learning expert query User Document Collection “ Information Need” +”Michael Jordan” -basketball ,[object Object],[object Object],[object Object],[object Object],Ranked list of retrieved documents IR System documents No. 1 is good, Rest are bad feedback Revised ranked list of retrieved documents ,[object Object],[object Object],[object Object],[object Object]

Information Retrieval vs. Data Retrieval ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Information Retrieval vs. Data Retrieval Database tables, structured Free text, unstructured Data Knowledgeable users or automatic processes Non-expert humans Accessibility Unordered Ordered by relevance Results Exact matches Approximate matches Results SQL, Relational algebras Keywords, Natural language Queries Data Retrieval Information Retrieval

Information Retrieval Systems IR System query processor text processor user query ranked retrieved docs User Corpus ranking procedure system query retrieved docs index indexer tokenized docs postings raw docs

Search Engines Search Engine query processor text processor user query ranked retrieved docs User Web ranking procedure system query retrieved docs index indexer tokenized docs postings crawler global analyzer repository

Classical IR vs. Web IR Hypertext Text Documents Large Small # of matches Partially accessible Accessible Data accessibility Huge Large Volume Link-based Content-based IR techniques Widely diverse Homogeneous Format diversity In flux Infrequent Data change rate Noisy, dups Clean, no dups Data quality Web IR Classical IR

Outline ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Abstract Formulation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Document Representation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Classic IR (Relevance) Models ,[object Object],[object Object]

The Boolean Model ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

The Boolean Model: Pros & Cons ,[object Object],[object Object],[object Object],[object Object]

The Vector Space Model (VSM) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Popular Similarity Measures ,[object Object],[object Object],[object Object],[object Object],d q d –q  d q

TF-IDF Score: Motivation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

TF-IDF Score: Definition ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

VSM: Pros & Cons ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Retrieval Evaluation ,[object Object],[object Object],[object Object],[object Object],[object Object],D L q D q Recall: Precision:

Recall & Precision: Example ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],List A Relevant docs: d 123 , d 56 , d 9 , d 25 , d 3 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],List B ,[object Object],[object Object]

Precision@k and Recall@k ,[object Object],[object Object],[object Object],Recall@k: Precision@k:

Precision@k: Example ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],List A ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],List B

Recall@k: Example ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],List A ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],List B

“ Interpolated” Precision ,[object Object],[object Object],[object Object],[object Object],Interpolated precision@ recall level r = max { precision@k : k >= k(r) }

Precision vs. Recall: Example ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],List A ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],List B

Query Languages: Keyword-Based ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Query Languages: Pattern Matching ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Text Processing ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Inverted Index Michael 1 Jordan 2 , the 3 author 4 of 5 “graphical 6 models 7 ”, is 8 a 9 professor 10 at 11 U.C. 12 Berkeley 13 . The 1 famous 2 NBA 3 legend 4 Michael 5 Jordan 6 liked 7 to 8 date 9 models 10 . d 1 d 2 author: (d 1 ,4) berkeley: (d 1 ,13) date: (d 2 ,9) famous: (d 2 , 2) graphical: (d 1 ,6) jordan: (d 1 ,2), (d 2 ,6) legend: (d 2 ,4) like: (d 2 ,7) michael: (d 1 ,1), (d 2 ,5) model: (d 1 ,7), (d 2 ,10) nba: (d 2 ,3) professor: (d 1 ,10) uc: (d 1 ,12) Vocabulary Postings

Inverted Index Structure Vocabulary File term1 term2 … Postings File postings list 1 postings list 2 … ,[object Object],[object Object]

Searching an Inverted Index ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Search Optimization ,[object Object],[object Object],[object Object]

Index Construction ,[object Object],[object Object],[object Object]

Index Maintenance ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Index Maintenance ,[object Object],[object Object],[object Object],[object Object]

Index Compression ,[object Object],[object Object],[object Object],michael: (1000007,5), (1000009,12), (1000013,77), (1000035,88),… michael: (1000007,5), (2,12), (4,77), (22,88),…

Variable Length Encodings ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Slides

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (6)

Semelhante a Slides

Semelhante a Slides (20)

Mais de butest

Mais de butest (20)

Slides