sadf

CS60057 Speech &Natural Language Processing Autumn 2007 Lecture 5 2 August 2007

WORDS The Building Blocks of Language

[object Object],[object Object]

Tokens, Types and Texts ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Extracting text from the Web ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Extracting text from NLTK Corpora ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Brown Corpus ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Corpus Linguistics ,[object Object],[object Object],[object Object],[object Object],[object Object]

What’s a word? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Another example ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Some Useful Empirical Observations ,[object Object],[object Object],[object Object],[object Object],[object Object]

Common words in Tom Sawyer but words in NL have an uneven distribution…

Text properties (formalized) Sample word frequency data

Frequency of frequencies ,[object Object],[object Object],[object Object],[object Object],[object Object]

Zipf’s Law ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Predicting Occurrence Frequencies ,[object Object],[object Object],[object Object],[object Object],Fraction of words with frequency n is: Fraction of words appearing only once is therefore ½.

Explanations for Zipf’s Law ,[object Object]

Zipf’s First Law ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Zipf’s Second Law ,[object Object],[object Object],[object Object]

Zipf’s Third Law ,[object Object],[object Object],[object Object]

Zipf’s Law Impact on Language Analysis ,[object Object],[object Object]

Vocabulary Growth ,[object Object],[object Object],[object Object]

Heaps’ Law ,[object Object],[object Object],[object Object],[object Object]

Word counts are interesting... ,[object Object],[object Object],[object Object],[object Object],[object Object]

Zipf’s Law on Tom Saywer ,[object Object],[object Object],[object Object],[object Object]

Plot of Zipf’s Law ,[object Object],[object Object]

Plot of Zipf’s Law (con’t) ,[object Object],[object Object]

Zipf’s Law, so what? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

N-Grams and Corpus Linguistics

A bad language model N-grams & Language Modeling

What’s a Language Model ,[object Object],[object Object],[object Object]

What’s a language model for? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Next Word Prediction ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Human Word Prediction ,[object Object],[object Object],[object Object],[object Object],[object Object]

Claim ,[object Object],[object Object]

Applications ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Overview ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Simple N-Grams ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

N-grams ,[object Object],[object Object],[object Object],[object Object]

Computing the Probability of a Word Sequence ,[object Object],[object Object],[object Object],[object Object],[object Object]

Bigram Model ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Using N-Grams ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

The n -gram Approximation ,[object Object],[object Object],[object Object],[object Object],[object Object]

n- grams, continued ,[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],[object Object],[object Object]

N-grams for Language Generation ,[object Object],Unigram: 5. …Here words are chosen independently but with their appropriate frequencies. REPRESENTING AND SPEEDILY IS AN GOOD APT OR COME CAN DIFFERENT NATURAL HERE HE THE A IN CAME THE TO OF TO EXPERT GRAY COME TO FURNISHES THE LINE MESSAGE HAD BE THESE. Bigram: 6. Second-order word approximation. The word transition probabilities are correct but no further structure is included. THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD FOR THE LETTERS THAT THE TIME OF WHO EVER TOLD THE PROBLEM FOR AN UNEXPECTED.

N-Gram Models of Language ,[object Object],[object Object],[object Object],[object Object],[object Object]

Counting Words in Corpora ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Terminology ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Corpora ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Training and Testing ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

A Simple Example ,[object Object],[object Object]

A Bigram Grammar Fragment from BERP .001 Eat British .03 Eat today .007 Eat dessert .04 Eat Indian .01 Eat tomorrow .04 Eat a .02 Eat Mexican .04 Eat at .02 Eat Chinese .05 Eat dinner .02 Eat in .06 Eat lunch .03 Eat breakfast .06 Eat some .03 Eat Thai .16 Eat on

.01 British lunch .05 Want a .01 British cuisine .65 Want to .15 British restaurant .04 I have .60 British food .08 I don’t .02 To be .29 I would .09 To spend .32 I want .14 To have .02 <start> I’m .26 To eat .04 <start> Tell .01 Want Thai .06 <start> I’d .04 Want some .25 <start> I

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

BERP Bigram Counts 0 1 0 0 0 0 4 Lunch 0 0 0 0 17 0 19 Food 1 120 0 0 0 0 2 Chinese 52 2 19 0 2 0 0 Eat 12 0 3 860 10 0 3 To 6 8 6 0 786 0 3 Want 0 0 0 13 0 1087 8 I lunch Food Chinese Eat To Want I

BERP Bigram Probabilities ,[object Object],[object Object],[object Object],[object Object],[object Object],459 1506 213 938 3256 1215 3437 Lunch Food Chinese Eat To Want I

What do we learn about the language? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object]

Approximating Shakespeare ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

N-Gram Training Sensitivity ,[object Object],[object Object]

Smoothing Techniques ,[object Object],[object Object],[object Object]

Add-one Smoothing ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],[object Object],Witten-Bell Discounting

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Good-Turing Discounting

Backoff methods (e.g. Katz ‘87) ,[object Object],[object Object],[object Object],[object Object],[object Object]

Summary ,[object Object],[object Object],[object Object],[object Object]

sadf

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (20)

Semelhante a sadf

Semelhante a sadf (20)

Último

Último (20)

sadf