1. 4.1 4.2
.
.
2010/10/12
4.1 4.2

2. AGENDA
1
2
.
3
.
4
.
. 4.1 4.2

3. AGENDA
1
2
.
3
.
4
.
. 4.1 4.2

4. 4
1
2
.
.
4.1 4.2

5. AGENDA
1
2
.
3
.
4
.
. 4.1 4.2

6. (classification categorization)
4.1 4.2

7. 2
SVM
4.1 4.2

8. AGENDA
1
2
.
3
.
4
.
. 4.1 4.2

9. d P(c|d) c∈C
P(c|d)
1
P(c)P(d|c)
P(c|d) =
P(d)
.
2 P(d) P(c)P(d|c)
c max
.
P(c)P(d|c)
c max = arg max
c P(d)
= arg max P(c)P(d|c)
c
. 4.1 4.2

10. P(d|c)
d
d
d
P(d|c)
d
-
-
4.1 4.2

11. AGENDA
1
2
.
3
.
4
.
. 4.1 4.2

12. -
P(d|c)
d
∏ δ
P(d|c) = pw,c (1 − pw,c )1−δw,d
w,d
w∈V
V
w
pc
P(c) pw,c pc 2
δ
pw,c (1 − pw,c )1−δw,d
w,d
c w d
4.1 4.2

13. -
∏ δ
P(c)P(d|c) = pc pw,c (1 − pw,c )1−δw,d
w,d
w∈V
c
pw,c
P(d|c)
P(d|c)
pw,c 1
4.1 4.2

14. D
D = {(d(1) , c(1) ), (d(2) , c(2) ), ..., (d|D| , c|D| )}
∑
log P( D) = log P(d, c)
(d,c)∈D
 
∑  ∏ δw,d
 

 
= log  pc


 pw,c (1 − pw,c )1−δw,d 



(d,c)∈D w∈V
 
∑ 
 ∑ 


log pc + 
= 

 (δw,d log pw,c + (1 − ww,d ) log(1 − pw,c ))



(d,c)∈D w∈V
∑ ∑∑ ∑∑
= N c log pc + Nw,c log pw,c + (N c − Nw,c ) log(1 − pw,c )
c c w∈V c w∈V
Nc : c
Nw,c : c w
4.1 4.2

15. pc
max . log P(D)
∑
s.t. pc = 1.
c
L(θ, λ)
∑ 


 


L(θ, λ) = log P(D) + λ 
 pc − 1



c
θ: { pw,c }winV,c∈C , {pc } c∈C
4.1 4.2

16. ∂L(θ, λ)
= 0
∂ pw,c
∂L(θ, λ)
= 0
∂ pc
∂L(θ, λ)
= 0
∂λ
4.1 4.2

17. 
∂L(θ, λ) ∂ ∑


∑∑
= 

 N c log pc + Nw,c log pw,c
∂pw,c ∂pw,c 
c c w∈V
∑∑ ∑ 

 

+ (N c − Nw,c ) log(1 − pw,c ) + λ 


 
pc − 1



c w∈V c
∂(1− pw,c )
Nw,c ∂ pw,c
= + (N c − Nw,c )
pw,c (1 − pw,c )
Nw,c (N c − Nw,c )
= −
pw,c 1 − pw,c

∂L(θ, λ) ∑
∂ 

∑∑
= 

 N c log pc + Nw,c log pw,c
∂pc 
∂pc c c w∈V
∑∑ ∑ 

 

+ 

(N c − Nw,c ) log(1 − pw,c ) + λ 
 
pc − 1



c w∈V c
Nc
= +λ
pc
4.1 4.2

18. pw,c
Nw,c (N c − Nw,c )
− = 0
pw,c 1 − pw,c
(1 − pw,c )Nw,c − pw,c (N c − Nw,c ) = 0
pw,c (N c − Nw,c + Nw,c ) = Nw,c
Nw,c
pw,c =
Nc
4.1 4.2

19. pc
Nc
+λ = 0
pc
Nc
pc = −
λ
∑
pc = 1
c
1∑
− Nc = 1
λ c
∑
λ = − Nc
c
Nc Nc
pc = − = ∑
λ c Nc
4.1 4.2

20. c w
pw,c =
c
c
pc =
4.1 4.2

21. 4.1
P 3
d(1) = ”good bad good good”
d(2) = ”exciting exciting”
d(3) = ”good good exciting boring”
N 3
d(4) = ”bad boring boring boring”
d(5) = ”bad good bad”
d(6) = ”bad bad boring exciting”
P N
4.1 4.2

22. 4.1
V = {bad, boring, exciting, good}
N P = 3, N N = 3, N bad,P = 1, N bad,N = 3,
N boring,P = 1, N boring,N = 2, Nexciting,P = 2, Nexciting,N = 1,
N good,P = 2, N good,N = 1,
NP NN
pP = N P +N N
= 3+3 = 0.50
3
pN = N p+NN = 3+3 = 0.50
3
N bad,P N bad,N
pbad,P = N P = 1 = 0.33 3
pbad,N = NN = 3 = 1.00 3
N boring,P N bof ing,N
pboring,P = N P = 3 = 0.33 1
pbof ing,N = NN = 2 = 3
0.67
Nexciting,P
pexciting,P = N P = 2 = 0.67 3
Nexciting,N 1
pexciting,N = = 3
= 0.33
N good,P N good,N
pgood,P = N P = 2 = 0.67 3
pgood,N = NN = 1 = 0.33 3
4.1 4.2

23. 4.2
4.1 d
d = ”good good bad boring”
pP pd|P pN pd|N
pP pd|P = pP × pbad,P × pboring,P × (1 − pexciting,P ) × pgood,P
= 0.5 × 0.33 × 0.33 × (1 − 0.67) × 0.67 = 0.012
pN pd|N = pN × pbad,N × pboring,N × (1 − pexciting,N ) × pgood,N
= 0.5 × 1.00 × 0.67 × (1 − 0.33) × 0.33 = 0.074
4.1 d N
4.1 4.2

24. 4.2
4.1 d
d = ”good good bad boring”
pP pd|P pN pd|N
pP pd|P = pP × pbad,P × pboring,P × (1 − pexciting,P ) × pgood,P
= 0.5 × 0.33 × 0.33 × (1 − 0.67) × 0.67 = 0.012
pN pd|N = pN × pbad,N × pboring,N × (1 − pexciting,N ) × pgood,N
= 0.5 × 1.00 × 0.67times(1 − 0.33) × 0.33 = 0.074
4.1 d N
4.1 4.2

25. 4.3
4.1 d(1)
d(1) = ”good bad good good fine”
d
d = ”bad bad boring boring fine”
4.1 4.2

26. 4.3
“fine” fine
N f ine,P N f ine,N
p f ine,P = NP
= 1
3
= 0.33 p f ine,N = NN
= 0
3
= 0.00
pP pd|P = pP × pbad,P × pboring,P × (1 − pexciting,P ) × p f ine,P × (1 − pgood,P )
= 0.5 × 0.33 × 0.33 × (1 − 0.67) × 0.33 × (1 − 0.67) = 0.002
pN pd|N = pN × pbad,N × pboring,N × (1 − pexciting,N ) × p f ine,N × (1 − pgood,N )
= 0.5 × 1.00 × 0.67 × (1 − 0.33) × 0.00 × 0.67 = 0.00
P
4.1 4.2

27. 4.3
d “bad” ”boring” ”good”
”exciting” P
p f ine,N = 0.00
N pN pd|N = 0.00
0
MAP
4.1 4.2

28. MAP
0.00
MAP
∏  ∏ 
∑

  
×
α−1 
( α−1 )
α−1 
log P(θ) + log P(D) = 


log   
pc   
  pw,c (1 − pw,c )

+

 log P(d, c) + (const.)
 
c w,c (d,c)∈D
∑ ∑( )
= (α − 1) log pc + (α − 1) log pw,c + log(1 − pw,c )
c w,c
 
∑  ∏ δw,d
 

 
+ log  pc


 ( pw,c (1 − pw,c )1−δw,d ) + (const.)



(d,c)∈ D w∈V
∑
c p(c) = 1
4.1 4.2

29. MAP
∑ 


 


L(θ, λ) = log P(θ) + log P( D) + λ 
 pc − 1



c
∂L(θ, λ) (α − 1) (α − 1) Nw,c N c − Nw,c
= +− + −
∂ pw,c pw,c 1 − pw,c pw,c 1 − pw,c
∂L(θ, λ) (α − 1) N c
= + +λ
∂pc pc pc
4.1 4.2

30. MAP
∑
0 c pc = 1
Nw,c + (α − 1)
pw,c =
Nc + 2
Nc + 1
pc = ∑
c N c + |C|
α
4.1 4.2

31. 4.4
4.3
MAP
α=1
P 3
d(1) = ”good bad good good fine”
d(2) = ”exciting exciting”
d(3) = ”good good exciting boring”
N 3
d(4) = ”bad boring boring boring”
d(5)
= ”bad good bad”
d(6) = ”bad bad boring exciting”
4.1 4.2

32. 4.4
Table:
MAP MAP
pP 0.50 0.50 pN 0.50 0.50
pbad,P 0.33 0.40 pbad,N 1.00 0.80
pboring,P 0.33 0.40 pboring,N 0.67 0.60
pexciting,P 0.67 0.60 pexciting,N 0.33 0.40
p f ine,P 0.33 0.40 p f ine,N 0.00 0.20
pgood,P 0.67 0.60 pgood,N 0.33 0.40
MAP
smoothing
MAP
4.1 4.2

33. AGENDA
1
2
.
3
.
4
.
. 4.1 4.2

34. V 1 |d|
P(d|c)
d w nw,d
 ∑  (∑ n )! ∏

 
 w w,d
P(d|c) = P  K =
 nw,d  ∏
 nw,d

 
 qw,c
w w∈V nw,d ! w∈V
K:
( ∑ ) ∑
P K = w nw,d : w nw,d
4.1 4.2

35. c
∑  (∑ n )! ∏

 
 w w,d
pc P 
 nw,d  ∏
 nw,d
P(c)P(d|c) = 
 
 qw,c
w w∈V nw,d ! w∈V
∑  (∑ n )! ∏

 
 w w,d
arg max P(c)P(d|c) = arg max pc P 
 nw,d  ∏
 n

 
 q w,d
c c
w w∈V nw,d ! w∈V w,c
∏
nw
= arg max pc qw,c
c
w∈V
∏ nw
c pc w∈V qw,c
4.1 4.2

36. 2
4.1 4.2

37. ∑
log P( D) = log P(d, c)
(d,c)∈ D
 
∑  p(|d|)|d|!
 ∏ n  

 w,d 
= log  ∏

 pc qw,c 


(d,c)∈ D w∈Vn ! w,d w∈V
∑ P(|d|)|d|! ∑ ∑ ∑
= log ∏ + log pc + nw,d log qw,c
(d,c)∈ D w∈V nw,d ! (d,c)∈ D (d,c)∈D w∈V
∑ P(|d|)|d|! ∑ ∑∑
= log ∏ + log nc pc + nw,c log qw,c
(d,c)∈ D w∈V nw,d ! c c w∈V
max. log P( D)
∑
s.t. pc = 1.
c∈C
∑
qw,c = 1; ∀c ∈ C
w∈V
4.1 4.2

38.    
∑ ∑
 
 ∑
 


   
L(θ, β, γ) = log P(D) + βc 

 qw,c − 1 + γ 


 

 pc − 1



c∈C w∈V c∈C
∂L(θ, β, γ)
= 0
∂qw,c
∂L(θ, β, γ)
= 0
∂ pc
∂L(θ, β, γ)
= 0
∂β
∂L(θ, β, γ)
= 0
∂γ
4.1 4.2

39. 
∂L(θ, β, γ) ∂  ∑

 P(|d|)|d|! ∑ ∑∑
= 

 log ∏ + nc log pc +

 nw,c log qw,c
∂qw,c ∂qw,c (d,c)∈D w∈V nw,d ! c c w∈V

∑ ∑ ∑ 



βc ( −1) + γ( pc − 1)


c∈C w∈V c∈C
nw,c
= + βc = 0
qw,c
nw,c
qw,c =
βc
4.1 4.2

40. βc
∑
qw,c = 1
w∈V
1 ∑
nw,c = 1
β c w∈V
1
βc = ∑
w∈V nw,c
nw,c
qw,c = ∑
w nw,c
pc
4.1 4.2

41. c w
qw,c =
c
c w
pw,c =
c
4.1 4.2

42. MAP
0.00
MAP
MAP
∏  ∏ 
∑

  
  

log P(θ) + log P(D) ∝ log 

  
pα−1  × 
  

qα−1  +

 c   w,c 
 log P(d, c)
c w,c (d,c)∈D
   
∑


∑ 


∑  P(|d|)|d|!


∏ n 
w,d 

= 
(α − 1) 
 log pc + log qw,c  +

 log  ∏

 pc qw,c 

   n ! 
c w,c (d,c)∈D w∈V w,d w∈V
∑ ∑
c p(c) = 1 w qw,c = 1
4.1 4.2

43. MAP
L(θ, β, γ) = log P(θ) + log P(D)
   
∑ ∑  
 ∑
 

   
+ βc 


 pw,c − 1 + γ 


 

 pc − 1



c∈C w∈V c∈C
∂L(θ, β, γ) (α − 1) nw,c
= + + βc
∂qw,c qw,c qw,c
∑
0 w∈V qw,c = 1
nw,c + (α − 1)
qw,c = ∑
w nw,c + |W|(α − 1)
4.1 4.2

44. AGENDA
1
2
.
3
.
4
.
. 4.1 4.2

45. d
MAP
4.1 4.2

46. ( )
Ml for nlp chapter 4
4.1 4.2

47. 4.1 4.2