最近の強化学習の研究の流れ

7. Qo
(s,a) = r(s,a)+γ max
a'
Qo
(s',a')
Qo
L = (r(s,a)+γ max
a'
Qθ
o
(s',a')−Qθ
o
(s,a))2

10. ∇θ J = ∇θ Eπθ
[ γ τ
Rτ ]
τ =0
∞
∑
= ∇θ P( ′s | st ,a)πθ (a | st ) γ τ
Rτ
τ =0
∞
∑
a
∑
′s
∑
= P( ′s | st ,a)∇θπθ (a | st ) γ τ
Rτ
τ =0
∞
∑
a
∑
′s
∑
= P( ′s | st ,a)πθ (a | st )
∇θπθ (a | st )
πθ (a | st )
γ τ
Rτ
τ =0
∞
∑
a
∑
′s
∑
= P( ′s | st ,a)πθ (a | st )∇θ log(πθ (a | st )) γ τ
Rτ
τ =0
∞
∑
a
∑
′s
∑
= Eπθ
[∇θ log(πθ (a | st )) γ τ
Rτ ]
τ =0
∞
∑

11. Eπθ
[∇θ log(πθ (a | st )) γ τ
Rτ ]
τ =0
∞
∑
=
1
M
∇θ log(πθ (ai
T
| si
T
))( γ τ
Rτ
T
)
τ =0
∞
∑
i
∑
T
∑
T = s0
T
,a0
T
,r0
T
,!sn
T
,an
T
,rn
T

12. 1
M
∇θ log(πθ (ai
T
| si
T
))( γ τ
Rτ
T
τ =0
∞
∑
i
∑
T
∑ )

13. 1
M
∇θ log(πθ (ai
T
| si
T
))( γ τ
Rτ
T
τ =0
∞
∑
i
∑
T
∑ )
1
M
∇θ log(πθ (ai
T
| si
T
))
i
∑
T
∑ A(si
T
,ai
T
)

21. Qaux
(a,i, j)
LQ = E[(Rt:t+n +γ n
max
a'
Q(s',a';θ−
)−Q(s,a;θ))2
]

22. LVR = Eπ [(Rt:t+n +γ n
V(st+n+1,θ−
)−V(st ,θ))2
]

29. Ep[ f (x)] = p(x) f (x)x∑
Eq[ f (x)] = q(x) f (x)x∑
=
q(x)
p(x)
p(x) f (x)x∑
= p(x)
q(x)
p(x)
f (x)x∑
= Ep[
q(x)
p(x)
f (x)]

36. LA3C = Lπ + LV − Es∼π [αH(π(⋅| s))]

37. !Qπ
(s,a) = α(log(π(s,a)+ Hπ
(s))+Vπ
(s)

43. Q∗
(s,a) = r(s,a)+γτ log exp(Q∗
(s',a') /τ )a'∑
Q∗

44. V∗
(s) = −τ logπ∗
(a | s)+ r(s,a)+γV∗
(s')
−V∗
(s1)+γ t−1
V∗
(st )+ R(s1:t )−τG(s1:t ,π∗
) = 0
R(sm:n ) = γ i
r(sm+i ,am+i )
i=0
n−m−1
∑ G(sm:n,π) = γ i
logπ(am+i | sm+i )
i=0
n−m−1
∑

45. Cθ,φ (s1:t ) = −Vφ (s1)+γ t−1
Vφ (st )+ R(s1:t )−τG(s1:t ,πθ )
Δθ ∝Cθ,φ (s1:t )∇θG(s1:t ,πθ )
Δφ ∝Cθ,φ (s1:t )(∇φVφ (s1)− ∇φγ t−1
Vφ (st ))