Enviar pesquisa
Carregar
機械学習ゼミ: Area attenttion
•
Transferir como PPTX, PDF
•
0 gostou
•
558 visualizações
K
KCS Keio Computer Society
Seguir
機械学習ゼミにおいて発表されたプレゼンテーション資料
Leia menos
Leia mais
Dados e análise
Vista de apresentação de diapositivos
Denunciar
Compartilhar
Vista de apresentação de diapositivos
Denunciar
Compartilhar
1 de 12
Baixar agora
Recomendados
[DL輪読会]Z-Forcing: Training Stochastic Recurrent Networks (NIPS2017)
[DL輪読会]Z-Forcing: Training Stochastic Recurrent Networks (NIPS2017)
Deep Learning JP
Large scale gan training for high fidelity natural
Large scale gan training for high fidelity natural
KCS Keio Computer Society
Imagenet trained cnns-are_biased_towards
Imagenet trained cnns-are_biased_towards
KCS Keio Computer Society
機械学習ゼミ 2018/10/17
機械学習ゼミ 2018/10/17
KCS Keio Computer Society
機械学習ゼミ2018 06 15
機械学習ゼミ2018 06 15
KCS Keio Computer Society
Control by deep learning
Control by deep learning
KCS Keio Computer Society
深層学習 第6章
深層学習 第6章
KCS Keio Computer Society
Vector-Based navigation using grid-like representations in artificial agents
Vector-Based navigation using grid-like representations in artificial agents
KCS Keio Computer Society
Recomendados
[DL輪読会]Z-Forcing: Training Stochastic Recurrent Networks (NIPS2017)
[DL輪読会]Z-Forcing: Training Stochastic Recurrent Networks (NIPS2017)
Deep Learning JP
Large scale gan training for high fidelity natural
Large scale gan training for high fidelity natural
KCS Keio Computer Society
Imagenet trained cnns-are_biased_towards
Imagenet trained cnns-are_biased_towards
KCS Keio Computer Society
機械学習ゼミ 2018/10/17
機械学習ゼミ 2018/10/17
KCS Keio Computer Society
機械学習ゼミ2018 06 15
機械学習ゼミ2018 06 15
KCS Keio Computer Society
Control by deep learning
Control by deep learning
KCS Keio Computer Society
深層学習 第6章
深層学習 第6章
KCS Keio Computer Society
Vector-Based navigation using grid-like representations in artificial agents
Vector-Based navigation using grid-like representations in artificial agents
KCS Keio Computer Society
文章生成の未解決問題
文章生成の未解決問題
KCS Keio Computer Society
Word2vec alpha
Word2vec alpha
KCS Keio Computer Society
テンソル代数
テンソル代数
KCS Keio Computer Society
Hindsight experience replay
Hindsight experience replay
KCS Keio Computer Society
Kml 輪読514
Kml 輪読514
KCS Keio Computer Society
ゼロから作るDeepLearning 5章 輪読
ゼロから作るDeepLearning 5章 輪読
KCS Keio Computer Society
ELBO型VAEのダメなところ
ELBO型VAEのダメなところ
KCS Keio Computer Society
ゼロから作るDeepLearning 2~3章 輪読
ゼロから作るDeepLearning 2~3章 輪読
KCS Keio Computer Society
ゼロから作るDeepLearning 4章 輪読
ゼロから作るDeepLearning 4章 輪読
KCS Keio Computer Society
Soft Actor Critic 解説
Soft Actor Critic 解説
KCS Keio Computer Society
ゼロから作るDeepLearning 3.3~3.6章 輪読
ゼロから作るDeepLearning 3.3~3.6章 輪読
KCS Keio Computer Society
[論文略説]Stochastic Thermodynamics Interpretation of Information Geometry
[論文略説]Stochastic Thermodynamics Interpretation of Information Geometry
KCS Keio Computer Society
Graph Convolutional Network 概説
Graph Convolutional Network 概説
KCS Keio Computer Society
AIを作ることと理解すること
AIを作ることと理解すること
KCS Keio Computer Society
Random forest の解説
Random forest の解説
KCS Keio Computer Society
Outrageously Large Neural Networks:The Sparsely-Gated Mixture-of-Experts Laye...
Outrageously Large Neural Networks:The Sparsely-Gated Mixture-of-Experts Laye...
KCS Keio Computer Society
ハミルトニアンモンテカルロ法についての説明
ハミルトニアンモンテカルロ法についての説明
KCS Keio Computer Society
AI班ハードウェア領域
AI班ハードウェア領域
KCS Keio Computer Society
Phonetic Posteriorgrams for Many-to-One Voice Conversion without Parallel Dat...
Phonetic Posteriorgrams for Many-to-One Voice Conversion without Parallel Dat...
KCS Keio Computer Society
U-Net: Convolutional Networks for Biomedical Image Segmentationの紹介
U-Net: Convolutional Networks for Biomedical Image Segmentationの紹介
KCS Keio Computer Society
Mais conteúdo relacionado
Mais de KCS Keio Computer Society
文章生成の未解決問題
文章生成の未解決問題
KCS Keio Computer Society
Word2vec alpha
Word2vec alpha
KCS Keio Computer Society
テンソル代数
テンソル代数
KCS Keio Computer Society
Hindsight experience replay
Hindsight experience replay
KCS Keio Computer Society
Kml 輪読514
Kml 輪読514
KCS Keio Computer Society
ゼロから作るDeepLearning 5章 輪読
ゼロから作るDeepLearning 5章 輪読
KCS Keio Computer Society
ELBO型VAEのダメなところ
ELBO型VAEのダメなところ
KCS Keio Computer Society
ゼロから作るDeepLearning 2~3章 輪読
ゼロから作るDeepLearning 2~3章 輪読
KCS Keio Computer Society
ゼロから作るDeepLearning 4章 輪読
ゼロから作るDeepLearning 4章 輪読
KCS Keio Computer Society
Soft Actor Critic 解説
Soft Actor Critic 解説
KCS Keio Computer Society
ゼロから作るDeepLearning 3.3~3.6章 輪読
ゼロから作るDeepLearning 3.3~3.6章 輪読
KCS Keio Computer Society
[論文略説]Stochastic Thermodynamics Interpretation of Information Geometry
[論文略説]Stochastic Thermodynamics Interpretation of Information Geometry
KCS Keio Computer Society
Graph Convolutional Network 概説
Graph Convolutional Network 概説
KCS Keio Computer Society
AIを作ることと理解すること
AIを作ることと理解すること
KCS Keio Computer Society
Random forest の解説
Random forest の解説
KCS Keio Computer Society
Outrageously Large Neural Networks:The Sparsely-Gated Mixture-of-Experts Laye...
Outrageously Large Neural Networks:The Sparsely-Gated Mixture-of-Experts Laye...
KCS Keio Computer Society
ハミルトニアンモンテカルロ法についての説明
ハミルトニアンモンテカルロ法についての説明
KCS Keio Computer Society
AI班ハードウェア領域
AI班ハードウェア領域
KCS Keio Computer Society
Phonetic Posteriorgrams for Many-to-One Voice Conversion without Parallel Dat...
Phonetic Posteriorgrams for Many-to-One Voice Conversion without Parallel Dat...
KCS Keio Computer Society
U-Net: Convolutional Networks for Biomedical Image Segmentationの紹介
U-Net: Convolutional Networks for Biomedical Image Segmentationの紹介
KCS Keio Computer Society
Mais de KCS Keio Computer Society
(20)
文章生成の未解決問題
文章生成の未解決問題
Word2vec alpha
Word2vec alpha
テンソル代数
テンソル代数
Hindsight experience replay
Hindsight experience replay
Kml 輪読514
Kml 輪読514
ゼロから作るDeepLearning 5章 輪読
ゼロから作るDeepLearning 5章 輪読
ELBO型VAEのダメなところ
ELBO型VAEのダメなところ
ゼロから作るDeepLearning 2~3章 輪読
ゼロから作るDeepLearning 2~3章 輪読
ゼロから作るDeepLearning 4章 輪読
ゼロから作るDeepLearning 4章 輪読
Soft Actor Critic 解説
Soft Actor Critic 解説
ゼロから作るDeepLearning 3.3~3.6章 輪読
ゼロから作るDeepLearning 3.3~3.6章 輪読
[論文略説]Stochastic Thermodynamics Interpretation of Information Geometry
[論文略説]Stochastic Thermodynamics Interpretation of Information Geometry
Graph Convolutional Network 概説
Graph Convolutional Network 概説
AIを作ることと理解すること
AIを作ることと理解すること
Random forest の解説
Random forest の解説
Outrageously Large Neural Networks:The Sparsely-Gated Mixture-of-Experts Laye...
Outrageously Large Neural Networks:The Sparsely-Gated Mixture-of-Experts Laye...
ハミルトニアンモンテカルロ法についての説明
ハミルトニアンモンテカルロ法についての説明
AI班ハードウェア領域
AI班ハードウェア領域
Phonetic Posteriorgrams for Many-to-One Voice Conversion without Parallel Dat...
Phonetic Posteriorgrams for Many-to-One Voice Conversion without Parallel Dat...
U-Net: Convolutional Networks for Biomedical Image Segmentationの紹介
U-Net: Convolutional Networks for Biomedical Image Segmentationの紹介
機械学習ゼミ: Area attenttion
1.
Area attention @UMU____ 20181028 arXiv:1810.10126
2.
Significance • Attentionのfocusを「single-item方式」から「隣 接しているmulti-itemを一括で参照する方式に 変えることで,SOTAを達成
3.
Background: Attention • 辞書のようなNNを構成し,クエリで(辞書を)引く ℎ
𝑞 (クエリ側) ℎ 𝑑 (辞書側) (𝑘𝑖, 𝑣𝑖) 𝑞 𝑓𝑞(⋅) 𝑓𝑘,𝑣(⋅) 𝑎𝑖:𝑞と𝑘𝑖の類似度のようなもの 総和が1(Attention) Attenttion出力 イメージ:ki=qであればviが出てくる辞書を微分可能にした
4.
Background: Attention • 𝑓𝑎𝑡𝑡
𝑞, 𝑘 について [Luong et al., 2015] 𝑓𝑎𝑡𝑡 𝑞, 𝑘 = 𝑞 ⋅ 𝑘 [Bahdanau et al., 2014] 𝑓𝑎𝑡𝑡 𝑞, 𝑘 = 𝑊1 𝑞 + 𝑊2 𝑘 + 𝑏 𝑊, 𝑏 𝑡𝑟𝑎𝑖𝑛𝑎𝑏𝑙𝑒
5.
Background: Problem of
Attention • 普通のAttentionの問題点:single item focus • 複数のitemにattentionできないので表現力が制限 ℎ 𝑞 (クエリ側) ℎ 𝑑 (辞書側) (𝑘𝑖, 𝑣𝑖) 𝑞 𝑓𝑞(⋅) 𝑓𝑘,𝑣(⋅) クエリがq一つ:single item focus
6.
Background: Problem of
Attention • 普通のAttentionの問題点:single item focus →multi item focusにする [Vaswani et al., 2017] Multi head attention ℎ 𝑞 (クエリ側) ℎ 𝑑 (辞書側) (𝑘𝑖, 𝑣𝑖) 𝑞 𝑓𝑞(⋅) 𝑓𝑘,𝑣(⋅) ℎ 𝑞 (クエリ側) ℎ 𝑑 (辞書側) (𝑘𝑖, 𝑣𝑖) 𝑞 𝑓𝑞(⋅) 𝑓𝑘,𝑣(⋅) ℎ 𝑞 (クエリ側) ℎ 𝑑 (辞書側) (𝑘𝑖, 𝑣𝑖) 𝑞 𝑓𝑞(⋅) 𝑓𝑘,𝑣(⋅)
7.
Background: Problem of
Attention • 普通のAttentionの問題点:single item focus →multi item focusにする [Pedersoli et al., 2016] areas of attention 画像の部分的な箇所をfocusするattention
8.
Method (本論文) Multi item
focusを導入 • Single item focus: 要素ひとつひとつが辞書の要素 • Multi item focus: 要素ひとつひとつ+2つづつ+3つづつ… ℎ 𝑑 (辞書側) h1 h2 h3 hN… ℎ 𝑑 (辞書側) h1 h2 h3 hN… h1と h2 h2と h3 h1と h2と h3 … …
9.
• Nつづつ?実際には キーの計算: バリューの計算: ・または, キーとして,平均だけでなく分散やArea面積を 入れたものを用いることもできる. Method 詳細 単に該当するitemのキーと バリューを平均するだけ
10.
Experiments: Neural Machine Translation(vs
Transformer) BLEU (character level) BLEU (token level)
11.
Experiments: Neural Machine Translation(vs
LSTM) Negative Log likelihood (character level)
12.
Experiments: Image captioning Test
acculary
Baixar agora