Build Message Bot With Neural Network
•
7 gostaram•1,171 visualizações
The document discusses building message bots using neural networks. It provides an overview of recurrent neural networks (RNNs) including long short-term memory networks (LSTMs) and gated recurrent units (GRUs). Applications mentioned include machine translation, question answering, conversations, image captioning, and troubleshooting chatbots. Sequence-to-sequence models with attention and memory networks are also summarized as approaches for natural language processing tasks.
Recomendados
Recomendados
Mais conteúdo relacionado
Destaque
Destaque (20)
Semelhante a Build Message Bot With Neural Network
Semelhante a Build Message Bot With Neural Network (20)
Último
Último (20)
Build Message Bot With Neural Network
- 1. Build Message Bot With Neural Network 2016/08/22 Billy Yang
- 2. Register New Bot • The callback URL must use HTTPS • Channel ID, Secret and MID • Server IP Whitelist • Dummy friend for testing. • Trial bot can send/receive message. • Business account support Mission Sticker, Link Message, Make browsing your website from the LINE app seamless for users.
- 3. Mission Sticker Example : Tokyo Disney http://official-blog.line.me/tw-biz/archives/57396837.html
- 4. Send/Receive Message Send message method=POST url=https://trialbot-api.line.me/event Receive message method=POST url=YOUR CALLBACK URL {'result': [ {'content': { 'contentType': 1, 'createdTime': 1469373366186, 'deliveredTime': 0, 'from': USER_MID, 'id': ‘4656604667064’, 'text': 'Hello LINE Bot API', 'to': [MID], 'toType': 1},…..
- 6. Message Bot • ELIZA(1966, http://nlp-addiction.com/eliza/) • seemed to be able to fool users into believing that they were conversing with a real human • recognition of cue words or phrases in the input, and the output of corresponding pre-prepared or pre-programmed responses
- 7. Message Bot " " Plurk Plurk YAHOO . .
- 8. Message Bot Viv Techcrunch demo Microsoft Tay
- 9. Recurrent Neural Network(RNN) ht-1 + xt wi wh sigmoid ht wo softmax yt RNN RNN x y h RNN x0 y0 RNN x1 y1 RNN x2 y2 h0 h1 time
- 11. Long Short-Term Memory Network(LSTM) Output Layer Input Layer Forget Layermemory LSTM signal signal signal x y
- 12. LSTM : Forget Layer ht-1 xt ht yt LSTM Ct-1 x Ct Forget Gate w1 w2
- 13. LSTM : Input Layer ht-1 xt ht yt LSTM tanh w4w3 + w6w5 Ct-1 x x Ct Forget Gate w1 w2 Input Gate
- 14. LSTM : Forget Layer + Input Layer ht-1 xt ht yt LSTM tanh w4w3 + w6w5 Ct-1 x x + Ct Forget Gate w1 w2 Input Gate
- 15. Long Short-Term Memory Network(LSTM) Output Gate Input Gate Forget Gatememory LSTM signal signal signal
- 16. LSTM : Forget Layer + Input Layer ht-1 xt ht yt LSTM tanh w4w3 + w6w5 Ct-1 x x + Ct Forget Gate w1 w2 Input Gate
- 17. LSTM : Output Layer ht-1 xt ht yt LSTM tanh w4w3 + w6w5 Ct-1 x x + Ct w8w7 xtanh Forget Gate w1 w2 Input Gate Output Gate
- 18. LSTM : All Layers ht-1 xt ht yt LSTM tanh w4w3 + w6w5 Ct-1 x x + Ct w8w7 xtanh softmax Forget Gate w1 w2 Input Gate Output Gate
- 19. LSTM : All Layers ht-1 + xt w2w1 sigmoid ht yt LSTM sigmoid tanh + w4w3 + w6w5 Ct-1 x x + Ct + w8w7 sigmoid xtanh softmax
- 20. Long Short-Term Memory Network(LSTM) LSTM x y h,c LSTM x0 y0 LSTM x1 y1 LSTM x2 y2 h0 h1 time c0 c1 LSTM reduce vanishing gradient problem
- 21. Recurrent Neural Network(RNN) ht-1 + xt wi wh sigmoid ht wo softmax yt RNN RNN x y h RNN x0 y0 RNN x1 y1 RNN x2 y2 h0 h1 time
- 22. Gated Recurrent Network(GRU) ht-1 xt yt GRU tanh w4w3 + w6w5 x x + ht softmax Update Gate w1 w2 Reset Gate x 1-
- 23. GRU : Don’t Update Memory! ht-1 xt yt GRU tanh w4w3 + w6w5 x x + ht softmax Don’t Update w1 w2 Reset Gate x 1- 0 1
- 24. GRU : Update Memory ht-1 xt yt GRU tanh w4w3 + w6w5 x x + ht softmax Update w1 w2 Don’t Reset x 1- 1 0 1
- 25. GRU : Reset Memory ht-1 xt yt GRU tanh w4w3 + w6w5 x x + ht softmax Update w1 w2 Reset x 1- 1 0 0 0
- 27. Show and Tell: A Neural Image Caption Generator Google Inception Network + Batch Normalization
- 29. Sequence to sequence Neural Network * On the WMT’14 English to French translation task,we obtained a BLEU score of 34.81 * This is by far the best result achieved by direct translation with large neural net-works. For comparison, the BLEU score of an SMT baseline on this dataset is 33.30 venturebeat.com “Google Translate could become more accurate soon thanks to deep learning” 2016/03 engadget.com “Google is using neural network to improve Translate” 2016/03
- 30. Sequence to sequence Neural Network * Each box in the picture most commonly a GRU cell or an LSTM cell. RNN x RNN RNN RNN time RNN y RNN EOF encoder decoder Good RNN Job context vector
- 31. Sequence to sequence Neural Network • Reversing source words improve test BLEU scores time EOF Good Job context vector layer 1 layer 3 layer 4 32M 32M • Made sure that all sentences in a minibatch are roughly of the same length
- 32. Sequence to sequence Neural Network • We can use this model for machine translation, question/ answering, and conversations without major changes in the architecture • A potential issue with this encoder–decoder approach is that a neural network needs to be able to compress all the necessary information of a source sentence into a fixed- length vector. This may make it difficult for the neural network to cope with long sentences,
- 33. Application . IT Helpdesk Troubleshooting dataset Google Inbox App Smart Reply announce at 2015/11 Machine:hi Human:hello Machine:could you please let me know what are the operating systems you are using ? Human:i am using linux Machine:ok it was in the ssl vpn on which you stated it was broken …. Image Recognition announce at 2015/07, Not Seq2Seq
- 35. Sequence to sequence Neural Network+Attention(RNNSearch) RNN x0 RNN x1 RNN x2 h0 f h1 f RNN x2 RNN x1 RNN x0 h0 bh1 b time RNN EOF Good RNN Job [h1 b,h1 f] [h0 b,h0 f] c0 c1 + + a00 a01 a10 a11 context vector annotation bi-directional RNN
- 36. Facebook Research : Question/Answering * All NLP/AI tasks can be reduced to question answering * The bAbI project(Artificial Tasks for Artificial Intelligence)
- 37. End-to-End Memory Network : Single Hop input story weighted sum{c1,c2,c3} {a1,a2,a3} softmax {0.2, 0.1, 0.7} dot product input question WC WA ? u1 WB +o1 u2 softmax WO { :0.9, :0.1} constructing memory vectors with Bag-of-Words m1=V +V +V +V Memory Module Controller
- 38. weighted sum End-to-End Memory Network : Two Hops input story softmax dot product input question WC2 WA1 ? u1 WB +o2 u3 softmax WO { :0.9, :0.1} Memory Module Controller +o1 u2 weighted sum softmax dot product weight sharing methods 1. Ak+1=Ck, 2. A1=A2=…=AK and C1=C2=…=CK WA2 WC1 http://cims.nyu.edu/~sainbar/memnn_nips_pdf.pdf
- 39. Memory Network Compare with RNNSearch, memory network can be considered as an attention model with multiple hops(lookup) and out of order
- 41. Ask Me Anything: Dynamic Memory Networks for Natural Language Processing http://cs224d.stanford.edu/lectures/CS224d-Lecture17.pdf
- 42. Reference Sequence to Sequence Learning with Neural Networks. Ilya Sutskever(OpenAI Research Director, Andrew Ng PostDoc), Oriol Vinyals & Quoc V. Le 2014/09 Neural Machine Translation by Jointly Learning to Align and Translate. Dzmitry Bahdanau(Theano Contributor), Kyunghyun Cho &Yoshua Bengio 2014/09 A Neural Conversational Model Oriol Vinyals(Tensorflow Contributor) & Quoc Le 2015/01 Memory Networks Jason Weston, Sumit Chopra & Antoine Bordes 2014/10 End-To-End Memory Networks Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston & Rob Fergus 2015/01
- 43. Thank You :)