O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Chatbot的智慧與靈魂

20170427 iThome ChatBot Day

  • Entre para ver os comentários

Chatbot的智慧與靈魂

  1. 1. ChatBot 的智慧與靈魂 YUN-NUNG (VIVIAN) CHEN 陳縕儂 Apr 27th, 2017 @iThome #ChatBot Day HTTP://VIVIANCHEN.IDV.TW
  2. 2. Intelligent Assistants 2 Apple Siri (2011) Google Now (2012) Google Assistant (2016) Facebook M & Bot (2015) Google Home (2016) Microsoft Cortana (2014) Amazon Alexa/Echo (2014)
  3. 3. Why We Need?  Get things done  E.g. set up alarm/reminder, take note  Easy access to structured data, services and apps  E.g. find docs/photos/restaurants  Assist your daily schedule and routine  E.g. commute alerts to/from work  Be more productive in managing your work and personal life 3 “Hey Assistant”
  4. 4. Why Natural Language?  Global Digital Statistics (2015 January) 4 Global Population 7.21B Active Internet Users 3.01B Active Social Media Accounts 2.08B Active Unique Mobile Users 3.65B The more natural and convenient input of devices evolves towards speech.
  5. 5. Intelligent Assistant Architecture 5 Reactive Assistance ASR, LU, Dialog, LG, TTS Proactive Assistance Inferences, User Modeling, Suggestions Data Back-end Data Bases, Services and Client Signals Device/Service End-points (Phone, PC, Xbox, Web Browser, Messaging Apps) User Experience “restaurant suggestions”“call taxi”
  6. 6. Dialogue System / ChatBot  Dialogue systems are intelligent agents that are able to help users finish tasks more efficiently via spoken interactions.  Dialogue systems are being incorporated into various devices (smart-phones, smart TVs, in-car navigating system, etc). 6 JARVIS – Iron Man’s Personal Assistant Baymax – Personal Healthcare Companion Good dialogue systems assist users to access information conveniently and finish tasks efficiently.
  7. 7. App  Bot  A bot is responsible for a “single” domain, similar to an app 7 Seamless and automatic information transferring across domains  reduce duplicate information and interaction 愛食記 Map LINE Goal: Schedule a lunch with Vivian KKBOX
  8. 8. GUI v.s. CUI (Conversational UI) 8 https://github.com/enginebai/Movie-lol-android
  9. 9. GUI v.s. CUI (Conversational UI) 網站/APP內GUI 即時通訊內的CUI 情境 探索式,使用者沒有特定目標 搜尋式,使用者有明確的指令 資訊量 多 少 資訊精準度 低 高 資訊呈現 結構式 非結構式 介面呈現 以圖像為主 以文字為主 介面操作 以點選為主 以文字或語音輸入為主 學習與引導 使用者需了解、學習、適應不同的介面操 作 如同使用即時通訊軟體,使用者無需另行 學習,只需遵循引導 服務入口 需另行下載App或進入特定網站 與即時通訊軟體整合,可適時在對話裡提 供服務 服務內容 需有明確架構供檢索 可接受龐雜、彈性的內容 人性化程度 低,如同操作機器 高,如同與人對話 9
  10. 10. Two Branches of Bots Task-Oriented  Personal assistant, helps users achieve a certain task  Combination of rules and statistical components  POMDP for spoken dialog systems (Williams and Young, 2007)  End-to-end trainable task-oriented dialogue system (Wen et al., 2016)  End-to-end reinforcement learning dialogue system (Li et al., 2017; Zhao and Eskenazi, 2016) Chit-Chat  No specific goal, focus on natural responses  Using variants of generation model  A neural conversation model (Vinyals and Le, 2015)  Reinforcement learning for dialogue generation (Li et al., 2016)  Conversational contextual cues for response ranking (AI-Rfou et al., 2016) 10
  11. 11. Dialogue System Framework MODULAR SYSTEM 11
  12. 12. System Framework 12 Speech Recognition Language Understanding (LU) • Domain Identification • User Intent Detection • Slot Filling Dialogue Management (DM) • Dialogue State Tracking • Policy Optimization Generation Hypothesis are there any action movies to see this weekend Semantic Frame request_movie genre=action, date=this weekend System Action/Policy request_location Text response Where are you located? Screen Display location? Text Input Are there any action movies to see this weekend? Speech Signal current bottleneck  error propagation Backend Database
  13. 13. 1 2 3 4
  14. 14. Interaction Example Intelligent Agent Q: How does a dialogue system process this request? Good Taiwanese eating places include Din Tai Fung, etc. What do you want to choose? 好的台式餐廳包含鼎泰豐…,你想要選哪一間? find a good eating place for taiwanese food 我想找好的台式食物 14
  15. 15. Language Understanding
  16. 16. System Framework 16 Speech Recognition Language Understanding (LU) • Domain Identification • User Intent Detection • Slot Filling Dialogue Management (DM) • Dialogue State Tracking • Policy Optimization Generation Hypothesis are there any action movies to see this weekend Semantic Frame request_movie genre=action, date=this weekend System Action/Policy request_location Text response Where are you located? Screen Display location? Text Input Are there any action movies to see this weekend? Speech Signal Backend Database
  17. 17. 1. Domain Identification Requires Predefined Domain Ontology 17 Organized Domain Knowledge (Database) Restaurant DB Taxi DB Movie DB Classification! find a good eating place for taiwanese food 我想找好的台式食物
  18. 18. find a good eating place for taiwanese food 我想找好的台式食物 2. Intent Detection Requires Predefined Schema 18 Restaurant DB FIND_RESTAURANT FIND_PRICE FIND_TYPE : Classification!
  19. 19. find a good eating place for taiwanese food 我想找好的台式食物 3. Slot Filling Requires Predefined Schema 19 Restaurant DB Restaurant Rating Type Rest 1 good Taiwanese Rest 2 bad Thai : : : FIND_RESTAURANT rating=“good” type=“taiwanese” SELECT restaurant { rest.rating=“good” rest.type=“taiwanese” }Semantic Frame Sequence Labeling O O B-rating O O O B-type O
  20. 20. Dialogue Management State Tracking Dialogue Management Policy Optimization
  21. 21. System Framework 21 Speech Recognition Language Understanding (LU) • Domain Identification • User Intent Detection • Slot Filling Dialogue Management (DM) • Dialogue State Tracking • Policy Optimization Generation Hypothesis are there any action movies to see this weekend Semantic Frame request_movie genre=action, date=this weekend System Action/Policy request_location Text response Where are you located? Screen Display location? Text Input Are there any action movies to see this weekend? Speech Signal Backend Database
  22. 22. Dialogue State Tracking Requires Hand-Crafted States 22 location rating type loc, rating rating, type loc, type all i want it near to my office 我希望可以離我辦公室近一點 NULL find a good eating place for taiwanese food 我想找好的台式食物 i want it near to my office 我希望可以離我辦公室近一點 find a good eating place for taiwanese food 我想找好的台式食物
  23. 23. Dialogue State Tracking Requires Hand-Crafted States 23 location rating type loc, rating rating, type loc, type all NULL i want it near to my office 我希望可以離我辦公室近一點 find a good eating place for taiwanese food 我想找好的台式食物
  24. 24. Dialogue State Tracking Handling Errors and Confidence 24 FIND_RESTAURANT rating=“good” type=“taiwanese” FIND_RESTAURANT rating=“good” type=“thai” FIND_RESTAURANT rating=“good” location rating type loc, rating rating, type loc, type all NULL ? ? find a good eating place for taiwanese food 我想找好的台式食物 rating=“good”, type=“thai” rating=“good”, type=“taiwanese” ? ?
  25. 25. Dialogue Policy for Agent Action  Inform(location=“Taipei 101”)  “The nearest one is at Taipei 101”  Request(location)  “Where is your home?”  Confirm(type=“taiwanese”)  “Did you want Taiwanese food?”  Task Completion / Information Display  ticket booked, weather information 25
  26. 26. Generation
  27. 27. System Framework 27 Speech Recognition Language Understanding (LU) • Domain Identification • User Intent Detection • Slot Filling Dialogue Management (DM) • Dialogue State Tracking • Policy Optimization Generation Hypothesis are there any action movies to see this weekend Semantic Frame request_movie genre=action, date=this weekend System Action/Policy request_location Text response Where are you located? Screen Display location? Text Input Are there any action movies to see this weekend? Speech Signal Backend Database
  28. 28. Output / Natural Language Generation  Goal: generate natural language or GUI given the selected dialogue action for interactions  Inform(location=“Taipei 101”)  “The nearest one is at Taipei 101” v.s.  Request(location)  “Where is your home?” v.s.  Confirm(type=“taiwanese”)  “Did you want Taiwanese food?” v.s. 28
  29. 29. Challenges and Recent Trends DEEP LEARNING / MULTIMODALITY/ DIALOGUE COVERAGE / DIALOGUE COMPLEXITY 29
  30. 30. Challenges  Predefined semantic schema Chen et al., “Matrix Factorization with Knowledge Graph Propagation for Unsupervised Spoken Language Understanding,” in ACL-IJCNLP, 2015.  Data without annotations Chen et al., “Zero-Shot Learning of Intent Embeddings for Expansion by Convolutional Deep Structured Semantic Models,” in ICASSP, 2016.  Semantic concept interpretation Chen et al., “Deriving Local Relational Surface Forms from Dependency-Based Entity Embeddings for Unsupervised Spoken Language Understanding,” in SLT, 2014.  Predefined dialogue states Chen, et al., “End-to-End Memory Networks with Knowledge Carryover for Multi-Turn Spoken Language Understanding,” in Interspeech, 2016.  Error propagation Hakkani-Tur et al., “Multi-Domain Joint Semantic Frame Parsing using Bi-directional RNN-LSTM,” in Interspeech, 2016.  Cross-domain intention/bot hierarchy Sun et al., “An Intelligent Assistant for High-Level Task Understanding,” in IUI, 2016. Sun et al., “AppDialogue: Multi-App Dialogues for Intelligent Assistants,” in LREC, 2016. Chen et al., “Leveraging Behavioral Patterns of Mobile Applications for Personalized Spoken Language Understanding,” in ICMI, 2016.  Cross-domain information transferring Kim et al., “New Transfer Learning Techniques For Disparate Label Sets,” in ACL-IJCNLP, 2015. 30 FIND_RESTAURANT rating=“good” rating=5? 4? HotelRest Flight Travel Trip Planning
  31. 31. Deep Learning for LU  IOB Sequence Labeling for Slot Filling  Intent Classification 31 𝑤0 𝑤1 𝑤2 𝑤 𝑛 ℎ0 𝑓 ℎ1 𝑓 ℎ2 𝑓 ℎ 𝑛 𝑓 ℎ0 𝑏 ℎ1 𝑏 ℎ2 𝑏 ℎ 𝑛 𝑏 𝑦0 𝑦1 𝑦2 𝑦 𝑛 (a) LSTM (b) LSTM-LA (c) bLSTM-LA (d) Intent LSTM intent 𝑤0 𝑤1 𝑤2 𝑤 𝑛 ℎ0 ℎ1 ℎ2 ℎ 𝑛 𝑦0 𝑦1 𝑦2 𝑦 𝑛 𝑤0 𝑤1 𝑤2 𝑤 𝑛 ℎ0 ℎ1 ℎ2 ℎ 𝑛 𝑦0 𝑦1 𝑦2 𝑦 𝑛 𝑤0 𝑤1 𝑤2 𝑤 𝑛 ℎ0 ℎ1 ℎ2 ℎ 𝑛
  32. 32. Deep Learning for LU  Joint multi-domain intent prediction and slot filling  Information can mutually enhanced 32 Hakkani-Tur, et al., “Multi-Domain Joint Semantic Frame Parsing using Bi-directional RNN-LSTM,” in Interspeech, 2016. ht-1 ht+1ht taiwanese B-type food please O O hT+1 EOS FIND_REST Slot Filling Intent Prediction
  33. 33. Contextual LU (Chen et al., 2016) 33 just sent email to bob about fishing this weekend O O O O B-contact_name O B-subject I-subject I-subject U S I send_emailD communication  send_email(contact_name=“bob”, subject=“fishing this weekend”) are we going to fish this weekend U1 S2  send_email(message=“are we going to fish this weekend”) send email to bob U2  send_email(contact_name=“bob”) B-message I-message I-message I-message I-message I-message I-message B-contact_nameS1 Domain Identification  Intent Prediction  Slot Filling Chen et al., "End-to-End Memory Networks with Knowledge Carryover for Multi-Turn Spoken Language Understanding,” in Interspeech, 2016.
  34. 34. Supervised v.s. Reinforcement  Supervised  Reinforcement Hello  Agent …… Agent ……. ……. …… Bad “Hello” Say “Hi” “Bye bye” Say “Good bye” Learning from teacher Learning from critics 34
  35. 35. Scenario of Reinforcement Learning Agent learns to take actions to maximize expected reward. 35 Environment Observation ot Action at Reward rt If win, reward = 1 If loss, reward = -1 Otherwise, reward = 0 Next Move
  36. 36. Dialogue Policy Optimization  Dialogue management in a RL framework 36 U s e r Reward ObservationAction Environment Agent Natural Language Generation Language Understanding Dialogue Manager Slides credited by Pei-Hao Su Optimized dialogue policy selects the best action that can maximize the future reward. Correct rewards are a crucial factor in dialogue policy training
  37. 37. Reinforcement Learning Based Bot User goal: Two tickets for “the witch” tomorrow 9:30 PM at regal meridian 16, Seattle. 37 RULE BASED SYSTEM User: Can I get 2 tickets for the witch? Agent: Which theater do you want? User: Regal meridian 16 Agent: What time would you like to see it? User: 9:30 pm Agent: What date would you like to see it? User: Tomorrow! Agent: How many people are you? User: Just two adults. Agent: What is your location? User: Seattle! Agent: What kind of video format do you like? User: Oh, any kind is okay. Agent: Do you have any price constraint? User: No, I don’t care. Agent: Great - I was able to purchase 2 tickets for you to see the witch tomorrow at regal meridian 16 theater in Seattle at 9:30 pm. User: Thank you. Agent: Thank you! REINFORCEMENT LEARNING SYSTEM User: Can I get 2 tickets for the witch? Agent: What date would you like to see it? User: Tomorrow Agent: Which theater do you prefer? User: Regal meridian 16 Agent: What is your location? User: Seattle Agent: What time would you like to see it? User: 9:30 pm. Agent: Great - I was able to purchase 2 tickets for you to see the witch tomorrow at regal meridian 16 theater in Seattle at 9:30 pm. User: Thanks. Agent: Thanks! The system can learn how to efficiently interact with users for task completion Li et al., "End-to-End Task-Completion Neural Dialogue Systems,” in arXiv, 2017.
  38. 38. Brain Signal for Understanding  Misunderstanding detection by brain signal  Green: listen to the correct answer  Red: listen to the wrong answer 38 Detecting misunderstanding via brain signal in order to correct the understanding results Sridharan et al., “NeuroDialog: An EEG-Enabled Spoken Dialog Interface,” in ICMI, 2012.
  39. 39. Video for Intent Understanding 39 Proactive (from camera) I want to see a movie on TV! Intent: turn_on_tv Sir, may I turn on the TV for you? Proactively understanding user intent to initiate the dialogues.
  40. 40. App Behavior for Understanding  Task: user intent prediction  Challenge: language ambiguity  User preference  Some people prefer “Message” to “Email”  Some people prefer “Outlook” to “Gmail”  App-level contexts  “Message” is more likely to follow “Camera”  “Email” is more likely to follow “Excel” 40 send to vivian v.s. Email? Message? Communication Considering behavioral patterns in history to model understanding for intent prediction. Chen et al., “Leveraging Behavior Patterns of Mobile Applications for Personalized Spoken Language Understanding,” in ICMI, 2015.
  41. 41. Evolution Roadmap 41 Single domain systems Extended systems Multi- domain systems Open domain systems Dialogue breadth (coverage) Dialoguedepth(complexity) What is influenza? I’ve got a cold what do I do? Tell me a joke. I feel sad…
  42. 42. Intent Expansion  Transfer dialogue acts across domains  Dialogue acts are similar for multiple domains  Learning new intents by information from other domains CDSSM New Intent Intent Representation 1 2 K : Embedding Generation K+1 K+2<change_calender> Training Data <change_note> “adjust my note” : <change_setting> “volume turn down” The dialogue act representations can be automatically learned for other domains postpone my meeting to five pm Chen et al., “Zero-Shot Learning of Intent Embeddings for Expansion by Convolutional Deep Structured Semantic Models,” in ICASSP 2016. 42
  43. 43. Policy for Domain Adaptation  Bayesian committee machine (BCM) enables estimated Q- function to share knowledge across domains QRDR QHDH QL DL Committee Model The policy from a new domain can be boosted by the committee policy #dialogues reward Gašić, et al., “Policy Committee for Adaptation in Multi-Domain Spoken Dialogue Systems,” in ASRU, 2015. 43
  44. 44. Evolution Roadmap 44 Knowledge based system Common sense system Empathetic systems Dialogue breadth (coverage) Dialoguedepth(complexity) What is influenza? I’ve got a cold what do I do? Tell me a joke. I feel sad…
  45. 45. High-Level Intention for Dialogue Planning  High-level intention may span several domains Schedule a lunch with Vivian. find restaurant check location contact play music What kind of restaurants do you prefer? The distance is … Should I send the restaurant information to Vivian? Users can interact via high-level descriptions and the system learns how to plan the dialogues Sun, et al., “An Intelligent Assistant for High-Level Task Understanding,” in IUI, 2016. 45
  46. 46. Empathy in Dialogue System  Embed an empathy module  Recognize emotion using multimodality  Generate emotion-aware responses 46 Emotion Recognizer vision speech text Fung et al., “Towards Empathetic Human-Robot Interactions,” in arXiv, 2016.
  47. 47. Conclusion 47
  48. 48. Conclusion  The conversational systems can manage information access via spoken interactions  A domain is usually constrained by the backend database  Semantic schema should be predefined  Cross-domain knowledge and intention is difficult to handled  Human-like dialogue system  Robustness  error handling via interactions / reasoning  Dialogue coverage  domain switching / open-domain system  Dialogue complexity  common sense and empathy 48
  49. 49. Q & A THANKS FOR YOUR ATTENTION! 49

×