O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time

284 visualizações

Publicada em

Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time.
Ting-Hao K. Huang, Joseph Chee Chang, Jeffrey P. Bigham.
In Proceedings of Conference on Human Factors in Computing Systems 2018 (CHI 2018), 2018, Montréal, Canada.

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time

  1. 1. 1 Live Note/QA: http://tinyurl.com/Evorus Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time Ting-Hao (Kenneth) Huang* Joseph Chee Chang Jeffrey P. Bigham [ Question / Feedback: http://tinyurl.com/Evorus ] *Kenneth will join in Fall 2018
  2. 2. 2 Live Note/QA: http://tinyurl.com/Evorus
  3. 3. 3 Live Note/QA: http://tinyurl.com/Evorus
  4. 4. 4 Live Note/QA: http://tinyurl.com/Evorus
  5. 5. 5 Live Note/QA: http://tinyurl.com/Evorus
  6. 6. 6 Live Note/QA: http://tinyurl.com/Evorus
  7. 7. 7 Live Note/QA: http://tinyurl.com/Evorus What just happened? • Open Conversation • Multi-turn interaction • Multiple domains • Personalized • Coherent dialog • Mix of task-oriented and social conversation
  8. 8. 8 Live Note/QA: http://tinyurl.com/Evorus Today’s Conversational Assistants… “What’s new with Alexa?”“Talking to Siri”
  9. 9. 9 Live Note/QA: http://tinyurl.com/Evorus Existing Approaches to Open Conversation • Combining multiple task-oriented dialog systems • DialPort (Zhao, et al., 2016) • End-to-end framework for dialogue systems • Serban, et al. 2016; Li, et al. 2017 • Adapting a model to many other domains • Walker, et al., 2007; Sun, et al., 2016 • Chit-chat systems (social bot) • Hold social conversations (Banchs, et al., 2012) • Still a very hard problem…
  10. 10. 10 Live Note/QA: http://tinyurl.com/Evorus Existing Approaches to Open Conversation • Combining multiple task-oriented dialog systems • DialPort (Zhao, et al., 2016) • End-to-end framework for dialogue systems • Serban, et al. 2016; Li, et al. 2017 • Adapting a model to many other domains • Walker, et al., 2007; Sun, et al., 2016 • Chit-chat systems (social bot) • Hold social conversations (Banchs, et al., 2012) • Still a very hard problem… MIT Technology Review Feb 27, 2018
  11. 11. 11 Live Note/QA: http://tinyurl.com/Evorus A Top-Down Approach Fully-Automated System Human- Powered System
  12. 12. 12 Live Note/QA: http://tinyurl.com/Evorus A Top-Down Approach Fully-Automated System Crowd-AI System Human- Powered System
  13. 13. 13 Live Note/QA: http://tinyurl.com/Evorus A Top-Down Approach Fully-Automated System Crowd-AI System Human- Powered System
  14. 14. 14 Live Note/QA: http://tinyurl.com/Evorus In Today’s Talk… Fully-Automated System Evorus Chorus
  15. 15. 15 Live Note/QA: http://tinyurl.com/Evorus Chorus: A Crowd-Powered Conversation Assistant Evorus
  16. 16. 16 Live Note/QA: http://tinyurl.com/Evorus •Crowd workers collectively hold a conversation by: 1. Propose Responses 2. Vote Responses 3. Take Notes Reward points for each action "Is there anything else I can help you with?": Challenges in Deploying an On-Demand Crowd-Powered Conversational Agent. Huang, et al. HCOMP 2016. Chorus: A Crowd-Powered Conversation Assistant
  17. 17. 17 Live Note/QA: http://tinyurl.com/Evorus User Interface 17 / 31
  18. 18. 18 Live Note/QA: http://tinyurl.com/Evorus User & Worker Interface 18 / 31
  19. 19. 19 Live Note/QA: http://tinyurl.com/Evorus
  20. 20. 20 Live Note/QA: http://tinyurl.com/Evorus We Deployed Chorus • Launched on May 20th, 2016 • On Google Hangouts • 2200+ conversations, 420+ users • TalkingToTheCrowd.org
  21. 21. 21 Live Note/QA: http://tinyurl.com/Evorus Fully-Automated System Evorus Chorus Evorus: A Crowd-Powered Conversational Assistant Built to Automate Itself Over Time
  22. 22. 22 Live Note/QA: http://tinyurl.com/Evorus Automating Chorus
  23. 23. 23 Live Note/QA: http://tinyurl.com/Evorus Automating Chorus
  24. 24. 24 Live Note/QA: http://tinyurl.com/Evorus Automating Chorus
  25. 25. 25 Live Note/QA: http://tinyurl.com/Evorus Automating Chorus
  26. 26. 26 Live Note/QA: http://tinyurl.com/Evorus Chatbots How to select chatbots automatically?
  27. 27. 27 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Chatbot’s Performance Topic Similarity Likelihood of a Chatbot
  28. 28. 28 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Chatbot’s Performance Topic Similarity Likelihood of a Chatbot ~= Overall Message Acceptance Rate
  29. 29. 29 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Topic Similarity User Message Domain of the Chatbot Hey what should I eat in Montreal?
  30. 30. 30 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Topic Similarity User Message Domain of the Chatbot Hey what should I eat in Montreal? Find me some good restaurants ! Where can I get Chinese food? Example Triggering Message
  31. 31. 31 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Topic Similarity User Message Domain of the Chatbot Hey what should I eat in Montreal? Example Triggering Message Find me some good restaurants ! Where can I get Chinese food? Topic Similarity
  32. 32. 32 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Chatbot’s Performance Topic Similarity Likelihood of a Chatbot
  33. 33. 33 Live Note/QA: http://tinyurl.com/Evorus Ranking Chatbots: Performance & Topic Chatbot’s Performance Topic Similarity Likelihood of a Chatbot  Add more chatbots over time !
  34. 34. 34 Live Note/QA: http://tinyurl.com/Evorus
  35. 35. 35 Live Note/QA: http://tinyurl.com/Evorus Automatic Upvote How to estimate the impact of an automation?
  36. 36. 36 Live Note/QA: http://tinyurl.com/Evorus Find the Best Confidence Threshold • High Threshold • Only vote when pretty sure • High precision, but little benefit • Low Threshold • Nearly always vote • Grant agreement bonus by mistake • Damage conversation quality
  37. 37. 37 Live Note/QA: http://tinyurl.com/Evorus Find the Best Confidence Threshold Expected Reward Points Saved
  38. 38. 38 Live Note/QA: http://tinyurl.com/Evorus Automating Chorus Over Time
  39. 39. 39 Live Note/QA: http://tinyurl.com/Evorus Automating Open Conversations • Setup • A 5-month-long deployment, 80 Users • 4 chatbots + 1 voting bot • Result • Automated responses were chosen 12.44% of the time. • Human upvotes were reduced by 13.81%. • The cost of each message is reduced by 32.76%. • Conversation quality and user satisfaction level remains. • Conversation Quality: Satisfaction, Clarity, Responsiveness, Comfort (Liu, et al., 2010)
  40. 40. 40 Live Note/QA: http://tinyurl.com/Evorus Conclusion & Future Work • Automating some part of an open conversation is possible. • Future Work • 1000+ Chatbots! • Deploy Chorus as an Open Research Platform • Chorus on Smart Devices
  41. 41. 41 Live Note/QA: http://tinyurl.com/Evorus Conclusion & Future Work • Automating some part of an open conversation is possible. • Future Work • 1000+ Chatbots! • Deploy Chorus as an Open Research Platform • Chorus on Smart Devices @windx0303 KennethHuang.cc Ting-Hao (Kenneth) Huang QA: http://tinyurl.com/Evorus
  42. 42. 42 Live Note/QA: http://tinyurl.com/Evorus Backup Slides
  43. 43. 43 Live Note/QA: http://tinyurl.com/Evorus
  44. 44. 44 Live Note/QA: http://tinyurl.com/Evorus Features of Automatic Voting

×