UXPA2019 Not Your Average Chatbot: Using Cognitive Intercept to Improve Information Discovery
•Transferir como PPTX, PDF•
2 gostaram•330 visualizações
This presentation from UXPA 2019 will review cognitive intercept as pertains to search, and how it extends to an additional domain (live agent chat). Evidence that it helps users and lowers help desk volumes will be discussed.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a UXPA2019 Not Your Average Chatbot: Using Cognitive Intercept to Improve Information Discovery
Semelhante a UXPA2019 Not Your Average Chatbot: Using Cognitive Intercept to Improve Information Discovery (20)
Mais de UXPA International
Mais de UXPA International (20)
Último
Último (18)
UXPA2019 Not Your Average Chatbot: Using Cognitive Intercept to Improve Information Discovery
- 1. Not Your Average Chatbot: Using Cognitive Intercept to Improve Information Discovery Jon G. Temple, Ph.D. Claude Elie Lisa Schenkewitz Paula Nezbedova Dabby Phipps Kenn Roberson Josef Scherpa Mark Malatesta
- 2. We built this Jon Temple, CIO Design Claude Elie, CIO Design Lisa Schenkewitz, CIO Systems & Tools Dabby Phipps, CIO Design Josef Scherpa, formerly IBM CIO Design Paula Nezbedova, Digital Workplace Engineering Kenn Roberson, CIO Network Engineering
- 3. Background Our project: a self-service IT support portal Common user complaint: “I cannot find a solution to my problem” or more simply, “search sucks!” Search engine was ineffective 1 Users didn’t notice existing/findable content 3 Missing content 2 Did not confirm Problem confirmed but already solutioned Suggested we consider a Watson chatbot…
- 4. Chatbots & advantages • Natural language understanding • Limited ability to use context • Disambiguation • Complex information flows possible (e.g., self-service functionality) • Feels personal
- 5. Meeting high expectations • Expectations — at least initially — are high! • Siri, Alexa, Cortana, and Watson – Oh My! • High cost of not meeting expectations (abandonment) • Potential to both frustrate and delight • Could a chatbot help us improve information discovery? 5 This is what users envision and expect when you say you are adding Watson to an offering.
- 6. Enterprise chatbots Enterprise chatbots / virtual assistants • Commonplace • Easy to build • Dedicated UI • Text-based natural language • Targeted to a single knowledge domain • Takes time to train • Limited training 6 Type message… Anna Kirkwood I would like to reset a password Virtual Agent Sure, I can help you with that. Which password do you want to reset? Virtual Agent Mac Voicemail Intranet Traditional chatbot History Full NL conversation required Dedicated entry field Feedback Disambiguation and buttons Dedicated interface Displayed even if not trained
- 7. Chatbot challenges Chatbots need to be trained to the right level of specificity • Providing specific answers to specific answers to specific questions is challenging • May be difference between delight and disappointment • Danger of feeling like an interactive FAQ Show me jobs available in Arizona? I can help with that! Here is a list of available jobs
- 8. Chatbot challenges Chatbots need to provide direct answers, not just search results • Chatbots degrade to search • Search is limited; lacks basic search functions such as filters or pagination How do I change the boot order of my servers on Linux? Let me search the web for you. Check this out.
- 9. Chatbot challenges Chatbots must communicate their domains • Learn the rules: What works, what doesn’t • How do you correct mistakes? Does it use context? What’s the magic word? • Challenging with a large domain Can I skip a payment next month without any penalty or interest? Here are some things you can ask me: “What is my credit card balance?” “When is my credit card due?” “What is my APR?” “What is the difference between paying off credit card debt and indentured servitude?”
- 10. Chatbot challenges Users must learn a new interaction vehicle • Cognitive overhead for chatbot interface itself; each chatbot is a little different • Adequate lexicon • Natural language misunderstandings What year wasWheelerVs. California? Wheeler is a city inTexas. Map.
- 12. Our solution: cognitive intercept User initiates action (search, live chat, etc) Cognitive layer screens input Requested service retrieves result Display cognitive match Display requested result (search, live chat, etc.) Is there a high- confidence cognitive match? Yes No
- 13. How does it work? (1/3) 1 Search forms the base (includes pagination, filters, keywords) 2 3 Starts with an optional natural language query in the search field Search results (if any) are returned 7
- 14. How does it work? (2/3) 4 If AI is trained on a particular issue, display above any search results in a distinct style. 5 If AI requires a response from user, handle through a series of close-ended prompts (buttons) — not natural language 6 Conversations take place only in the area occupied by the AI
- 15. How does it work? (3/3) 7 Cognitive intercept may also leverages self-service widgets, such as this reset password function
- 16. Cognitive intercept applied to other contexts 1 User initiates the live chat to speak to a human agent Live Chat (asking to speak with a human being) Chat Monday - Thursday 9:00 AM - 5:00 PM (EST) Community All Communities (3) Support from other IBMers. Call All numbers (3) 1-800-IBM-Help Monday - Thursday 9:00 AM - 5:00 PM (EST)
- 17. Cognitive intercept applied to other contexts 2 User enters a description of their problem for the human agent 3 This description is screened by the cognitive layer Before you start your chat, I think this might help: Cancel my chat (issue resolved)Start chat now Watson took a look at your issue, and found that people with an issue similar to yours found this to be the solution…. Reset my Notes Password 4 If AI is trained on a particular issue, display the intent in the same window Otherwise, directly initiate chat 5 User may continue to live chat if they choose to or if AI failed to help
- 18. Comparison of core elements 18 Type message… Anna Kirkwood I would like to reset a password Virtual Agent Sure, I can help you with that. Which password do you want to reset? Virtual Agent Mac Voicemail Intranet Traditional chatbot Cognitive intercept (responsive) History Full NL conversation required Embedded in another interface Dedicated entry field Use entry field of existing interface NL is optional for entry Feedback Disambiguation and buttons Dedicated interface Displayed even if not trained Hidden except when high-confidence match
- 19. Advantages of cognitive intercept Single point of entry 1 Reduced learning curve 2 Natural language input is optional 3 Correcting mistakes 4 Follow up refinements 5 Intent-triggered advice 6 Graceful ramp up 7
- 20. Disadvantages of cognitive intercept Not able to reuse existing chatbot UIs 1 Not personal 2 Must integrate with a base interface 3
- 21. 4 14 27 26 0 5 10 15 20 25 30 Q1 2018 Q2 2018 Q3 2018 4Q2018 1Q2019 Cognitive intercept: Search introduced Cognitive intercept: Chat introduced Net Promoter Score (NPS) improvementNP S
- 22. User Feedback • Asked if users were satisfied (n=223 for search, n=452 for chat) • Satisfaction was highest when expectations were met % Satisfied with intercept functionality when… Search Chat The intercept solved their problem 100% 92% The intercept partially solved their problem 82% 70% The intercept did not solve their problem 12% 14% Not unique to cognitive intercept – a challenge for any chatbot implementation
- 23. Reasons intercept may not have solved the user’s issue… ChatSearch
- 24. How did we build it?
- 25. Anatomy of a Watson Intent • The user’s intention … their purpose or goal, stated as a verb. • Example: -The user wants to reset a password. Intents Entities • The noun(s) that the user wants to take action on. • Example: - Notes, w3id, macOS, Windows, Linux,GSA, Symantec PGP,VM, AT&T Net Client, IBM Mobility Client,VM @generic_application:No tes (Lotus Notes, IBM Notes, SmartCloud Notes, Smart Cloud Notes) Utterances • Example phrases of what the user might want to do • Example: - Reset my Notes password Dialog • Matches the user’s utterance (what user’s say) to a response (what Watson says back) • Example: o [user] Reset my Notes password o [Watson] If you know your current IBM Notes password AND are confident it hasn't been expired for 90 days or more, select Change. Otherwise, select Reset.
- 26. Intent creation process Intent creation reqires understanding the issue and the solution. Help@IBM content is personalized based on platform and location. • TopCall Drivers Based on HistoricalTicket Data • Interviews with Agents • Talk to ProductArea SMEs • Help@IBM Search Data • User Feedback • Data ScienceAnalysis • New Business (new) • Help@IBM Content Updates (new) Identify Low Complexity / High Volume Issues Manual and Data ScienceAnalysis Identify Issue Research Solution Document Solution Add to Corpus Test Prioritize Backlog Reach out to SMEs and review available support materials Create dialogs Create flow charts Create training phrases based on user examples for asking about this issue Work with developers to map out API integration Intents, entities, and dialog nodes are added to Watson and answer store Manually test intents in Dev; Automated testing in Test Identify phrases used by end users to inquire about the issue (utterances) Identify entities
- 27. Intent types Disambiguation node with branches, and optional self-service functionality Single dialog node Recommended by Watson - looks like a search result and links directly to a support article
- 28. • Watson Assistant has limited programmatic capabilities • Extend existing Help@IBM application to: • Integrate results from Watson Assistant into key flows in the applications • Create mechanism to bridge communications between components • Components and data formats had to be extensible so it could be used by other channels such as an HTML based chatbot or a telephony interface (Voice Response Unit or Interactive Voice Response) • Components had to be scalable for performance and reliability Architectural challenges
- 29. • Created an single entry point into Help@IBM search services • Combines results from multiple search locations (Help@IBM content, Watson Orchestration Layer, Support Options) • Single entry point into key services from Help@IBM to Watson Orchestrator • Designed presentation layer agnostic interface that allows for reuse of Watson Application Overview of frontend architecture Help@IBM Aggregation Content Help@IBM web site Help@IBM services Help@IBM
- 30. Service Orchestration Engine to extend Watson’s capabilities • Orchestrates messages going back and forth between the user channel and Watson • Converts incoming UI channel messages toWatson format • ConvertsWatson responses to UI channel format • Uses Watson responses and makes decisions on further actions to take, such as calling APIs, before returning response back to the UI channel Solution – Watson application side Help@IBM Aggregation Content Pluggable GUI Phone channel (Voice Gateway) Slack Bot Standalone Chatbot Help@IBM web site Service Orchestration Layer Help@IBM services 3rd Party MicroService Watson Conversations Watson Speech to Text , Text to Speech Transcripts And Answer Store Service endpoint & dispatcher Cloudant MicroService Help@IBM Component Watson MicroService Help@IBM
- 31. Application with Watson Architecture Summary Public IBM Cloud Services Application Cloudant Database Watson Microservice Application Watson Assistant (Conversation) Cloudant Microservice Availability Monitoring DevOps Tool Chaining Secure Gateway Help@IBM Services Application API Gateway Service Coordinates services between Help@IBM andWatson conversations ProvidesWatson and DevOps Services API gateway passes information to and from Help@IBM application services hosted in a different cloud environment to theWatson application cloud component Private Cloud Hosting Environment Auto Scaling Cloud Object Store (Backups) Born on IBM Cloud IBM Help application suite And website
- 32. Closing comments • Alternative to the traditional chatbot • Manages user expectations / less cognitive overhead • Effective at boosting information discovery • Success or failure dependent on the intents / match to the questions Parting advice • Coordinate cognitive intents • Say when to stop searching • Continue to research user needs / build out intents to better anticipate them • More applications for cognitive intercept
- 33. Thanks to the many people at IBM who have helped make this a reality: Francesco PaoloVitullo Vladimir Tkac Roman Kitko Sean Laval Sahana Ramaswamy Frantisek Laszlo Vladimir Lalik Jayprakash Patel Gerry Przybyszewski Marian Koprivnansky Roberto Navarrete Marek Menhert Sherry Marshall Daniel Boyle Kelsey Duke Rick Dorgan Special thanks to Mark Malatesta for his help with data analyses Jean Sander Joseph Riina Umesh Manam Steve Leonard Robert Irzyk Janet Parsons Kit McCaffrey
- 34. Backup slides
- 35. References 1. 7 Questions for a Starter ChatbotTemplate: A SimpleCase Study, https://chatbotslife.com/questions-for-a-starter- chat-bot-template-a-simple-case-study- fbb0a16ef84b 2. CliveThompson onWhy Kids Can’t Search, https://www.wired.com/2011/11/st_thompson_searchresults/ 3. Seven Web Search HabitsYou Should Know, https://www.lifewire.com/highly-effective-web-searchers-3482841 4. Google's vision for AI is the right one, https://mashable.com/2017/05/17/google-io-artificial-intelligence- plan/#lprL27eooqqx 5. Ignore the hype: Here’s the right strategy for your chatbot, https://chatbotslife.com/ignore-the-hype-heres-the- right-strategy-for-your-chatbot-43c9515f18f2 6. Chatbots were the next big thing: what happened?, https://blog.growthbot.org/chatbots-were-the-next-big-thing- what-happened 7. Tommy Hilfiger Bot: Artificial IntelligenceGone Wrong, https://chatbotslife.com/tommy-hilfiger-bot-artificial- intelligence-gone-wrong-90a545ac9a20 8. A hybrid approach to integrating Watson Assistant into an existing site, https://developer.ibm.com/articles/cc-hybrid-integrating-watson-assistant/
- 36. • Finally, explore additional vehicles for cognitive intercept • Next up:Voice interfaces (IVR/VRU), where it can silently screen questions before offering a help desk agent Example of cognitive intercept applied to a VRU/IVR
Notas do Editor
- We support IBM and the internal tools needed to keep IBMers productive Multidisciplinary/multinational team of UX designers, product owners, architects and developers Our designers are often focused on identifying pain points and designing solutions to remedy them
- There can be several reasons a user does not find the help they need even when an appropriate solution exists: Difficulty identifying a correct solution from a search result list Result is not salient enough visually Article titles may not convey enough information Too many similar results Result requires too much effort to extract benefit Difficulty constructing a good search query (an acquired skill) Failure to iterate and refine search
- Here we use the term chatbot in a generic sense that also includes virtual assistants and oersonal assistants Chatbots convey some unique advantages over standard search (although not necessarily capable of replacing search)
- Siri, Alexa, Cortana - Dedicated AI, voice recognition, heavily trained, wide domain knowledge
- Commonplace on support sites Easy to build Dedicated UI – that a user must learn Text-based natural language Targeted to a single knowledge domain Takes time to train Often limited training Our hesitancy in embracing a traditional chatbot approach stemmed from a number of concerns, studying them, using them, designing solutions with them, which I will now review a few of these
- Ultimately has to do with the depth of your knowledge base Corpus: Too shallow (Interactive FAQ) v. Deep (too long to market) Goldilocks Danger: Interactive FAQ – if that is the best you can do, stick with a FAQ! It is easier to use. Example: Chatbot responded with a list of jobs on in North America; user may be unaware it is not scoped correctly
- Better than nothing, but it is an admission the chatbot knows nothing about your question If you need to search, a chatbot is a poor substitute for a proper search interface Disappointing Example: “Let me search the Web for you” is a poor search experience, or a link to a video/document
- Recognition versus recall – chatbots force this in reverse by making you memorize; if you use a particular one, it may be worth the investment; but if it is a casual one-time deal, it can be another point of frustration Using a chatbot can be like getting a magic spell right. Recently reread Harry Potter with my daughter and kept thinking Hermione’s insistence on the exact words / cadence was very chatbot like in its inflexibility Example: A chatbot for a credit card company may have a limited domain but users still have to learn it
- In the enterprise world,. Chatbots pop up like mushrooms and each is a bit different than the other: Different words it understands Feedback is asked differently It knows different things You are expecting the user to learn your chatbot interface, even if they will use it just once or twice, or once or twice a year even; and the next chatbot you encounter will have it own user interface model and you have to learn how to use it too which will differ in many ways that won’t be immediately obvious until you have spent time with it Users don’t want to spend time hanging out with Clippy the Chatbot; they want in an out – I want an answer! That’s it. Example: A question about a court case confused with a geography question. Users need to learn what each chatbot is good for and the best way to get those answers.
- On support portals, search and chatbots in direct competition Search extremely popular Search can be very powerful (filters, keywords, Booleans, etc.) A chatbot degrades to search when it has no answers… Separate interfaces – not integrated – a UX problem! Confusion: How does a user know if their answer will be found better via search or chatbot? Classic task analysis would reveal the need to jump between solutions as a key problem you’d need to address Context shifts and uncertainty Designer often fix these issues through integration – best of both worlds… so… We’ve also discussed High expectations already too – so keep that in mind as we review our solution
- Combine the advantages of each into one Most of our users start with search Choose search as the foundation Sets the expectation – only expecting search results Chatbot “speaks up” only if something valuable to add (high confidence) You can call it an ‘unchatbot’ if you want since there is literally no chatting – but we don’t
- 5. Rather than open-ended responses that come with full conversational interfaces, with the user trying to guess which words the bot understands, we went with a closed-ended responses The AI just lays its cards out right on the table – I know these things – and you don’t need to spend the time trying to guess what I know
- Approximately 1/3rd users do not browse or search – they immediately contact Help Desk. Call deflection key to lowering support costs.
- Middle column (in bold) – common elements
- 4. A corollary to reduced learning curve – correcting mistakes with search (or whatever your base solution is) is a lot easier than trying to converse it out with your average chatbot, which frequently have rudimentary if any understanding of context – a bit like having a conversation with an amnesiac brain damage patient who is delighted to meet you everytime you reenter the room 5 – closed-ended responses are much easier to deal with; with a traditional chatbot you spend time playing 20 questions
- NPS collected every quarter to understand something like “satisfaction” There are better measures –take a look at our poster on NPS Only features added over time – inflection points
- Not surprisingly -- User satisfaction high when users received effective suggestions But low when they did not (frustrated by irrelevant or inapplicable suggestions)
- Focus on training your intercept to display when confidence is high and not to present the user with solutions they’ve already explored whenever possible Finding the sweet spot for confidence level requires lots of iterative measurement and a feel. We used 60% but 70% is generally recommended.
- Lisa is one of our architects who worked on the Watson backend Finding the right delivery vehicle and interaction model is just the start The hard work is on the classic chatbot backend, working with the intents, training the chatbot and ensuring you have a good match with what the users are looking for. She will tell you how we did that. In addition, we will even share our secret sauce – how you can build your own Unchatbot, Lisa?
- Any user speech can be broken down into several components, leading to a full dialog. Utterances are sample sentences that a user might say. Intents are essentially verbs Entities are essentially nouns Any of these can be built with synonyms to account for different ways a user says things Finally , they are combined into a full dialog with the user.
- So how did we do this? (look at left bar) So what does an intent author do? An intent author will research the solution and write a dialog flow tailored for the intended user channel. For example, a dialog that works on a web chat bot might not work very well if the user is calling on the phone. They may need to reach out to experts to find/review answers, and may look to historical ticket data to see how the issues have been answered in the past. The intent author will also list any potential ”training” phrases to Watson, to indicate how users might ask the question for an issue. The intent author will pass the data along to the Watson Conversation developer for development and test.
- A single dialog result is considered a final endpoint or answer Disambiguation is a word with means that we want to ask more questions to understand what the user’s intent is. A Disambiguation node will lead to other branches, sometimes with links to self service and sometimes with further options to define the problem. Recommended by Watson indicates a final response Each node type along the path gives the user an option to give feedback as to whether the suggestion was helpful. This user input is critical because it gives you feedback on optimizing your Watson training.
- The specific Watson product we used is called Watson Assistant which builds complete conversations, and uses the concepts of Utterances, intents, entities, and dialogs to build the conversation and train. However it had limited programmatic capabilities when it came to talking to other systems. In our case we needed to extend our existing IT help system to integrate with Watson. We wanted to display results alongside of the traditional results. And there were security issues that we needed to implement to make the system secure We had different front ends – a mobile app, a traditional html based chatbot, the IBM help phone system, different channels for the user to access the the system Lastly, the components had to be scalable
- For more details on integration architecture see IBM Developer article at https://developer.ibm.com/articles/cc-hybrid-integrating-watson-assistant/ So what we started with is our internal Help@IBM IT help system, which consists of a very robust website along with a content and document management system, and also a rich set of services such as “get me a user’s profile’ or “tell me what are the support options are forson a given product.” All these different services talked to each other and had dependencies on each other, so we created a single entry point as an entry to all these different features. It combines the traditional search results with Watson search results, as well as giving a single entry point to our Watson orchestrator.
- As I said before, Watson Assistant does not lend itself to rich programming capabilities when trying to talk to other systems, so we built an orchestration service to play monkey in the middle. It orchestrates messages going back and forth between Watson and Help@IBM, including changing the data format.
- Cognitive intercept is a new interaction model for chatbots that manages user expectations – raised everyday by amazing products like Alexa and all of the things you can do with it Integrates with existing UI, lowers cognitive overhead of learning a new chatbot Effective at boosting information discovery – hard not to notice! But as with a traditional chatbot, the advice is effective dependent on the quality of the intents and how well they match real the questions Parting advice Don’t show the same intent to the same user across multiple vehicles In the search context, use intercept to indicate when to stop searching Continue to research user needs / build out intents to better anticipate them Keeping adding new intents Refine user utterances to increase ability to distinguish between user topics Only return if a high-confidence intent match Finally, explore additional vehicles for cognitive intercept Next up: Voice interfaces (IVR/VRU), where it can silently screen questions before offering a help desk agent