1.
CIO Design, IBM
Chatbot in a box:
Introduction to conversation design and
conducting research on chatbots
Presenters: Jon G. Temple, Ph.D., Jason Telner, Ph.D., & Patricia Orsini
Contributing material: Dabby Phipps
Workshop@UXPA
IBM, CIO Design

2.
Who we are
Dabby Phipps (co-author; PoohBot creator)
Senior UX Designer/Researcher
Patricia Orsini
AI Experience Content Designer
Jason Telner, PhD
Senior UX Researcher
Jon G. Temple, PhD
IBM Design Principal

3.
Welcome!
• Introductions
• Today’s agenda
• Group activity: What do you want out of this workshop?

4.
What is a chatbot /
conversational agent /
virtual agent / digital
assistant?
“I don’t know that. Let me search the web for
you…”

5.
The first chatbot
was created in
the mid 1960’s…

6.
Meet ELIZA:
The first chatterbot
• ELIZA - natural language conversational
program (1966) by Joseph Weizenbaum
• Simulates Rogerian psychotherapist
• ELIZA takes the user input, looks for
keywords and synonyms, creates a
response - often a transformation of the
user’s input2
• ELIZA is not intelligent and can’t learn.
• May not make sense or be
grammatically correct
• Comp Sci 101 project in Pascal
1https://www.masswerk.at/elizabot/

7.
…as of 2018, there
were over 3k chatbots
…on Facebook alone

8.
Chatbots Today
• Software programs - leverage artificial
narrow intelligence (ANI)
• Answer user questions or perform
certain tasks within a narrow scope
• Natural language processing (NLP) -
utterance
• For chatbots, the key is natural
language understanding (NLU) - to
understand the user’s “intent” or what
they’re trying to achieve3
https://www.bold360.com/features/chat-and-messaging

9.
Why the Explosive Growth?
• Primarily the emergence of two
technologies:
• Messenger applications
• Artificial Intelligence / Machine
Learning

10.
Quick Facts1
• It's estimated that 1.4 billion people
now use chatbots on a fairly regular
basis. (Acquire)
• In 2020 it was estimated that $5 billion
would be invested in chatbots by 2021.
(Chatbots Magazine)
• Up to 64% of internet users surveyed
said round-the-clock support is the
biggest benefit of chatbots. (Drift)
• If the chatbot can't solve a problem,
79% of people want access to a real
person.

11.
Chatbot Myth
Watson on Jeopardy (2011)5

12.
What Users Really Want
• Businesses may want an AI with
personality or conversational
• Users aren’t looking for a virtual buddy
because they are lonely
• Users just want answers or assistance
- efficiently as possible
• For example, people using an IT help
system rated the same system
different based solely on outcome
(NPS: 31 v. -32)
• Best solution ever! – if it helps me;
otherwise, it is the worst!!
• If chatbots or a conversational UI can
provide the appropriate answers
faster, they will be preferred

13.
The Pros and Cons of Chatbots: The Business Perspective
Pros
• Automate simple conversations
• Available 24x7
• Multiple customers simultaneously
• Great for small businesses
• User in control
• NLP conversational style of interaction
Cons
• Hard to support broad domains
• Poor performance leads to decreased user
confidence
• Takes time and effort to build the knowledge
corpus
• May require natural language when not needed

14.
The User Perspective: What Do They Like/Dislike?
Likes
• High availability
• Not having to talk to a person
• No wait time
• Able to use conversational language
• Still seems high tech and kind of cool
Dislikes
• Not able to talk to a person
• Forced to use conversational language when other
approaches may be more efficient
• Being told it will search the web
• Failure to use context
• An indirect answer:
“Watch this 10-min video!”
”Read this long document!”
• Having to guess the chatbot’s domain
• Failing to get an answer / not having expectations
met (high risk of abandonment)
• A complex and cluttered UI that belies the
promised simplicity of a conversation

15.
Relevant Design Roles
UX researcher
Visual design
Interaction/UX design
Conversation design

16.
How these roles work together
UX researcher
The Business:
“I want a chatbot!”
Validate business
assumptions with
users
Discovery phase –
What do users
need?
Write new cognitive
intents / entities/
dialogs
Validate with users
UX researcher
UX researcher Conversational designer

17.
Activity (2 mins)
Think back to a recent experience with a chatbot
• What did you find useful?
• What needed improvement?

18.
The knowledge of
chatbots and the
conversation designer

19.
Where does the chatbot's knowledge come from?
• Cognitive theories of mind -
“homunculus problem" as a straw man
• Cognitive processes - little people
inside your head
• Recursion - “what’s inside the head of
the homunculus?”
• Yet - this is true for chatbots:
knowledge comes from conversation
designers (aka people)
• Chatbots aren’t reading textbooks,
forums and writing own answers
• Machine learning - (if used) to improve
the query

20.
Chatbot knowledge
It comes from people!*
*Soylent Green (1973)

21.
Role of Conversation Designer
• Conversation designers need to
determine:
• Chatbot’s purpose
• Type of interaction (i.e., playful,
neutral, professional)
• The user intents within the domain
• Craft concise responses - engage the
user and address their need

22.
Be engaged. Feel empowered. Get excited.
Conversation Design
IBM | CIO Design
The job of the conversation designer
Engage the user Make it a good experience Keep them coming back

23.
Determine the Chatbot’s Purpose
• End users
• Solve user problems
• Business goals
• Orgs’s vision
• Specific job or jobs for the chatbot

24.
Tone of Voice
• Dimension of formality:
• Informal & friendly: "Hi there, how
can I help you?”
• More formal, still friendly: "Good
morning. How may i help you today?
Type your question in the box below
to get started."
• Formal, not very friendly: "This
service is designed to answer your
question in the box below.”
• Consider: Your brand, target audience,
tone of other content sources used by
the solution, nature of the specific
questions and responses

25.
What Are Your Users’ Intentions?
• Why are users interacting with your
chatbot? What tasks are they trying to
complete? What goals are they trying to
achieve?
• Once you’ve identified the intents, the
next step is to create a list of user
examples or how your users might state
their intentions.
• A minimum of five user examples
should be created for each intent, but
10-15 is better.

26.
Chatbot Components
• Why are users
interacting with your
chatbot? What tasks are
they trying to complete?
What goals are they
trying to achieve?
• Example:
• The user wants to
reset their email
password.
Intents Utterances
Entities Dialog
• What are the nouns the
user might want to take
action on? What are
synonyms for those
nouns?
• A minimum of five user
examples should be
created for each intent,
but 10-15 is better.
• Examples:
• #Entity : synonyms
• Email : mail, outlook,
notes, verse, e-mail,
apple mail
• For each intent, create a
list of user examples or
how your users might
state their intentions.
• Examples:
• I need to reset my
@email password.
• I want to change my
@email password.
• How do I change my
@email password?
• Error says to reset my
@email password.
• Matches the user’s
utterance (user’s stated
goal) with a dialog
response (chatbot’s
response).
• Example:
• [user] How do I reset
my email password?
• [chatbot] Outlook now
uses your w3id and
password to sign in.
To change your w3
password ….

27.
Common intents
Insurance Real Estate Finance/Banking Retail Employee/HR
Receive insurance quote Find information on
available property
Ask about a charge Learn about product or
service details
Learn about benefits
Change insurance plan Find information on listing a
property
Pulling up information on
accounts
Check on order status and
delivery times
Find out information on
internal ticketing processes
Get help with insurance
policy information
Provide recommendations
on real estate vendors
Report fraud Find out how to return a
product
Find out subject matter
experts within an
organization
Upgrade service or contract Find out return policy Review information on a
ticket
Learn about service details Report missing products Find information about
customers and
relationships
Troubleshoot an issue Find information on
organization’s services and
products

28.
Example
Utterance: Show me yesterday’s financial news
Intent: #showNews
Entities/slots: @dateTime @newTypes
Response: < Stocks are down again. Here are the financial news. >

29.
Chatbot Flow Overview
• Intents - user’s intention or what they
want to do (verb)
• Entities - keywords modify the intent
(noun) - the what
• Dialog - response generated by the
chatbot based on its understanding of
the user’s input using intents and
entities
User input
(utterance)
Bot assesses
user intent
Bot identifies
best solution
Bot provides
response (dialog)

30.
Role of Training and Machine Learning
• Training does not create chatbot knowledge
• Machine learning and training improves discoverability
• If a user asks a question, machine learning improves the likelihood the correct
answer will be found (if it exists)
• The AI assigns a “confidence level”
• Should the confidence level not meet the threshold (e.g. .60), the response
isn’t displayed

31.
How to identify user needs
for cognitive intents:
Research!

32.
Intent Creation Process
Prioritize backlog
Identify end user
phrases for the issue
Determine entities
Talk to SME’s &
review support materials
Generate utterances
Create flow charts
Write dialog
Work with developers to
map out API integrations
Add intents, entities, and
dialog to chatbot system
Manual and automated
testing

33.
Example: Researching a new intent
Identify top call drivers
from support tickets
“We get a lot of calls about
resetting two types of
passwords. I have a script that I
read to callers”.
Interview human support
agents
Input from SME’s Search data
User feedback New content/products Data science analysis
“We have an API that can be
used to reset the intranet
password easily; but to reset your
email password, you need the
vendor to provide us new
services”.
“I have too many different kinds
of passwords to keep track of
and I often forget to change them
on time”. — Ima Uzer
Top call drivers for August:
1. Email
2.Reset intranet password
3. VPN…
Top search queries:
1. Outlook, HCL Notes, Apple mail
2. Checkpoint
3.Intranet password change,
reset…
Added 200K Macs to user base.
New support content written for:
Mac account access, Filevault,
Screentime, Apple ID…
Data science report:
Patterns found in multiple sources -
surveys, searches: password reset
complexity in multiple contexts,
including…
Reset password

34.
Overview of user research for chatbots –User needs and pain points “empathy” research to
determine intents
User interviews
Contextual inquiry
Journey Maps
Personas
Friction analysis
Surveys
Task completion /Help desk calls
Comment analysis
NPS scores

35.
• Testing intents with users
o Creating a user test plan
o Best practices for testing intents
o Using field metrics once your chatbot is deployed
Chat bot in a box | UXPA 2022
Overview of user research for chatbots

36.
Historical data –NPS scores, task
completion/help desk calls and
survey comment analysis
36

37.
0 1 2 3 4 5 6 7 8 9 10
0% 0% 0% 14% 14% 0% 0% 0.0% 0% 30% 43%
Promoters – Detractors = NPS
= 73-28 = 45
How likely are you to recommend application X to a friend
or colleague?
• Net Promoter Score (NPS) – A measure of
how likely someone is to recommend a
service or application
• Typically asked in surveys
• Should have a minim N of 30 respondents
for acceptable Margin or Error
• Comments most useful for recognizing
reasons for low NPS i.e. detractor
comments
Net Promoter Score (NPS)

38.
w3U – IT Support
User Research
IBM | w3 Unification
4Q20
4Q21

39.
Top Task Failures Goal Completion Analysis
IBM | CIO Design
0 50 100 150 200 250 300
IT Support: Ticket_open_update_close_status
Search: Other
Search: IT_support_help
IT Support: Contact_support_call_chat
IT Support: Other
News: Other
IT Support: Password_access_login
Top Task Failures
No Partially Yes

40.
w3U-IT Support: Goal Completion
.
Other:

41.
w3U-IT-Support: Survey Comments – Primary Category
Reasons Goal Not Completed + Suggestions for Improvement
• 26%: Ticket & Help Desk issues
• 21%: Search + Specific Topic issues
• 7%: Technical issues (Failed to load, Too
Slow, Login Failed, etc)
• 7%: Missing features
• 4%: Difficult / Cumbersome / Complicated
127 responses were collected, 127 responses were provided to this question. A random sample of 127 responses were analyzed, which resulted in 132 comments.

42.
w3U-IT-Support: Survey Comments – Secondary Category
Reasons Goal Not Completed + Suggestions for Improvement

43.
User interviews and contextual
inquiry
43

44.
User interviews
• Better for discovery research
• Asking users questions typically with a moderator 1:1
• Great for understanding your users, their tasks,
sentiments, needs and pain points.
• Great for understanding which intents might be helpful
for their needs and pain points
• Can be used for persona creation and journey mapping
• Limited in how many can conduct but get rich feedback
Chat bot in a box | UXPA 2022

45.
Contextual inquiry
• Better for discovery research
• Observing users in the work environments and the tasks
they do
• Can observe their needs and pain points for their tasks
• Can observe and ask questions
• Often can be a segway to journey maps
• Limited in how many can conduct but get rich feedback
Chat bot in a box | UXPA 2022

46.
User interview scenarios for discovery research
1) You find out that Slack is currently down, where do you go to learn about and resolve this problem.
Probe on:
• Where would you go first? Why there?
• Where else would you go?
• Any problems/difficulties?
• How could it be improved?
2) You are experiencing a problem with JIRA and cannot update your work items. Where do you go to learn about and resolve the issue?
Probe on:
• Where would you go first? Why there?
• Where else would you go?
• Any problems/difficulties?
• How could it be improved?
3) You cannot access your Outlook calendar and urgently need to find out your meetings for the day. Where do you go to learn about and resolve the issue?
Probe on:
• Where would you go first? Why there?
• Where else would you go?
• Any problems/difficulties?
• How could it be improved?
Chat bot in a box | UXPA 2022

47.
User interview scenarios for discovery research
Scenario Pain points for searching for
solution
You find out that Slack is currently down, where do you go to learn
about and resolve this problem.
• Not clear when search “Slack support” in w3 search where to find solution in search results
must search for “support” in w3 search then get support options
• No phone/chat listed directly on support page
• Might visit Slack.com instead of using w3
• Nothing about outages on IT support page
• Nothing to directly create ticket or chat/call support fom IT support page
You are experiencing a problem with JIRA and cannot update your
work items. Where do you go to learn about and resolve the issue?
• Do not see correct w3 search results for “JIRA update work items” or JIRA support” –JIRA
help desk query not accurate result
• Would first go to JIRA website as external product
• Need search “support” in w3 search, then get support options, nothing directly on page no
chat or phone, must search JIRA, then
• Prefer use Google search than w3 search is better
• expect see outages/notifications on w3 page
You cannot access your Outlook calendar and urgently need to find out
your meetings for the day. Where do you go to learn about and resolve
the issue?

48.
Journey mapping for pain points
48

49.
Journey mapping for pain points
• Useful for visualizing a user's workflow in their tasks from
beginning to end and plotting where their pain points and
problem areas occur in the task flow
• Data can be gathered from user interviews and
contextual inquiries
• Can be useful when performing friction analysis in terms
of breaking the task down and which points to keep
aware of for pain points and increased friction
Chat bot in a box | UXPA 2022

50.
Journey map for searching for IT information for application problem
User navigates to w3 and look for
notifications/outages
User goes to w3 search for issue and
look for relevant results
User navigates to get support options and
chat/phone/community/create ticket
Phase 1 Phase 2 Phase 3 Phase 4
1
2
3
4
5
6
User navigates to w3
website
User looks for
notifications/outages
User searches w3
for issue
User tries to find relevant
search result
User navigates to IT
support to get support
options
User finds appropriate
application they need help
with
7
User begins chat…not clear
they will get a ticket
Pain point
Pain point
Chat bot in a box | UXPA 2022

51.
Search w3 for “Slack
support” or “Slack
is down”
Search w3 for
“support”
Slack@IBM channel Slack.com
Slack at IBM -
Support
Slack User Guide w3
Support
Support
Chat/call IBM Help
IT Support (formerly
Help@IBM)
Get support options
Slack app
Chat/call IBM Help
Search for
outages./notifications
Call co-worker
Google search
“Slack is down”
Search for IT information for Slack is down
Post question in
channel
Look for anything on
outages or
notifications
Ask if their Slack is
down
Slack system status
page
Chat bot in a box | UXPA 2022

52.
Search w3 for “JIRA
cannot update work
items” or “JIRA not
updating”
Search w3 for
“support”
JIRA.com
No useful search
results
IT Support (formerly
Help@IBM)
Get support options
Search for JIRA
Chat/call IBM Help
Search for outages
notifications
Google search JRA
Search for IT information for JIRA cannot update items
Create ticket only if
cannot resolve
Atlassian page
Search issue get
documentation
Search w3 for “JIRA
support”
JIRA help desk
query
Nothing useful
Atlassian page
Expect FAQ common
issues encountered
Chat bot in a box |
UXPA 2022

53.
Search w3 for
“cannot see Outlook
calendar”
IT support link Got o Slack
Go to Outlook
calendar first results
Search for Outlook
calendar
Read relevant
articles if nothing
then chat/call
support
Chat/call support
Create ticket last resort
only if articles,
chat/call cannot solve
or do not give them
ticket after chat/call
Search for IT information for Outlook cannot view calendar
Post question in
channel –UXRA
channel about Outlook
or use IBM support
Slack channels
IT support tab if
nothing from Slack
channels
Search w3 for “IT
support”
Get support options
Outlook
Go to Outlook
calendar first results
Support- Community
link post question in
there
Chat/call support
Chat/call support
assumes create
ticket at the end.
Get support options
Outlook
Chat/call support
assumes create
ticket at the end.
Chat bot in a box | UXPA 2022

54.
Personas
54

55.
Personas
• Can be created from data collected from user interviews,
contextual inquiry and surveys
• User personas are realistic representations of your key
user groups that you can use for reference when making
design and feature decisions.
• At their core, personas are about creating products with
specific users in mind.
• The personas should reflect the current state of a product
and how users interact with it today; they should not be
used to demonstrate how a user will interact with the
system in the future.
Chat bot in a box | UXPA 2022

56.
Friction analysis
56

57.
Friction analysis and PURE method
• Friction analysis is a quantitative method that breaks
tasks down into individual steps and provides a
quantitative rating of the complexity and difficulty in
completing a task.
Chat bot in a box | UXPA 2022

58.
Friction analysis and PURE method
Benefits Drawbacks
• Working under time constraints
• Large number of tasks that need to be tested
and compared
• Not feasible to test users due to time
constraints or length of testing required
• Quantitative measure of task complexity and
pain points most important
• Just doing simplified friction analysis lacks
deeper qualitative feedback you get from user
interviews
• You need at least one more rater for interrater
reliability, need to resolve disputes
Chat bot in a box | UXPA 2022

59.
Friction analysis and PURE method
• PURE = Practical User
Experience Rating by
Experts
• Measure of friction, 3-point
score
• Total of scores for each
step in a task
• We will modify it for CIO
purposes
Chat bot in a box | UXPA 2022

60.
Friction analysis and PURE method
w3DA December 2021
w3DA May 2021
Including the steps
link in the w3DA
results in more
direct solution
rather than going to
devices first.
“I want to badge in at an IBM printer”
Chat bot in a box | UXPA 2022

61.
Surveys
61

62.
Surveys
• Can be used to validate findings from discovery research
and for personas.
• Can be used to gather qualitative and quantitively data
around user needs, pain points, sentiments and attitudes
toward towards applications, user tasks, and chatbots.
• Can collect data from hundreds of respondents
• Can be useful for creating personas using cluster
analysis.
• Can perform sentiment analysis on the open-ended
questions.
Chat bot in a box | UXPA 2022

63.
Method
• Survey was fielded from March 8 –March 16, 2022.
• Survey took 5-10 minutes to complete.
• 600 IBMers were sent an invite to the survey recruited from the UXRA participant
database (DIG).
• Used the UXRA survey platform to send out the invites.
• 265 respondents responded to the survey (got nearly the 30% completion rate as
expected).
Chat bot in a box | UXPA 2022

64.
Rank ordering of importance of various features that would be most useful for an IBM
conversational AI
Top ranking
of features
for
conversation
al AIs at IBM
were focused
on IT support
Some comments that had this question been
worded using “digital assistants” the rankings
would have been different and more focused on
tasks rather than conversational features.
Chat bot in a box | UXPA 2022

65.
Explanation for ranking of features for an IBM conversational AI
18.9%
15.0%
14.2%
7.1%
5.5%
4.7%
4.7%
4.7%
3.9%
3.9%
3.9%
3.1%
3.1%
2.4%
0% 5% 10% 15% 20% 25%
Support for problems
Order of usefulness
Day to day activities most important
New hire tasks most important
Other items are easy
Schedule meetings most important
No need for it
answer questions
need help fast
Notifications most important
Most complex
outages
Great for automating tasks
Random
% of Total Categorized Comments (N=127)
with 90% Confidence Intervals
253 responses were collected, 119 responses were provided to this question. A random sample of 119 responses were analyzed, which resulted in 127 comments.
Question: Explain the reason for your ranking.
Top reasons for
rankings of the
items were support
for problems most
important, ranked
for order of
usefulness, and day
to day activities
most important.
Chat bot in a box | UXPA 2022

66.
Activity (15 mins)
• Create a test plan with appropriate methods for identifying user needs

67.
Conversational
intents: Best
practices

68.
What Makes a Chatbot Good?
• Objectives as a conversation designer:
• Easy
• Intuitive
• Intentional language choices
• Feel more like a conversation with
another human
• User walks away feeling satisfied

69.
Writing Tips
• Active voice
• Use contractions
• Keep sentences short
• Use formatting to improve readability
(bulleted lists, paragraphs, commas,
dashes)
• Check for misspelled words and other
grammar issues
• Avoid “Yes" and "No" answers
• Incorporate part of the user's question
in response
• Be succinct and accurate answers (link
out to website for longer responses)

70.
Erika Hall's Eight Principles for Conversation(al) Design
Cooperative
Turn-based
Goal-oriented Context-aware Quick & clear
Truthful Polite Error-tolerant

71.
Principle 1: Cooperative
• Back and forth flow
• Mimic how people have conversations -
pattern understood by the user

72.
Principle 2: Goal-oriented
• Main purpose - help users reach their
goals
• Chatbot should reflect this the user’s
intentions
• Help your customers achieve their goals
helps your business achieve its goals

73.
Principle 3: Context-aware
• Don’t rely on overly automated
messages
• Can frustrate users
• Does not feel like a natural conversation
• Builds confidence in the user
User: What are your holiday hours?
Chatbot: Our normal hours of operation
are …..

74.
Principle 4: Quick and clear
• Conversational, but as efficient as
possible
• Use plain language and keep it simple
• Offer logical next steps
• Keep the conversation moving
• Stay on point
User: How many flavors of ice cream do
you have?
Chatbot: We have a huge selection of ice
cream flavors so we’re sure we have
something for everyone.

75.
Principle 5: Turn-based
• Replies - succinct and to the point. No
monologues!
• Do not send a string of responses
• Best conversations are back and forth
dialogs
• Be clear if response from the user is
needed
• Validate user input to avoid errors

76.
Principle 6: Truthful
• Validate are accurate and truthful
• Plan for dialog maintenance. Answers
change over time - may need to be
updated
• Verify it is behaving as expected
• Responses should align with user
expectations

77.
Principle 7: Polite
• Don’t be rude
• Be respectful of your user’s time
• Users come to a chatbot with a goal.
Stay on topic
• Don’t try to push your own agenda if not
aligned
• Once the chatbot has solved the user’s
issue, offer to help with other issues,
next steps

78.
Principle 8: Error-tolerant
• To errr is human, to forgive divine.
• Teach chatbot common spelling
alternatives or errors
• Disambiguate the request as needed
User: How do I reset my pwd?
Chatbot: To reset your password…

79.
Example of a poorly designed chatbot experience
Hi, how can I
assist you
today?
I need help
with my wifi
connection.
I’m not sure how
to help you with
that.
Can I speak
with an agent?
We have a promotion
happening for 20%
your monthly bill…
Here are a few links that may
be helpful:
www.IT.resourcesdoc.com
www.helpIT.com
Hi, I need help
with my internet
connection…
Hi, I need help
viewing my
Internet bill
Your bill can be found by
going to your billing under
your account from the main
menu.
The chatbot
provides no
introduction and
does not instruct
what it can and
cannot help with
The chatbot
provides no
alternatives and is
not empathetic to
its lack of
knowledge
The chatbot ignore
the request for an
agent and
references an
irrelevant
promotion
The chatbot pushes generic
links, nothing specific, more
searching required on part
of user and no better than a
search with results
Answer is disorganized,
lacks steps and
sequencing.

80.
Best practices designing chatbots
• Make sure your users know upfront what the chatbot can
do and set expectations.
• Providing specific answers to specific questions, don't just
ignore the user’s question or push generic links. The
snippets in Google search -- those are quite specific
answers, short concise, and well formatted.
• Present solutions in their task sequence of operations and
label the step number in the solution with a number.
• Consider removing obvious solutions or solutions the user
could easily find elsewhere.
• Ensure the chat bot follows up quickly with users after it
gives an instruction or action and does not leave them
hanging.
• Ensure your chat bot has an engaging personality that
resonates with your users.
• Use buttons to allow users to quickly choose options
instead of typing them out.

81.
Best practices designing chatbots
• Chat bot replies should be written from first person perspective
as a human would respond.
• Ensure your chat bot breaks up information by mimicking the
flow of regular human messaging conversations
• Ensure buttons are functional at all points of the conversation
• Have your chat bot account for multiple word choices, phrasings
and start options to allow for flexibility of interaction with your
users.
• Design your chat bots for possible misunderstandings at every
step
• Have the chat bot clarify the user’s question or answer if it
detects an ambiguity before offering an answer, through a
related question to remedy misunderstandings.
• Consider including a ”did you mean” or please try rephrasing
your question if the chat bot does not completely understand
the user’s question.
• Do not lock your users into a conversation with your chatbot and
make it easy for your user to get to a human agent or back a
main menu.

82.
Types of cognitive
intents

83.
Response Types
• Substantive - answers the question (q:
what's your favorite movie? a: My
favorite movie is Bladerunner b/c …).
Core topics.
• Deflecting - responds to the question
without actually answering (q: what's
the weather like in london? a: humans
must be fascinated by the weather b/c
they're always asking me about it!)
Non-core topics.
• Redirecting - redirects the user back to
the core areas (q: what do you think of
religion? a: religion is not my area of
expertise, but I do know about ABC
banking cards!)

84.
Response Types: Disambiguation intent
Confusion between 2 or more intents, with similar training phrases (utterances). Create
an intent that helps system find the correct intent.
When there is a vast list of supported options, conversational and open ended. But when
the supported list is small, just list the options in a menu – don’t play 20 questions.
“I want to buy something”
Huge list of things you can buy
Good open ended/conversational:
What is it you want to buy?
Bad menu:
[Food] [Water] [Cars] [Weapons] [Rocks]
[Plants] [Pets] [Livestock] [Real
estate] [Bitcoin] [Bridges] [Gardening]
[Books] [Movies] [Tickets] [Vacations]
[Storage] [Tech] [Photo] [Drug] […] […]
“I want to reset my password”
Lots of types, just 3 supported
Good menu:
Which password do you want to reset?
[Mac] [iPhone] [IBMid]
Bad open ended/conversational
What kind of password is it?

85.
System statements
• Welcome statements – sets the tone; consider
return visits and time of day; may also inform user
about the chatbot domain.
• Interleaving solutions – follows final answers and
encourage more questions; consider variations and
useful tips.
• Topping and tailing statements – indicates trust
level, or whether more info is needed.
• Return statements – return the user back to a
process flow.
• Did not know statements (low confidence or no
match)
“Hi, I’m HAL 9000. Welcome to Discovery Two. I see this
is your second trip to Jupiter. You may ask me anything
about your stay on the ship.”
“Unfortunately, it is not possible to open the pod bay
doors at this time. Got another question, just ask me!”
“Here’s some information I found on the internet about how
much spare oxygen an escape pod can hold. Check it out!”
“Going back to where we were before you were accidentally ejected into
space, would you like to resume our chess game? You were winning.”
“I’m sorry but I do not know how you will survive without
oxygen.”

86.
Chit Chat & Small Talk
• Exclamations (yes, no, okay, sure, oh)
• Personal
• Topic-based (e.g. Weather, geography)
• Threats, insults, and pleas for help
• Explicit and offensive language
• Compliments
• Complaints
• Fallback or Repair (I don't understand,
what do you mean, why)

87.
Planning Dialog Flow
• Ok! Now we’re ready to start thinking
about the chatbot’s dialog!
• Before you start writing the exact
words, think about how the
conversation with your users will flow
for each intent you’ve identified.
• The mapping will help you understand if
there are redundant dialog nodes across
scenarios and how your scenarios might
interact.

88.
Best practices for dialog flow
• Consider the most common situations –
Conversational scripts need to account
for the most common scenarios. User
research can help identify these.
• Prepare different versions of an answer
– sounds less robotic and more like
natural conversations, which are varied,
diverse, novel, or dynamic.
• Work on the sequence of dialogue –
Once you know the scenarios, think
through the sequence of the dialogue so
appropriate answers will be provided for
specific queries.
Source: https://www.revechat.com/blog/chatbot-scripts/

89.
Practice Activity (10 mins)
Writing intents for ordering pizza

90.
Instructions: Design an intent
• 5-7 mins/3-5 mins discussion
• Use the worksheet
• User’s intent: Ordering a pizza with toppings (#orderPizza)
• Generate up to 3 utterances for that intent
• Define toppings as an entity (@toppings)
• Create different values for that entity
• Create some synonyms for each value

91.
Utterances for #orderPizza
• i want a pizza
• i'd like a pizza
• give me a pie
• please give me a pizza
• order a pizza pie

92.
Entity/slots
@toppings:
[value: synonyms]
meat: pepperoni, sausage, ham, anchovies, chicken
veggies: black olives, green olives, onions, bell peppers, jalapenos, artichoke
cheese: mozzarella, white cheddar, asiago, three cheese blend
values synonyms
Entity label (category)

93.
Utterances for #orderPizza(updated)
• i want a @toppings pizza
• i'd like a pizza with @toppings
• give me a @toppings pie
• please give me a @toppings pizza
• order a pizza pie and I want @toppings on it

94.
Responses within the dialog
- Excellent choice! Pizza is our specialty. Do you want toppings? Yes | No
Yes: Alright. Three toppings come for free. Each additional topping is $.50. List of
available toppings: [list]
No: jump to size
- Sounds like you want a [variable] pizza. Do you want any other toppings? Yes | No
Yes: Alright. Three toppings come for free. Each additional topping is $.50. List of
available toppings: [list]
No: jump to size
- Let's make sure I have this right. You want a pizza with [selected options]. Is that correct?
Yes: Great! What size do you want? [@size]
No: Oops! Let's go over the toppings again. What toppings do you want on your pizza?
- Ok, so you want a large. Is that correct? Yes | No
Yes: Would you like anything else? We also have appetizers, desserts, and drinks.
No: Sorry, I must have mozarella in my ears. What size pizza do you want?

95.
Dialog Flow

96.
PoohBot demo

97.
Intent

98.
Entities

99.
Synonyms

100.
Dialog

101.
Team Activity (20 mins)
Writing intents for the uses of honey

102.
Instructions: Design an intent
• 15 mins/5 mins discussion
• Divide into teams
• Use the worksheets
• User’s intent: Things to do with honey (#thingsToDo)
1. Generate up to 5 utterances for that intent
2. Define honey uses as an entity (@uses)
3. Create different values for that entity
4. Create some synonyms for each value
5. Craft some responses for the entity or entity:value
(@uses or @uses:value)
BONUS INTENTS
1. Write a disambiguation intent related to
#thingsToDo
1.What is the new intent?
2.What is the entity?
3.Any synonyms?
4.What are the utterances?
2. Write an example of a chit chat response
with PoohBot’s tone

103.
Utterances for #thingsToDo
• can I put @honey in my @uses?
• can you @uses with @honey?
• can you make @uses with @honey?
• how is @honey @uses?
• is @honey only good for @uses?
• what @uses are good with @honey?
• what @uses can you make with @honey?

104.
Entity
@uses:
[value: synonyms]
antiseptic: clean wounds, cleaning wounds, boo boo, ouchie
cooking: baking, salad dressing, marinade
drinks: tea, milk, coffee, kool-aid, lemonade
mead: alcohol, cocktail

105.
Responses in Dialog
Entity level
@uses
Hunny has so many great uses I smear it on everything
Entity:value level
@uses:antiseptic
Whenever I get a boo boo, I put hunny on it, but I always forget and lick it off
@uses:drinks
Tea just isn't the same without hunny
@uses:mead
Mr. Owl really likes mead but he always acts so silly afterwards
@uses:desserts
Hunny makes yummy desserts, especially baklava
@uses:cooking
Piglet likes to marinade himself in a sauce made with hunny

106.
Bonus intents!
Disambiguation intent:
Choose a type of honey
Intent name:
#chooseType
Entity:
@honeyType:
[value: synonyms]
honeyType: acacia, buckwheat, clover, eucalyptus, orange blossom, tupelo, sage
Utterances:
- I want to try some @honeyType @honey
- My favorite @honey is @honeyType
Chit chat:
- On dear, hunny has so many uses, doesn’t it?
- So wonderful you enjoy hunny as much as I do

107.
Validating cognitive
intents with users

108.
Creating a user test plan
• What is the research problem?
• What are goals of research session
• What are you testing?
• What are scenarios?
• How will you test participants?
• Number of participants (typically 6-10)
• Participant requirements
• Outline procedure
• Introductory text to read to participants (length of
session, expectations, privacy, tasks…)
• Background questions
• Exploratory tasks
• Specific scenarios
• Probe questions
• Post test questions
Chat bot in a box | UXPA 2022

109.
Chat bot in a box | UXPA 2022

110.
Chat bot in a box | UXPA 2022

111.
Chat bot in a box | UXPA 2022

112.
Best practices for testing intents
• Test the Chatbot Onboarding Process
• Test the Chatbot Using Generic Templates
• Check Link Navigation
• Test the Chatbot’s Conversational Flow
o Does my chatbot understand the user’s questions?
o Does it respond promptly to them?
o Are its responses accurate enough and relevant?
o Are there enough conversation steps or… too many?
o Does it keep the user engaged?
• Run a Chatbot-Error Handling Test
• Test the Chatbot’s Context Awareness
• Test Text with Alternative but Valid Responses

113.
Field Metrics
Months when
users are most
and least active
with chatbot
queries
Months when
new users come
use the chatbot
the most and
least
Months when
users abandon
the chatbot the
most

114.
Field Metrics Buttons were
used more than
typing in this
chatbot.
25% of queries
fell below the
confidence
threshold set an
could not be
answered.

115.
Activity (15 mins)
• Update your test plan with methods for validating intents

116.
Revenge of the
PoohBot

117.
Activity (10 mins)
Interactive demo: Your honey intents played back in PoohBot

118.
Bonus content!!

119.
How is writing intents for voice bots different than for
chatbots?
• As a general rule of thumb it is recommendation to
design voice bots that minimizes demand on working
memory and attention required because of their reliance
on the auditory system.
• Keep conversation short so users do not have to
interrupt and less reliance on working memory
• Ensure user responses are brief as well
• Use brief pauses liberally so users can being speaking
without interrupting
• Ask only necessary questions
• Obtain information one piece at a time in consumable
pieces
• Be sensitive to word synonyms
• Be considerate to speech accents
• Privacy and background noise might be issues

120.
Failure to meet user
expectations
• New chatbots lack a full blown knowledge
corpus on launch
• Build out the knowledge corpus to answer direct
questions (e.g., point to a long video for the
answer)
• In 2018, Gartner expected that 40% of first-
generation chatbot/virtual assistant applications
launched in 2018 would be abandoned by 20208
• A chatbot that fails to live up to user
expectations will quickly lose adoption
(abandonment)

121.
Cognitive Intercept: Overview
• With cognitive intercept9, embed the
chatbot technology into an existing user
experience (single point of entry)
• Monitors in the background, assesses the
user’s input
• Intercepts if it has high confidence
• Deploy a chatbot that covers a narrow
scope of topics without frustrating the
user
• Continue to gather user input to increase
its breadth and depth of coverage, as well
as improving accuracy
User initiates
action
High
confidence
match?
Continue with
standard
experience
Intercept
Cognitive layer
screens input
in background
Yes No

122.
Cognitive Intercept: Real World Example - No Match
• Where cognitive intercept has been used:
• Web search (first integration point)
• Voice Response Unit (VRU)
• Online chat
• Users enter their input in a standard
search field
• System evaluates the input as both
traditional search terms and using NLP
• If a high-confidence match is not found,
the “chatbot” remains silent
• Running in the background effectively
manages expectations around what
chatbots can or cannot deliver

123.
Cognitive Intercept: Real World Example - Match
• If the chatbot is trained on a particular
user intent and finds a high-confidence
match, then the chatbot “intercepts” and
displays associated dialog
• Full conversations using NLP are avoided
by using close-ended prompts to
represent user options
Cognitive intercept benefits:
• Manages user expectations
• Provides a single point of entry
• Reduces errors associated with NLP
• Permits a graceful ramp up of the
chatbot’s knowledge corpus

124.
References
https://www.masswerk.at/elizabot/
https://www.chatbotpack.com/how-eliza-chatbot-works/
https://www.bold360.com/learn/what-is-natural-language-processing
https://www.smallbizgenius.net/by-the-numbers/chatbot-statistics/#gref
https://www.csmonitor.com/Science/2011/0214/IBM-s-Watson-Can-a-computer-outsmart-a-Jeopardy!-braniac
https://chatbotsmagazine.com/to-build-a-successful-chatbot-ask-these-5-questions-b7fe3776c74
https://marvelapp.com/blog/principles-of-conversational-design/
https://www.gartner.com/smarterwithgartner/2-megatrends-dominate-the-gartner-hype-cycle-for-artificial-intelligence-
2020/
Temple, J.G., Elie, C., Schenkewitz, L., Nezbedova, P., Phipps, D., Roberson, K., Scherpa, J., & Malatesta, M. (June, 2019. Not
Your Average Chatbot: Using Cognitive Intercept to Improve Information Discovery. Paper presented at the UXPA
International Conference, Scottsdale, AZ.