Keynote at ICWSM-2018 workshop on Chatbots: June 25, 2018, Palo Alto, CA My interest in and vision for Computing for Human Experience is centered on developing and using AI to serve, rather than replace, humans. In order to make technology appear more human-like, it is critical to align technology with human’s multi-sensory capabilities, and correspondingly deal with multimodal information. Two decades ago, Mark Weiser’s ubicomp vision brought welcome advancement in human-computer interaction. Next frontier for more natural interactions between man and machines is voice and language, which I am sure will be followed by integration with other senses, esp. vision, leading to chatbots merging into the broader technology of robots. In the interim, chatbots that interact with humans in more natural ways hold tremendous promise. The bigger challenge in my view is not the “front end” such as speech recognition and transcription, but the “back end” - processing information as a human brain would, making interactions with computing feel more natural to humans. This calls for a further breakthrough in contextualizing and personalizing information exchange between a human and chatbot to better understand the human needs and the actions that can be taken. In this talk, I will focus on chatbots that are narrow but deep (very good in a well-defined application or domain) to help address humans in the way an expert (e.g., a clinician in a healthcare context) would. I will take examples from the augmented personalized health applications we are pursuing at Kno.e.sis using our kHealth technology for achieving better outcomes for managing asthma, post-surgery care, and depression. I seek to explain what a human-like chatbot would be expected to do, how knowledge-enhanced AI and big data approaches may advance the current state of the technologies in Natural Language Understanding (NLU) and Q/A and if successful, how this can demonstrate the promise of the machines serving vital human needs and wants. For Augmented Personalized Health and Computing for Human Experiences: See http://knoesis.org/vision

1. Human-like Chatbots:
Promises, Challenges and Implications
Icon source used in the entire presentation - https://thenounproject.com
Presentation template by SlidesCarnival
Photographs by Unsplash
Prof. Amit Sheth
LexisNexis Ohio Eminent Scholar
Executive Director, Kno.e.sis - Ohio Center of Excellence in Knowledge-enabled Computing
& BioHealth Innovation

2. Machine-centric to Human-centric Computing
Artificial
Intelligence
2
Ambient
Intelligence
Augmenting
Human Intellect
Human-Computer
Symbiosis
Computing for
Human Experience
Machine-centric Human-centric
John McCarthy Mark Weiser Douglas Engelbart Joseph C.R. Licklider
Figure: Views along the spectrum of machine-centric to human-centric computing.
At the far right is our work on Computing for Human Experience, which explores paradigms such as
Semantic, Cognitive, and Perceptual Computing. http://bit.ly/SCP-Magazine
Kno.e.sis Center
http://bit.ly/k-Che,
http://slidesha.re/k-che

3. Machine Replacing Humans, Possible?
3
Source: https://medium.com/@mijordan3/artificial-intelligence-the-revolution-hasnt-happened-yet-
5e1d5812e1e7?token=12BqnkqpquQiaXYw
Source: https://www.wsj.com/articles/ai-cant-reason-why-1526657442
Source: https://www.quantamagazine.org/to-build-truly-intelligent-machines-teach-them-cause-and-effect-20180515/

4. 4
Current AI is Far from Singularity
Source: https://twitter.com/amit_p/status/920361898226446338

5. Computing for Human Experience
5
Source: http://corescholar.libraries.wright.edu/cgi/viewcontent.cgi?article=1919&context=knoesis
http://wiki.knoesis.org/index.php/Computing_For_Human_Experience
“Computing for Human Experience will employ
a suite of technologies to nondestructively and
unobtrusively complement and enrich normal
human activities, with minimal explicit concern
or effort on the humans’ part.”

6. HOW TO MAKE
TECHNOLOGY MORE
HUMAN-LIKE?
Image Source: https://www.flickr.com/photos/gleonhard/28251977573/in/photostream/
Critical to align technology with
human’s multi-sensory capabilities
AND correspondingly
deal with multimodal information

7. Richness of Interactions
between Man and Machines
Natural Language
Voice
CHATBOTS
Language
Source: http://agilemodeling.com/essays/communication.htm
Media Richness Theory

8. 9
CHATBOTS, CONVERSATIONAL AGENTS, DIGITAL ASSISTANTS
http://www.businessinsider.com/the-messaging-app-report-2015-11
“Messaging is the new browser,
and bots are the new websites.”
- Mike Roberts, Kik's head of messenger services
Messaging Apps are where people
spend their time online.
Chatbots are a way to reach them.
http://www.bbc.com/news/technology-35977220
Example Companies Adopting Chatbot
https://www.inc.com/larry-kim/10-examples-of-how-brands-are-using-chatbots-to-de.html
9
Messaging Apps are where people
spend their time online.
Chatbots are a way to reach them.

9. 10
CHATBOTS Applications
(Health) Capturing data that is
otherwise not available to physicians
Personal Assistant
Engaging
Personalized
Automation
Shallow and Broad Narrow and Deep
Types of Chatbot
Opens up a higher dimension for
human and machine
Richness & Expressiveness
ExampleChatbotApplicationsonVariousDomains
INSURANCE Chatbot
Allstate Business Insurance expert (or ABIe)
https://hbr.org/2016/07/how-companies-are-benefiting-from-lite-artificial-intelligence
FINANCIAL Chatbot
Capital One Financial (Eno)
Info on credit card balances, transactions, due dates, and limit
https://www.capitalone.com/applications/eno/
HOTEL Chatbot
Marriott International’s
Book travel in more than 4,700 hotels
http://news.marriott.com/2017/09/marriott-internationals-ai-powered-chatbots-facebook-messenger-slack-
alofts-chatbotlr-simplify-travel-guests-throughout-journey/
THERAPY & HEALTH Chatbot
10
Benefits over Mobile Applications

10. 11
General Outline of Conversational AI Techniques
We are (artificially) intelligent.
We are only as smart as
the words you feed me.
You don’t have to install any apps
to talk to me.
Conversation Intent
Conversational Datasets, Commonsense Reasoning and
Knowledge Ingestion
Natural Language Understanding (NLU) Techniques
● Named-entity Recognition (NER) and Disambiguation
● Sentence Completion
● Topic and Domain Detection
● Implicit Entity Recognition
● Relation Extraction
● Text Summarization
● Sentiment, Emotion, and Intent Detection
● Emoji Sense Disambiguation
● Machine Translation
● Ranking and Selection (Open-domain social conversations)
Response GenerationConversational Topic Tracker
Inappropriate and Offensive Speech Detection

11. 12 Source: http://www.kpcb.com/blog/2016-internet-trends-report

12. “
13
The bigger challenge in my
view is not the “front end” such
as speech recognition and
transcription, but the “back
end” - processing information
as a human brain would,
making interactions with
computing feel more natural
to humans.’

13. “Calls for a further breakthrough in
contextualizing and
personalizing information
exchange between a human and
chatbot to better understand the
human needs and
the actions that can be taken.
14

14. A Deep Learning Based Chatbot implemented using the Seq2Seq model and trained on the
Twitter 2017 and Cornell Movie Dialogs Corpus.
Resources and References:
http://suriyadeepan.github.io/2016-06-28-easy-seq2seq/ http://suriyadeepan.github.io/2016-12-31-practical-seq2seq/
https://github.com/marsan-ma/chat_corpus https://github.com/suriyadeepan/datasets
https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html
15
State-of-the-Art
Seq2Seq
Chatbot
(RNN-LSTM)
❖ As good as the data trained on
❖ But lacks context & personalization
https://youtu.be/wy9lxXW15mQ
VIDEO ON THE
NEXT SLIDE

15. Intelligent Chatbots?
● COGNITIVE UNDERPINNING &
EXPLAINABILITY with Deeper Actor
Model, Domain Model/Knowledge
Graphs and Protocols
● CONTEXTUALIZATION
● PERSONALIZATION
● ABSTRACTION

16. 17
Contextualization
refers to data interpretation in terms of knowledge (context).
Without Domain Knowledge With Domain Knowledge
Chatbot with domain (drug)
knowledge is potentially more
natural and able to deal with
variations.

17. Personalization
refers to future course of action by taking into account the contextual factors such as user’s health
history, physical characteristics, environmental factors, activity, and lifestyle.
18
Without
Contextualized Personalization
With
Contextualized Personalization
Chatbot with
contextualized (asthma)
knowledge is potentially
more personalized and
engaging.

18. Abstraction
A computational technique that maps and associates raw data to action-related information.
19
With AbstractionWithout Abstraction
.

19. 20
CHATBOTS @ KNO.E.SIS
a. ReaCTrack
Personalized Adverse Reaction Conversational-based Tracker for Clinical Depression
http://bit.ly/ReaCTrack
b. kBOT
Knowledge-enabled (kHealth) Personalized ChatBot for Asthma

20. 21
A sample video demo of ReaCTrack. Main objectives are:
(i) Monitor patient’s depression severity score
(ii) Track medication adherence and mood changes over the course of prescribed medications.
(iii) Document ADRs and side-effects from antidepressant medications
(iv) Answer domain-specific (depression) questions, specifically the use and side-effects of
antidepressant using semantic technologies. https://youtu.be/0FrB1hnplmY
A sample video demo of kBOT:
Knowledge-enabled (kHealth) Personalized ChatBot for Asthma
Contextualized & Personalized Conversations from Multimodal data and IoT sensor data
https://youtu.be/0FrB1hnplmY
VIDEO ON THE
NEXT SLIDE
VIDEO ON THE
NEXT SLIDE

21. 22
Sensor, Social, Clinical Datastreams: Informed & Intelligent Questions
Weather information
(temperature, pollen,
humidity, etc)
Elasticsearch (ES)
Database Query & Rule Abstract raw values
into information
Asthma Domain Knowledge
https://bioportal.bioontology.org/ontologies/AO
http://www.childhealthservicemodels.eu
Patient Data from EMR & PGHD
(Compliance score, prescribed
medications, asthma control level)
IoTs (Foobot &
Fitbit)
Conversation Rules & Scripts
(DialogFlow)
Sensor, Social, Clinical Architectural Framework for Intelligent & Informed
Conversations: kBOT Asthma
Sensor (IoTs) & Cyber
Datastream
Clinical (Baseline) Datastream
Patient Consented Social
Data (Facebook, Instagram,
Twitter Activity)
Social Datastream
Knowledge Datastream
★ Smarter & engaging agent
★ Minimize active sensing
(Questions to be asked)
★ Ask only informed & intelligent questions
★ Relevant & Contextualized
conversations
★ Personalized & Human-Like
Human-Like Aspect

22. Contextualization and
Personalization
kBOT initiates greeting
conversation.
Understands the patient’s health
condition (allergic reaction to high
ragweed pollen level) via the
personalized patient’s knowledge
graph generated from EMR, PGHD,
and prior interactions with the kBOT.
Generates predictions or
recommended course of actions.
Inference based on patient’s
historical records and background
health knowledge graph containing
contextualized (domain-specific)
knowledge.
Figure: Example kBOT conversation which
utilizes background health knowledge graph
and patient’s knowledge graph to infer and
generate recommendation to patients.
★ Conversing only information relevant
to the patient
23

23. TOWARDS
HUMAN-LIKE CHATBOTS
A chatbot should have a deeper
understanding of real world entities and
the person it is interacting with, backed
with knowledge and facts.
24

24. What does it mean to be Human-like?
Medication reminder
Option 1:
Alarm alerts patient to take timely medication.
25
Option 2:
Pill sensor bottle detects medications are not taken timely and sounds alarm.
Option 3:
Chatbot knows you have not taken your medications, sounds alarm and pops up a
reminder (optionally, the system even knows you are now in the bedroom which has medicine closet).
Option 4:
Robot: “You seem to have missed your medications. Shall I get you your pill bottle?”

25. But how do we “teach” these bots real world entities?
Not every rule can be programmed, bots have to learn
progressively just like how humans would.
26 Image Source: https://pixabay.com/en/binary-code-privacy-policy-woman-2175285/

26. “
Progressive Intelligence:
A form of intelligence by which
one learns progressively through
continuous stream of data,
understand the Semantic
Associations in a given context,
distill the knowledge, and
synthesize the right decision(s).
27

27. 30
Value of Knowledge-enhanced AI (Research)
Li et al, 2016: Build personalized conversation engine by
adding personal information as extra input.
https://arxiv.org/abs/1603.06155
Xing et al, 2016: Incorporate topic information into the
seq2seq framework to generate informative and
interesting responses for chatbots.
https://arxiv.org/abs/1606.08340
Mazumder, Ma, and Liu, 2018: Propose a general
knowledge learning engine for chatbots to continuously
and interactively learn new knowledge during
conversations (Lifelong interactive Learning and
inference (LiLi)).
https://arxiv.org/abs/1802.06024
With Knowledge
Sordoni et al, 2015: Represent utterances in previous
turns as context vector and incorporated them into
response generation.
https://arxiv.org/abs/1506.06714
Yao, Zweig, and Peng, 2015: Added an extra RNN
between the encoder and decoder of the Seq2Seq model
to represent intentions.
https://arxiv.org/abs/1510.08565
Gu et al, 2016: Introduce copynet to simulate the
repeating behavior of human in conversation.
https://arxiv.org/abs/1603.06393
Without Knowledge
Sheth, Perera, Wijeratne, and Thirunarayan, 2017: Discuss the indispensable role of knowledge for
deeper understanding of complex text and multimodal data. https://dl.acm.org/citation.cfm?id=3109448
Knowledge will Propel Machine Understanding of Content: Extrapolating from Current Examples
Sheth, Thomas, and Mehra, 2010: Continuous Semantics to Analyze Real-Time Data
https://ieeexplore.ieee.org/document/5617065/
Kno.e.sis Center

28. Various distributed
knowledge sources
Update Knowledge
Graph
Core Services
Personalized Health
Management
Evolving Open Health
Knowledge Graph
Disease Progression
Intervention
CDI and CAC
31
http://bit.ly/Knowledge-AI
Role of Knowledge
31

29. Semantic Browsing
Extraction
Data Integration and Interlinking
Entity
Complex Extraction
Aberrant
Drug-related
Behaviour
Neuro-Cognitive
Symptoms
Adverse
Drug
Reaction
Relation Event Severity
Personal Sensor Data De-identified EMR Blog Post
Context Representation Relevant Subgraph Selection
Semantic Search
Disease-specific
Chatbot
Visualization
Health
Knowledge Graph
32
Intent
Open Health Knowledge Graph
32

30. 33
SOCIAL -MEDIA TEXT
(July 12,2016)
EVENT-SPECIFIC
SCHEMA-BASED
KNOWLEDGE

31. 34
Semantic, Cognitive, Perceptual Computing:
Paradigms That Shape Human Experience
http://bit.ly/SCPComputing

32. CHATBOTS IN HEALTHCARE
Putting it all together with 3 Pedagogical Examples
kHealth Asthma, Depression, Elder health

33. 36
LIMITED DATA due to episodic visits
TIME CONSTRAINT during clinical visits
Significant information seeking time is required every time
Comprehending clinical notes every time which contains only
text is difficult
Each individual is DIFFERENT and thus, personalised
treatment is needed
Insufficient time and data for personalization
Image Source:
https://www.istockphoto.com/gb/vector/woman-doctor-examining-patient-by-stethoscope-gm541296730-96809003
WHY Healthcare? [a technology take]
Traditional
Healthcare

34. 37
Solution: Augmented Personalized Health with CHATBOT and IoTs
Medical
Internet
of Things
Patient Generated Health Data (PGHD)
with Medical Internet-Of-Things (IoTs)
Digital footprint representative
of patient’s health
★ Episodic to Continuous Monitoring
★ Clinic-centric to Patient-centric
★ Clinician controlled to Patient empowered
★ Disease Focused to beyond Medical intervention
★ Sparse data to 360 Multimodal data
37 http://bit.ly/APHealthcare
But how to turn these “Big Data” into
Insights into Actionable Information?

35. 40
Example 1: kHealth Asthma
40
Data Sources
Heterogeneous data and collection method
(1852 data points/ patient /day)
Semantic, Cognitive, Perceptual
Computing Framework
http://bit.ly/SCPComputing
Smarter conversations with
actionable meaningful information.
http://bit.ly/kHealth-Asthma

36. “
41
Modern Healthcare is not just
about diagnosing disease and
prescribing medication. Patient-
doctor communication exerts a
placebo “healing” effect on the
patient. (Human-like) Chatbot can
be a useful vehicle to emulate
such relationship.

37. 42
Example 2:
Depression
Chatbot that engages the patient on
a daily basis understands the
emotion and social well-being for
better interactions.
42

38. 43
Example 3: Proposed “Putting-it-all” Together Architecture for
Elder Care
43

39. 44
In Short,
❖ Far more useful in interacting compared to other alternatives (browser) when
physician is not available.
❖ Emulate social-humanistic element.
Promises
❖ Chatbots that interact in more natural ways hold tremendous promise.
❖ Opportunities in disease-specific health, general fitness, and well-being.
Challenges
❖ The biggest challenge is to process information as a human brain would.
❖ Multi-sensory capabilities, multimodal information
❖ Contextualization, Personalization, Abstraction
Implications

40. 46
Special Thanks
Hong Yung (Joey) Yip
(Graduate Student)