Keynote @ 2018 AAAI Joint Workshop on Health Intelligence (W3PHIAI 2018), 2 February 2018, New Orleans, LA [Video: https://youtu.be/GujvoWRa0O8] Related article: https://ieeexplore.ieee.org/document/8355891/ Abstract Healthcare as we know it is in the process of going through a massive change - from episodic to continuous, from disease-focused to wellness and quality of life focused, from clinic centric to anywhere a patient is, from clinician controlled to patient empowered, and from being driven by limited data to 360-degree, multimodal personal-public-population physical-cyber-social big data-driven. While the ability to create and capture data is already here, the upcoming innovations will be in converting this big data into smart data through contextual and personalized processing such that patients and clinicians can make better decisions and take timely actions for augmented personalized health. In this talk, we will discuss how use of AI techniques on semantically integrated patient-generated health data (PGHD), environmental data, clinical data, and public social data is exploited to achieve a range of augmented health management strategies that include self-monitoring, self-appraisal, self-management, intervention, and Disease Progression Tracking and Prediction. We will review examples and outcomes from a number of applications, some involving patient evaluations, including asthma in children, bariatric surgery/obesity, mental health/depression, that are part of the Kno.e.sis kHealth personalized digital health initiative. Background: Background: http://bit.ly/k-APH, http://bit.ly/kAsthma, http://j.mp/PARCtalk

1. Augmented
Personalized Health
using AI techniques on semantically integrated multimodal data for
patient empowered health management strategies
2018 AAAI Joint Workshop on Health
Intelligence (W3PHIAI 2018) @ AAAI 2018
Keynote on Feb 2, 2018
Prof. Amit. P. Sheth
LexisNexis Ohio Eminent Scholar, Executive Director,
Kno.e.sis
Wright State University
Background: http://bit.ly/k-APH, http://bit.ly/kAsthma, http://j.mp/PARCtalk
Icon source used in the entire presentation - https://thenounproject.com

2. Kno.e.sis:
Ohio Center of Excellence in Knowledge-enabled Computing & BioHealth Innovation
NSF
● Harassment on Social Media
● Citizen & Physical Sensing
● Twitris - Collective Intelligence
● Aerial Surveillance
● Visual Experience
● Web Robot Traffic
NIH
● kHealth - Asthma
● eDrugTrends
● eDarkTrends
● Depression on Social
Media
● Drug Abuse Early Warning
DoD & Industry
• Metabolomics &
Proteomics
• Medical Info Decisions
• Human Detection
on Synthetic FMV
• Sensor & Information
• Material Genomics
• Cardiology Semantic
Analysis
15 faculty from 4
colleges + ~60 Funded
Students
• 40 PhD
• 16 MS
• 10 BS
Kno.e.sis conducts research in AI
techniques that convert physical-
cyber-social big data into smart data,
enabling building of intelligent
systems for clinical, biomedical,
policy, and epidemiological
applications.
Example clinical/healthcare
applications include major diseases
such as asthma, obesity,
depression, cardiology, dementia
and GI.
This is complemented by social and
development challenges such as
marijuana legalization policy,
harassment on social media,
gender-based violence, and disaster
coordination.
Kno.e.sis’ research in World Wide Web ranks Wright State University among the top 10 organizations in the world based
on 10-yr impact [MAS: 2016]. Its total budget for currently active projects is $13+ million [2017]. World-class
interdisciplinary research is complemented by exceptional student outcomes and commercialization with local economic
impact. 2

3. Active Healthcare Projects in Kno.e.sis
kHealth
for
Asthma
kHealth
for
Bariatrics
kHealth for
Dementia
Depression
EDrugTrends
EDarkNets
Alchemy
for
Healthcare
Knowledge Graph
Development
Platform
Contextual Knowledge
Representation
3

4. HealthChallenges
(Also, Dementia,
Obesity,
Parkinson’s, Liver
Cirrhosis, ADHF )
Public Policy/
Population
Epidemiology
Personalized
Health
PCS + EMR
kHealth
Asthma in Children
Bariatric Surgery
Physical(IoT)/Cyber/
Social (PCS)+ EMR
Marijuana Social
Drug Abuse Social
Mental Health
Depression (Suicide) Social + Public + EMR
Health Knowledge
Graph
Services
Social + Clinical Data
Health Related Studies at Kno.e.sis [overview]
...and infrastructure
technologies: Context-
aware KR (SP), KG
development, Smart
Data from PCS Big Data,
Twitris,....
4

5. Goal of Medicine [a revisionist view]
1) Ultimate goal of
medicine: fight
and prevent
diseases
3) Predict: must
know how one
reacts in order to
predict
2) How?
Outcome has to
be predicted in
order to be
prevented
5) But, each
patient is
different.
Generalised
approach is not
efficient
6) Personalised
data collection -
Augmented
Personalised
Health
7) Personalised
data - accurate
prediction - better
prevention
4) Data must be
collected over a
period of time to know
the reaction of the
patients to particular
disease
5

6. Traditional Healthcare [a technology take]
- Limited data due to episodic visits
- Time constraints in knowing the patient
during clinical visits
- Significant information seeking time is
required every time
- Comprehending clinical notes every time
which contains only text is difficult
- Each individual is different and thus,
personalised treatment is needed
- Insufficient time and data for personalization
Image Source - https://www.istockphoto.com/gb/vector/woman-doctor-examining-patient-by-stethoscope-gm541296730-96809003 6

7. Health Care: Transition
Material Paternalism Therapeutic and
Preventive Alliance
Certain health decisions are
best left in the hands of a
clinician
Clinician-patient interaction for
effective treatment to control and
prevent disease
To
Image source(left) - https://www.freepik.com/free-vector/medicine-pills-in-a-blister_1250631.htm 7

8. Therapeutic and Preventive Alliance
- Relationship between health care professional and the patient
- Patient and clinician hope to engage in making clinical decisions for the patient
- Improves patients outcome as the patient is educated about the disease
Challenges
- Difficult to know patients well only with clinical visits due to time constraints
- Frequent interaction about the patient health with clinician is necessary
Solution
- Insights from Patient Generated Health Data (PGHD) enabled by Internet of Things (IoT) devices,
community, clinical and public health data (Physical, Cyber Social and Clinical Data)
8

9. PGHD with Internet of Things
a digital footprint representative of patient’s health
● By 2020, 40% of Internet of Things based devices
will be related to health care [1]
● Monitors health indicators specific to patients
disease
● Enables remote and continuous monitoring for long
term chronic illness
● Multimodal data
● Integrated and standardized technology backbone
enables disease surveillance and treatment support
Medical
Internet
of Things
[1] Dimitrov, Dimiter V. "Medical internet of things and big data in healthcare." Healthcare informatics research 22.3 (2016): 156-163.
9

10. Healthcare: Before and After IoT (+ mobile + social)
● Episodic to Continuous
● Clinic - centric to Patient - centric
● Clinician controlled to Patient empowered
● Disease Focused to beyond Medical intervention
● Limited data to 360 Multimodal data
10

11. Big Data
• Diverse and multimodal data - Patients, Social
and Clinical data
• Eight weeks of data from seven sensors
collected for 16 Parkinson’s patients is about
12 GB of data [kaggle]
• According to expert forecasts, smart devices
will produce one-tenth of the total amount of
information on Earth, up to 44 ZB in 2020
[sam-solutions]
11

12. Big Data into Smart Data
making sense of the data
Massive amount
multimodal data
collected from
various medical IoT
sensors
Turning Big Data into Insights into Actionable information
Analyze the data
to find out what
the data tells
about the patient
Provide timely
actionable
information
specific to patient
disease
12
Smart data makes sense out of Big data : It provides value from harnessing the challenges posed by volume, velocity, variety and veracity of big data, in-turn providing
actionable information and improve decision making.

13. Patient Generated Health Data to action
Contextualization, Abstraction and Personalization
SSN
Ontology
Interpreted Data
(abductive)
[OWL]
e.g., diagnosis
3
Annotated Data
[RDF]
e.g., label
Raw Data
[Text]
e.g., number
Interpreted Data
(deductive)
[OWL]
e.g., threshold
1
0
2
13

14. Augmented Personalised Health
Big Data to Smart Data
Augmented Personalised Health(APH) is a vision to enhance the healthcare by using AI
techniques on semantically integrated PGHD, environmental data, clinical data, public health data
& social data.
Data Components
PGHD, Clinical data, environmental
data and Social Data
Smart Data
meaningful data obtained after
contextualised processing
Image Source - http://www.dqchannels.com/big-data-and-smart-data-big-drivers-for-smart-decision-making/ 14

15. Smart Data: Data with Knowledge
Smarterdata
Data Sophistication
Smart data should answer:
- What causes my disease severity?
- How well am I doing with respect
to prescribed care plan?
- Am I deviating from the care plan?
- I am following the care plan but my
disease is not well controlled. Do I
need treatment adjustments?
- How well controlled is my disease
over the time?
15

16. Health Management Strategies of APH
1. Self
Monitoring
Constant and remote
monitoring of disease
specific health
indicators for any given
patient
2. Self
Appraisal
Interpretation of the
data collected with
respect to disease
context for the patient
to evaluate
themselves
3. Self
Management
Identify the deviation from
normal and assist
patients to get back to
prescribed care plan
4. Intervention
Change in the care plan
- with the converted
smart data by APH,
provide decision support
for treatment
adjustments
5. Disease
Progression and
Tracking
Longitudinal data
collection and analysis
to enhance patients
health over the time
Sheth, et.al. How will the Internet of Things enable Augmented Personalized Health?
16

17. APH Applications [@Kno.e.sis, currently involving patient evals]
Asthma
• 6.3 million children in USA are affected by
Asthma; 300 million adults & children
worldwide [CDC]
• Non-adherence to medication makes it one
of the poorly controlled disease
• Multifactorial disease
• Difficult to diagnose based on episodic visits
and clinical records
Bariatric Surgery
● 36% of the adults in the United States
suffer from obesity [CDC]
● 65% of the world’s population lives in
countries where the occurrence of
death due to overweight and obesity is
higher than being underweight
[ASMBS]
● Bariatrics surgery is one of the
efficient ways to reduce weight
● Weight recidivism: significant subset
of patients regain weight due to non
adherence to post surgical guidelines
17

18. APH for Asthma and Bariatrics: patient centric drivers
This not only prevent the disease, but also enhances the patient’s health
BariatricsAsthma
18

19. kHealth Asthma
A multisensory approach for
personalised asthma care in
children
kHealth Bariatrics
A multisensory approach to
support patient post surgical
progress
In order to enhance patient’s health care by providing answers to
those questions...
Also, kHealth for Dementia, (Asthma-Obesity), (ADHF), (GI-liver cirrhosis)
19

20. kHealth-Asthma
20

21. kHealth Asthma Overview
Data Sources
Heterogeneous data and
collection method
Smart Data
Actionable meaningful information
from the data collected 21

22. kHealthDash
Knowledge-enabled Personalized DASHboard for Asthma Management
22

23. Duration Number of Patients Compliance
One month 40 patients 30 patients - above 75%
3 Months 1 patient 100%
Duration Number of Patients Compliance
3 Months 15 Patients 5 patients - 100%
10 patients - above 75%
Completed Trials (45 patients)
Total Data points - 1057 data points (42 patients)
Ongoing Trial
Data from Tablet Data from Tablet
Sensors Data points / day For 1107 days
Fitbit 8 8856
Peak Flow Meter 2 2214
Data Collection [ as of Jan 2018]
1325 days for 41 patients, average of compliance = 89%
which is 1107 days
23

24. Data Collection
Parameter # of data
points/day
Total
Pollen 2 /day (Every 12
Hours)
854
PM 2.5 24/day (Hourly) 10248
Ozone 24/ day (Hourly) 10248
Temperature 24/day (Hourly) 10248
Humidity 24/day (Hourly) 10248
Outdoor Environmental data
Dec 01, 2016 to present (as of Feb 2, 2018) = 427
days
Parameters # data points
/day
Total
CO2
288
(Every 5 minute,
24 x 12)
381600
(1325 days for 41
patients)
Volatile
Compounds
Temperature
Humidity
PM 2.5
Global Pollution
Index
Indoor Environmental Parameters of Completed Trials
collected using Foobot (41 patients)
Total = 381600 x 6 (Parameters) = 2289600 data points
and still collecting from current deployments
Total - 41, 846 data points and still
collecting...
24

25. Data Collection per day per patient
Active sensing
Tablet
Symptom - 6
Short acting med - 1
Long acting med - 1
Total - 8 x 2(twice a day) = 16
Peak Flow meter = 2 (twice a day)
Total = 16+2 = 18 data points/day
Passive sensing
Foobot
CO2, VOC, Humidity,
Temperature, PM2.5,
Global Pollution Index
Fitbit
Sleep - 4 (REM,Light sleep,Deep sleep, # minutes active)
Activity - 4 (minutes active, sedentary minutes, minutes lightly active, #
steps)
Subtotal - 8
Outdoor Parameters
Ozone, PM2.5, Temperature, Humidity = 24 x 4 = 96
Pollen = 12
Subtotal = 108
Total = 1844 data points / day
288(every five
minutes) x 6 = 1728
Total number of data points per patient per day = 18 + 1844 = 1862 data points/ day
25

26. 26
Abstraction
Actionable
Information
Google Virely watch collects physiological data [http://bit.ly/2ox4ZdA]
Modality of Data
Measure relevant signals for studies spanning cardiovascular, movement disorders, and other areas.
Examples include electrocardiogram (ECG), heart rate, electrodermal activity, and inertial movements.
Cohort/ Size/ Status
To be used in other clinical studies
IBM and Docdok.health to track progression of
Lung disease like COPD
Modality of Data
IoT device, to record symptoms and vital-signs,
such as cough intensity, sputum (saliva and
mucus) color, lung function, breath rate and
heart rate, oxygen saturation, as well as activity.
Cohort/ Size/ Status
100/ planned (2018) [https://ibm.co/2yeoDlT] Stanford Wearable Study for early
disease diagnoses (Lyme disease)
Modality of Data
Heart rate, skin temperature, and sleep
data
Cohort/ Size/ Status
60/ completed [http://bit.ly/2jMfplL]
kHealth Active Decompensated Heart
Failure (ADHF) to reduce readmission
Kno.e.sis-Wexner-Ohio State U.
Modality of Data
Mobile app Q/A (tablet), BP monitor,
scale
Cohort/ Size/ Status
Proof of concept/ on hold [6] kHealth Dementia
Kno.e.sis-Wright State Physicians
Modality of Data
Mobile app Q/A (tablet), sleep sensor,
number of steps, location, gait speed, gait
parameters including speed and cadence.
Cohort/ Size/ Status
40 (validation) + 20 (longitudinal)/
ongoing [8]
Technology Integrated Health Management
(TIHM) for Dementia
Led by Surrey and Borders NHS Foundation Trust
Modality of Data
Survey questions, sensors to measure physiological
symptoms (e.g. Blood pressure), physical activity,
and environmental data; the target is to reduce the
hospital admissions and provide more enhanced
care and support.
Cohort/ Size/ Status
Planned 350 (with technology) + 350 (without
technology)/ ongoing [http://bit.ly/2AKRYlc]
kHealth Asthma in Children for monitoring asthma control and
predict vulnerability (in future, self management)
Kno.e.sis-Dayton Children’s Hospital
Modality of Data
Mobile app Q/A (tablet), forced exhaled volume in 1 sec (FEV1), peak
expiratory flow (PEF), indoor temperature, indoor humidity,
particulate matter, volatile organic compound, carbon dioxide, air
quality index, pollen level, outdoor temperature, outdoor humidity,
number of steps, heart rate and number of hours of sleep. Also clinical
notes.
Cohort/ Size/ Status
200/ ongoing (~50% complete)
kHealth Bariatrics Pre and Post Surgery monitoring and self adherence
Kno.e.sis-Wright State Physicians
Modality of Data
Mobile app Q/A (tablet), weighing scale, pill bottle sensor, water bottle
sensor for reminder to drink water, number of steps, heart rate and
number of hours of sleep. Also clinical notes.
Cohort/ Size/ Status
TBD/ ongoing [5]
Example efforts involving PGHD and other
health relevant data
Data

27. Anecdotal Evidence using kHealthDash
27

28. Anecdotal Evidence using kHealthDash
Factors Frequency Probability
Pollen 7/7 1
Ozone 6/7 0.857
PM2.5 1/7 0.14
Symptoms - 6, Medications - 3
Total events - 7 (2 medications have been taken on the same day as symptom occurrence)
Patient gets symptom when the pollen is high and there is 87% and 10% chance that
the patient will get symptom when ozone and PM2.5 are high respectively.
To account for multiple
factors, we will be using
Regression in the future.
28

29. 29
Computing Predictors
Medications
Activity
Temperature
Humidity
Pollen
Air Quality
Spirometry
Outdoor, Indoor & Medical
(Predictors)
Logistic
Regression
Model
[A x1+ B x2+ C x3…..]
Weights
Computed
Cough
Cough
Symptoms
Outcome

30. Patient Health Score
How controlled is my asthma? How vulnerable am I today?
Patient Generated
Health Data (PGHD)
Population Signals
(Environmental
Parameters)
Public Signals
(Social Media Data)
Risk Assessment
Model
Domain Knowledge
Well Controlled
Moderately Controlled
Poorly Controlled
30

31. Bariatrics
Obesity
• 65% of the world’s population lives in countries where the occurrence of death due to overweight
and obesity is higher than being underweight
Problem
● Chances of regaining weight as stomach can still expand after surgery
● Continuous monitoring of the patients by the surgeon is very essential
Challenges Post Bariatric Surgery
● Patient acceptance and active participation involving continuous monitoring of the patient
● Cost and reimbursement models
● Challenging research in understanding of variety of data over long period
31

32. 32
A system capable of:
● remotely and continuously monitor patients
● identify non-compliance before and after surgery
● nudge/assist for better compliance for improved outcomes and
reduce recidivism
Solution: kHealth Bariatrics

33. kHealth Bariatrics
Pill Bottle Sensor
Reminds patients to
take their pills and
records it
Fitbit
(Activity, Sleep
and Heart Rate)
kHealth Bariatrics
App
Diet and Emotional
well being through
contextual questions
Water Bottle Sensor
Reminds patients to
hydrate and records
it
Bluetooth Weighing
Scale
Records patients weight
and send it to cloud
33

34. kHealth Post-Bariatric Surgery Proposed Method
Aggregate the data collected from the sensors, questionnaires and use AI
techniques to:
● analyse and predict the deviations that could cause the post surgical
complications and,
● serve as an assistant leading to better patient-compliance and
outcomes
34

35. How do we solve problems with real world complexity, gather
vast amount of data, diverse knowledge…. and come up with
intelligent decisions that works for an individual at a given
time?
next: a pedagogical take
35

36. Semantic Cognitive Perceptual computing
36

37. Interplay between Semantic, Cognitive and Perceptual Computing (SC, CC and PC) with examples
More here- Video, Slides
Semantic Cognitive Perceptual computing - use case: Asthma
37

38. Thank you
Special Thanks
kHealth Team Members
Revathy Venkataramanan
(Graduate Student)
Utkarshani Jaimini
(Graduate Student)
Hong Yung Yip
(Graduate Student)
Vaikunth Sridharan
(Graduate Student)
Dipesh Kadaria
(Graduate Student)
Quintin Oliver
(Undergraduate
Student)
Tanvi Banerjee
(Faculty)
Dr. KrishnaPrasad
Thirunarayan
(Faculty)
Clinical Collaborators
Dr. Maninder Kalra
(Pulmonologist at Dayton
Childrens Hospital)
Dr. Joon Shim
(Bariatric Surgeon at Miami Valley
Hospital)
The Project kHealth Asthma is funded by NIH 1 R01 HD087132-
01 38

39. 39