Semantic, Cognitive, and Perceptual Computing – three intertwined strands of a golden braid of intelligent computing

Amit Sheth
Kno.e.sis – Ohio Center of Excellence in Knowledge-enabled Computing:
Wright State University, Dayton, Ohio
Semantic, Cognitive, and Perceptual Computing:
three intertwined strands of a golden braid of intelligent computing
Keynote @ Web Intelligence 2017, Leipzig, Germany, 24 Aug 2017
http://webintelligence2017.com/program/keynotes/

http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7433358

Credit: Looi Consulting (http://www.looiconsulting.com/home/enterprise-big-data/)
● In 2008, data generated > storage available. Less than 0.5% of data gets analyzed.
● Vast variety of data: text << images << A/V < genome sequencing < IoT
● Of all the data generated, which data is relevant, and why? Which data to analyze? Which
data can offer insight? Who cares for what data? How to get attention to a human decision
maker? What we need is intelligent processing to get actionable, smart data.
A Big Challenge and Opportunity in Recent Times

How would an enterprise get actionable information?
http://www.slideshare.net/NamrataChatterjee/nokias-supply-chain-management-case-study, http://www.economist.com/node/7032258
● Weak crisis judgement.
● Failure to take prompt action.
● Single supplier reliability.
Fire at Royal Philips electronic
semiconductor plant, New
Mexico in March 2000.
8 trays of wafers
containing the
miniature circuitry to
make several
thousand chips for
mobile phones was
destroyed.
The expected time to recover was estimated to be a week.
● Fire breakout in clean room.
● Inability to determine the exact
damage to the clean room.
● Lack of emergency preparation.
● Early speculation of possible crisis.
● Preparedness against supply crisis.
● Finding alternative source of chip
supply.

http://www.technologyreview.com/featuredstory/426968/the-patient-of-the-future/
The Patient of the Future
MIT Technology Review, 2012
How would an individual get actionable information?

Smart Data
I introduced the term in 2004; redefined in 2013:
http://wiki.knoesis.org/index.php/Smart_Data

● The astounding bandwidth of
your senses is 11 million bits of
information every second.
● In conscious activities like
reading, the human brain
distills approximately 40 bits of
information per second.
http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/mobile-white-paper-c11-520862.html
The Brain: Inspiration for Intelligent Processing:
What if we could automate such interpretation of data?

● How can we take inspiration from human brain and derive an intelligent
processing of big data
● Take inspiration from cognition and perception, and for pedagogy, outline
three computing paradigms for building future intelligent systems focused
on converting data into abstractions that humans act upon for making
decision and taking action
● CHE: Humans and machines partner to enhance human experience
* 2010 article: http://wiki.knoesis.org/index.php/Computing_For_Human_Experience; distinct from “Human Experience of Computing by Booch:
Dec. 9, 2011 at http://computingthehumanexperience.com/category/computing/.
Sheth - 2008+ *
What is this talk about?

http://www.research.ibm.com/cognitive-computing/brainpower/, https://newtonsapplevce.wordpress.com/2016/01/06/neuromorphic-technology/
Recreating the brain in silicon
One Approach to Intelligence

A Second Approach to Intelligence

Two Use Cases: Healthcare and Traffic

Prior Medical Knowledge
D1
D3
D1
Medical History and
Past observations
S1
S2
S3
Sn
..
..
D1
D2
D3
S1
S2
S3
..
..
..
..
..
..
..
..
D1
D1
Doctor Patient
Q1
Q2
Qn
A1
A2
An
Blood Pressure
Heart Rate
Breathing Rate
Body Temperature
Multi-model Observations
Current: Observing a Snapshot of the Patient

Personalized Health and Objectives: one size does not fit all
Millions of people - > one treatment
Wearable and Sensor data

1Marcus, Philip, Kevin R. Murphy, Abid Rahman, and Christopher D. O’Brien. "Intrapatient symptom variability in
adults and children with asthma: Results of a survey." Advances in therapy 22, no. 5 (2005): 488-497.
“…pediatric patients report variability in asthma symptoms
over time, even when asthma medications are taken.”1
Near Future: Analyzing a Multifaceted Continuous Stream of Diverse Data
Personal level
Signals
Public level
Signals
Population level
Signals

ACTIONS
situation awareness useful
for decision making
ABSTRACTIONS
make sense to humans
KNOWLEDGE
for interpretation of observations
Contextualization
Personalization
DATA
Observations from machine
and social sensors
Converting Data to Actions

Semantics, perception, and cognition interact seamlessly.
● Semantic Computing can deal with basic big data challenges - make data more
meaningful, improve interoperability and integration.
● Cognitive Computing can use relevant knowledge to improve information
understanding for decision-making.
● Perceptual Computing can provide personalized and contextual abstractions
over massive amounts of multimodal data from the physical, cyber, and social
realms, enabling action.
https://www.linkedin.com/pulse/perceptual-computing-third-strand-golden-braid-amit-sheth
Semantic Computing, Perceptual Computing, Cognitive Computing

http://www.mezzacotta.net/garfield/?comic=1470, https://in.pinterest.com/pin/222435669066482336/
● Semantics is the “meaning or
relationship of meanings, or relating
to meaning ” (Webster).
● Meaning and use of data
(Information System).
● It is concerned with the relationship
between the linguistic symbols and
their meaning or real-world objects.
Semantics

Semantics attaches meaning to observation by providing a definition
within a system context or the knowledge that people possess.
Semantic Computing encompasses the technology required to represent
concepts and their relationships in an integrated semantic network that
loosely mimics the brain’s conceptual interrelationships.
Web of data
Semantics attached to
objects in the world
Semantic Computing

Population Level
Personal
Wheeze – Yes
Do you have tightness of chest? –Yes
ObservationsPhysical-Cyber-Social System Health Signal Extraction Health Signal Understanding
<Wheezing=Yes, time, location>
<ChectTightness=Yes, time, location>
<PollenLevel=Medium, time, location>
<Pollution=Yes, time, location>
<Activity=High, time, location>
Wheezing
ChestTightness
PollenLevel
Pollution
Activity
Wheezing
ChestTightness
PollenLevel
Pollution
Activity
RiskCategory
<PollenLevel, ChectTightness, Pollution,
Activity, Wheezing, RiskCategory>
<2, 1, 1,3, 1, RiskCategory>
.
.
.
Expert
Knowledge
Background
Knowledge
tweet reporting pollution level
and asthma attacks
Acceleration readings from
on-phone sensors
Sensor and personal
observations Signals from personal,
personal spaces, and
community spaces
Risk Category assigned by
doctors
Qualify
Quantify
Enrich
Outdoor pollen and pollution
Public Health
Well Controlled - continue
Not Well Controlled – contact nurse
Poor Controlled – contact doctor SSN
Semantic Computing for Our Personalized Digital Health (Asthma) Application

Cognition is the process by which an autonomous system perceives its environment, learns
from experiences, anticipates the outcome of the events, acts to pursue goals and adapts to
the changing environment. It is the result of a developmental process through which the
system becomes progressively more skilled and acquires the ability to understand
events, contexts, and action…
https://mitpress.mit.edu/books/artificial-cognitive-systems
PerceptionAction
Anticipate
Adapt Assimilate
Cognition is a cycle of anticipation, assimilation,
adaptation: embedded in, contributing to, and benefitting
from a continuous process of action and perception.
What is Cognition?

DARPA launched a Cognitive Computing project in 2002. Cognitive Computing was defined as the
ability to “reason, use represented knowledge, learn from experience, accumulate knowledge,
explain itself, accept direction, be aware of its own behavior and capabilities, [and] respond in a
robust manner to surprises.”
IBM describes the components used to develop, and behaviors resulting from, “systems that
learn at scale, reason with purpose and interact with humans naturally.” According to them,
while sharing many attributes with the field of artificial intelligence, it differentiates itself via the
complex interplay of disparate components, each of which comprise their own individual mature
disciplines [1].
Our Take
Cognitive computing uses annotated observations obtained from Semantic computing , or raw
observations from diverse sources and makes it meaningful in context –it provides the ability to
understand events, contexts, and action. Cognitive computing systems learns from their
experiences and improve when performing repeated tasks.
[1] https://en.wikipedia.org/wiki/Cognitive_computing
Cognitive Computing

http://www.rogerschank.com/fraudulent-claims-made-by-IBM-about-Watson-and-AI
Need to be Cautious with Claims (Roger Schank)

“where we see the world we do not decide to see it” Daniel Kahneman
Perception is an active cyclical process of exploration
and interpretation.
Perception enables individual to focus on most promising course
of action by incorporating background knowledge that provided
a comprehensive contextual understanding.
Perception

http://www.ted.com/talks/blaise_aguera_y_arcas_how_computers_are_learning_to_be_creative#t-345661
Image of a bird and a computer says it’s a bird.
Perception is an active imagination.
Perception: Google

Perceptual computing is the ability for a computer to recognize what is going on around it.
Computer can perceive the environment and the users in that environment. The computer
determines what needs a user might have and react to those needs without giving or
receiving any additional information.
Perception: Intel

http://www.jfsowa.com/pubs/cogcycle.pdf, http://www.jfsowa.com/talks/interop.pdf
Peirce’s Cycle of Pragmatism
John Sowa’s Perception Cycle

Perceptual Computing supports the ability to:
● Ask contextually relevant and personalized questions to explore and interpret
observations.
● Builds upon Semantic Computing and Cognitive Computing by providing the
machinery to ask the next question or derive a hypothesis.
● It helps generate abstractions that lead to actions.
Perceptual Computing: Our View

Hyperthyroidism
Elevated
Blood
Pressure
Systolic blood pressure of 150 mmHg
“150”
... ...
Levels of Abstraction

Making Sense of Physical-Cyber-Social Data With
Henson, et al. An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web, Applied Ontology, 2011.
Our First Approach to Implement Perceptual Computing

Explanation is the act of choosing the objects or events that best account for a set of
observations; often referred to as hypothesis building
Discrimination is the act of finding those properties that, if observed, would help distinguish
between multiple explanatory features
1
2
Explanation and Discrimination

Explanatory Feature: a feature that explains the set of observed properties
elevated blood pressure Hypertension
Hyperthyroidism
Pulmonary Edema
Clammy Skin
Palpitations
Explanation

Discriminating Property: is neither expected nor not-applicable
clammy skin
Hypertension
Hyperthyroidism
Pulmonary EdemaPalpitations
elevated blood pressure
Discrimination

Augmented Personalized Healthcare (APH) is expected to enhance healthcare by
personalizing the use of all relevant Physical, Cyber, and Social data obtained from
wearables, sensors and Internet of Things, mobile applications, electronic medical
records, web-based information, and social media for better health for an individual.
Converting big data into smart data through contextual and personalized processing
such that patient and clinician can make better decisions and take timely actions for
Data include traditional clinical data, PGHD and public health data, as well as environmental and social data that
could impact an individual’s health.
Augmented Personalized Healthcare

kHealth Kit: Android Application
kHealth Personalized Digital Health

Providing actionable information in a timely manner is
crucial to avoid information overload or fatigue
Sleep data Community dataPersonal
Schedule
Activity data Personal health
records
Data Overload for Patients/Health Aficionados

For collecting observations from both machine sensors
and from patients in the form of a questionnaire.
kHealth Kit: Android Application

kHealth Dashboard: a visual environment to identify correlations

Domain Knowledge
ICS= inhaled corticosteroid, LABA = inhaled long-acting beta2-agonist, SABA= inhaled short-acting beta2-agonist;
*consider referral to specialist
Asthma Control and Actionable Information
Asthma Domain Knowledge

Personal level
Signals
Public level
Signals
Population level
Signals
Domain
Knowledge
Risk Model
Events from
social streams
Take medication before
going to work.
Avoid going out in the
evening due to high pollen
levels.
Contact a doctor.
Analysis
Personalized
Actionable
Information
Data Acquisition &
aggregation
kHealth: Health Signal Processing Architecture

Risk assessment
model
Semantic
Perception
Personal level
Signals
Public level
Signals
Domain
Knowledge
Population level
Signals
GREEN: Well controlled
YELLOW: Not well controlled
Red: Poor controlled
How controlled is my asthma?
Patient Health Score (Diagnostic)

Risk assessment
model
Semantic
Perception
Personal level
Signals
Public level
Signals
Domain
Knowledge
Population level
Signals
Patient health
Score
How vulnerable* is my control level today?
*considering changing environmental conditions and current control level
Patient Health Score (Prognostic)

Foobot – for monitoring
environmental air quality
Wheezometer – for monitoring
wheezing sounds
Can I reduce my asthma attacks at night?
What are the triggers? What is the wheezing level?
What is the propensity toward asthma?
What is the exposure level over a day?
Commute to work
Luminosity
CO Level
CO in gush during
the daytime.
Actionable
Information
Personal level
Signals
Public level
Signals
Population level
Signals
What is the air quality indoors?
Close the window at home during day to avoid
CO2 inflow, to avoid asthma attacks at night
Decision Support for Doctors and Patients: A Scenario

Another Example
with a Focus on a Richer Model

We are still working on the simpler representations of the real world!
http://artint.info/html/ArtInt_8.html, http://en.wikipedia.org/wiki/Traffic_congestion
Solve
Represent Interpret Real world
Simplified representation
Compute
What did not change in data processing for quite some time?

We need computational paradigms to tap into the rich pulse of the human
populace, and utilize diverse continuous stream of data
Represent, capture, and compute with richer and fine-grained representations of
real-world problems
Solve
Represent
Interpret Real world
Richer representation
Compute
+
Richer representation of
traffic observations.
Effective solutions
People interpreting a
real-world event.
What should change?

By 2001 over 285 million Indians lived in cities, more than in all North American
cities combined (Office of the Registrar General of India 2001) 1.
1 The Crisis of Public Transport in India
2 IBM Smarter Traffic
Modes of Transportation in Indian Cities
Texas Transportation Institute (TTI)
Congestion report in U.S.
Severity of the Traffic Problem

• What time to start?
• What route to take?
• What is the reason for traffic?
– Wait for some time or re-route?
Questions Asked Daily

Image credit: http://traffic.511.org/index
Multiple Events
Varying influence
Interact with Each Other
Challenge: Non-linearity in Traffic Dynamics

7 × 24
LDS(1,1), LDS(1,2) ,…., LDS(1,24)
LDS(7,1), LDS(7,2) ,…., LDS(7,24)
.
.
.
di
hj
Mon.
Tue.
Wed.
Thu.
Fri.
Sat.
Sun.
Mon.
Tue.
Wed.
Thu.
Fri.
Sat.
Sun.
Speed/travel-time time
series data from a link.
Time series data for each hour of
day (1-24) for each day of week
(Monday – Sunday).
Mean time series computed for
each day of week and hour of day
along with the medoid.
168 LDS models for each link;
Total models learned = 425,712
i.e., (2,534 links × 168 models
per link).
Step 1: Index data for each link for day of
week and hour of day utilizing the traffic
domain knowledge for piece-wise linear
approximation
Step 2: Find the “typical” dynamics by
computing the mean and choosing the
medoid for each hour of day and day of
week
Step 3: Learn LDS parameters for the
medoid for each hour of day (24 hours)
and each day of week (7 days) resulting
in 24 × 7 = 168 models for each link
Learning Context-specific LDS Models

Compute Log Likelihood for
each hour of observed data
(di,hj) LDS(hj,di)
7 × 24
Lik(1,1), Lik(1,2) ,…., Lik(1,24)
Lik(7,1), Lik(7,2) ,…., Lik(7,24)
.
.
.
Train?
Yes (Training phase)
Tag Anomalous hours using the Log
Likelihood Range
No
(di,hj) (min. likelihood)
Anomalies
L =
Partition based on (di,hj)
Speed and travel-time time
Observations from a link
Log likelihood min. and
max. values obtained from
five number summary
Partition based on (di,hj)
7 × 24
LDS(1,1), LDS(1,2) ,…., LDS(1,24)
LDS(7,1), LDS(7,2) ,…., LDS(7,24)
.
.
.
di
hj
(Input)
(Output)
Tagging Anomalies with LDS Models

Hourly Traffic Dynamics Over a Day

Most of the drivers tend to go
5 km/h over the posted speed limit.
There are relatively few drivers who go more
than 10 km/h over the posted speed limit.
There are situations in a day where the drivers are going
(forced) below the speed limit e.g., rush hour traffic.
Do these histograms resemble any probability distribution?
Traffic Data: Possible Explanation

Public Safety
Urban Planning
Gov. & Agency
Admin.
Energy & water
Environmental
TransportationSocial Programs
Healthcare
Education
Twitter as a Source of City Events

Pramod Anantharam, Payam Barnaghi, Krishnaprasad Thirunarayan, and Amit Sheth. 2015. Extracting City Traffic Events from Social Streams.
ACM Trans. Intell. Syst. Technol. 6, 4, Article 43 (July 2015), 27 pages. DOI=10.1145/2717317 http://doi.acm.org/10.1145/2717317
Last O night O in O CA... O (@ O Half B-LOCATION Moon I-LOCATION Bay B-LOCATION
Brewing I-LOCATION Company O w/ O 8 O others) O http://t.co/w0eGEJjApY O
Extracting City Events from Textual Data

Complementary Events
Traffic Incident; road-construction
Textual Events from Tweets vs. 511.org: Complementary

Corroborative Events
Fog visibility-air-quality; fog
Textual Events from Tweets vs. 511.org: Corroborative

Timeliness
Concert Concert
Textual Events from Tweets vs. 511.org: Timeliness

Image Credit: http://traffic.511.org/index
Overturned Truck
Domain knowledge in the
form of traffic vocabulary
Domain knowledge of traffic flow
synthesized from sensor data
Explained-by
Horizontal operator: relating/mapping data from different
modality to a concept (theme) within a spatio-temporal
context;
Spatial context even include what it means to have a slow
traffic for the type of road
Understanding: Semantic Annotation of Sensor + Textual Data
Utilizing Background Knowledge

This example demonstrates use of:
– Multimodal data streams (types of events from text - signature from sensor data).
– Multiple sources of knowledge/ontologies.
– Semantic annotations and enrichments.
– Use of rich representation (PGM).
– Statistical approach to create normalcy models and understand anomalies using
historical data.
– Explain anomalies using extracted events.
– Provide actionable information.
How traffic analysis captures complexity of the real-world?

Thank you, and please visit us at http://knoesis.org
For more information on kHealth, please visit us at
http://knoesis.org/projects/khealth
Cognitive
Computing
Semantic
Computing
Perceptual
Computing
Contributors and collaborators for this talk:
Pramod
Anantharam
Cory Henson Dr. T.K. Prasad
Sujan Perera Utkarshani Jaimini
Thank You
Sanjaya Wijeratne

Part of my group @ Kno.e.sis
with some faculty colleagues

Ohio Center of Excellence in
Knowledge-enabled Computing
Wright State University

Semantic, Cognitive, and Perceptual Computing – three intertwined strands of a golden braid of intelligent computing

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Semantic, Cognitive, and Perceptual Computing – three intertwined strands of a golden braid of intelligent computing

Semelhante a Semantic, Cognitive, and Perceptual Computing – three intertwined strands of a golden braid of intelligent computing (20)

Último

Último (20)

Semantic, Cognitive, and Perceptual Computing – three intertwined strands of a golden braid of intelligent computing