Lecture 6: Watson and the Social Web (2014), Chris Welty

© 2011 IBM Corporation
Watson and the Social Web
Chris Welty
IBM Watson Group
ibmwatson.com
Do Not Record. Do Not Distribute.

What is Cognitive Computing?
§  Increasingly, machines are being asked to add their computational
power to problems which are not inherently solvable
§  Traditionally, these problems came from AI
– The hardest AI problems are the easiest for human intelligence:
vision, speech, natural language – these are not actually associated
with “being intelligent”
– Human intelligence provides solutions, but does not scale
§  Cognitive Computing is founded on four principles
Learn & improve. Cognitive computing systems
focus on inexact solutions to unsolvable problems
that utilize machine learning and improve over time.
Often they combine multiple approaches and must
integrate them effectively. They must learn from
humans, in more and more seamless ways.
Speed&Scale. Cognitive computing harnesses the
clear advantage machines have over humans in
their ability to perform mundane tasks of arbitrary
complexity repeatedly, whether it is the scale of the
data or the complexity of the task.
Interact in a natural way. Cognitive computing
provides technologies that support a higher level of
human cognition by adapting to human approaches
and interfaces...over the next several decades it will
incorporate essentially all the ways humans sense
and interact.
Assist & augment human cognition. Cognitive
computing addresses problems that lie squarely in
the province of human intelligence, but where we
can't handle the volume of information, penetrate the
complexity or otherwise extend our reach
(physically).
The goal is to be useful, not universally correct.
or
Computers can be incorrect and still prove useful!

Examples of Cognitive Computing
§ Web Search
§ Image Search
§ Event Search
§ Recommendations
§ Natural Language Processing

What is Watson?
§  Open Domain Question-Answering Machine
§  Given
–  Rich Natural Language Questions
–  Over a Broad Domain of Knowledge
§  Delivers
–  Precise Answers: Determine what is being asked & give precise response
–  Accurate Confidences: Determine likelihood answer is correct
–  Consumable Justifications: Explain why the answer is right
–  Fast Response Time: Precision & Confidence in <3 seconds
–  At the level of human experts
– Proved its mettle in a televised match
–  Won a 2-game Jeopardy match against
the all-time winners
–  viewed by over 50,000,000
4

What is Jeopardy?
§ Jeopardy! is an American quiz
show
– 1964 – Today
– Household name in U.S.
§ answer-and-question format
– contestants are presented with
clues in the form of answers
– must phrase their responses in
question form.
– Open domain trivia questions,
speed is a big factor
§  Example
–  Category: General Science
–  Clue: When hit by electrons, a
phosphor gives off electromagnetic
energy in this form
–  Answer: What is light?

Social Computing: What’s the connection?
§ Social Web as Data Source:
– The vast majority of sources Watson
used to answer questions came from
community-created data
–  Adapting Watson to a new problem
requires the same kind of information
about that problem
§  Social Machines:
–  Watson combined with people is a
powerful proposition
§  Social Web as Application:
–  Watson’s major advance is in
understanding natural language, the
technology can be useful to augment
social interaction

$200
If you are looking at
the wainscoating,
you are looking in
this direction.
$1000
The first person
mentioned by name in
‘The Man in the Iron
Mask’ is this hero of a
previous book by the
same author.
7
The Jeopardy! Challenge
Hard for humans, hard for machines
Broad/Open
Domain
Complex
Language
High
Precision
Accurate
Confidence
High
Speed
$600
In cell division, mitosis
splits the nucleus &
cytokinesis splits this
liquid cushioning the
nucleus
$800
The conspirators against
this man were wounded by
each other while they
stabbed at him
But hard for different reasons.
For people, the challenge is knowing the answer
For machines, the challenge is understanding the
question
What is down?
Who is
D’Artagnan?
What is
cytoplasm?
Who is Julius
Caesar?

The Winner’s Cloud
What It Takes to compete against Top Human Jeopardy! Players
Winning Human
Performance
2007 QA Computer System
Grand Champion
Human Performance
Top human
players are
remarkably
good.
Each dot – actual historical human Jeopardy! games
More Confident Less Confident
Develop
against a
metric!

2007 QA Computer System
In 2007, we committed to
making a Huge Leap!
More Confident Less Confident
Each dot – actual historical human Jeopardy! games
Computers?
Not So Good.
Winning Human
Performance
Grand Champion
Human Performance
The Winner’s Cloud
What It Takes to compete against Top Human Jeopardy! Players

DeepQA: The Technology Behind Watson
An example of a new software paradigm
. . .
Answer
Scoring
Models
Answer &
Confidence
Question
Evidence
Sources
Models
Models
Models
Models
ModelsPrimary
Search
Candidate
Answer
Generation
Hypothesis
Generation
Hypothesis and
Evidence Scoring
Final Confidence
Merging &
Ranking
Synthesis
Answer
Sources
Question &
Topic
Analysis
Question
Decomposition
Evidence
Retrieval
Deep
Evidence
Scoring
Hypothesis
Generation
Hypothesis and Evidence
Scoring
Learned Models
help combine and
weigh the Evidence
DeepQA generates and scores many hypotheses using an extensible collection of
Natural Language Processing, Machine Learning and Reasoning Algorithms.
These gather and weigh evidence over both unstructured and structured content to
determine the answer with the best confidence.
Content from
Community
Resources!

Example Question
In 1894 C.W. Post
created his warm
cereal drink Postum in
this Michigan city
Related Content
(Structured & Unstructured)
Primary
Search
1985
Post Foods
aramour
General Foods
Grand Rapids
…
Battle Creek
…
…
Candidate Answer Generation
1)  Battle Creek (0.85)
2)  Post Foods ( 0.20)
3)  1985 (0.05)
Merging &
Ranking
Evidence
Retrieval
Question
Analysis
Keywords: 1894, C.W. Post,
created …
Lexical AnswerType:
(Michingan city)
Date(1894)
Relations:
Create(Post, cereal drink)
…
[0.58 0 -1.3 … 0.97]
[0.71 1 13.4 … 0.72]
[0.12 0 2.0 … 0.40]
[0.84 1 10.6 … 0.21]
[0.33 0 6.3 … 0.83]
[0.21 1 11.1 … 0.92]
[0.91 0 -8.2 … 0.61]
[0.91 0 -1.7 … 0.60]
Evidence
Scoring
Need thousands of
Q/A pairs for training!

Planet Fitness
Role
of
Answer
Typing
in
QA

Type Information - a crucial hint to get the correct answer
ASTRONOMY:
In
1610
Galileo
named
the
moons
of
this
planet

for
the
Medici
brothers

Telescope
Giovanni Medici
Sidereus
Nuncius
Jupiter
Ganymede
Telescope
(Instrument)
Giovanni Medici
(Person)
Sidereus
Nuncius
(Book)
Jupiter
(Planet)
Ganymede
(Moon)
Terms
Associated
with
Clue
Context

(e.g.
via
Keyword
Search)

Planet Fitness
(Planet)

§  This
fish
was
thought
to
be
exLnct
millions
of
years
ago

unLl
one
was
found
off
South
Africa
in
1938

§  Category:
ENDS
IN
"TH"

§  Answer:

§  When
hit
by
electrons,
a
phosphor
gives
off
electromagneLc

energy
in
this
form

§  Category:
General
Science

§  Answer:

§  Secy.
Chase
just
submiXed
this
to
me
for
the
third
Lme-‐-‐guess

what,
pal.
This
Lme
I'm
accepLng
it

§  Category:
Lincoln
Blogs

§  Answer:

The type of thing
being asked for is
often indicated but
can go from specific
to very vague
coelacanth

light
(or
photons)

his
resigna4on

13

Answer Typing for Jeopardy!?

Broad Domain
Our Focus is on reusable NLP technology for analyzing vast volumes of as-is text.
Structured sources (DBs and KBs) provide background knowledge for interpreting the text.
We do NOT attempt to anticipate all
questions and build databases.
We do NOT try to build a formal
model of the world

Sources for typing evidence
§ DbPedia & Freebase
– Wide coverage of well-known entities
– Taxonomy (MountainsOfNepal → Mountain)
– Good type coverage, but not many synonyms
•  E.g. what about “summit”
§  Wikpedia Categories
–  Wide coverage of entities and type name synonyms
–  Noisy (many errors)
§  Wikipedia Intro
–  First sentence always indicates the most common type of the entity
–  Highly reliable, low coverage of types
Communities can
scale data collection!

Typing
Impact
on
Jeopardy!
clues

61.5%
62.0%
62.5%
63.0%
63.5%
64.0%
64.5%
65.0%
65.5%
66.0%
66.5%
An ensemble of TyCor components
+ ~10%

Many sources of evidence
In 1894 C.W. Post
created his warm
cereal drink Postum in
this Michigan city
Related Content
(Structured & Unstructured)
Primary
Search
1985
Post Foods
aramour
General Foods
Grand Rapids
…
Battle Creek
…
…
Candidate Answer Generation
1)  Battle Creek (0.85)
2)  Post Foods ( 0.20)
3)  1985 (0.05)
Merging &
Ranking
Evidence
Retrieval
Question
Analysis
Keywords: 1894, C.W. Post,
created …
Lexical AnswerType:
(Michingan city)
Date(1894)
Relations:
Create(Post, cereal drink)
…
[0.58 0 -1.3 … 0.97]
[0.71 1 13.4 … 0.72]
[0.12 0 2.0 … 0.40]
[0.84 1 10.6 … 0.21]
[0.33 0 6.3 … 0.83]
[0.21 1 11.1 … 0.92]
[0.91 0 -8.2 … 0.61]
[0.91 0 -1.7 … 0.60]
Evidence
Scoring

Watson as part of a social machine
§  Watson makes mistakes:
– This woman was the first to witness her husband resign from the U.S. Presidency.
– This U.S. City’s largest airport is named for a world-war II hero; its second largest for a
world-war II battle.
§  These mistakes are typically obvious to people
– Even when they don’t know the answer
– Watson isn’t stupid, it solves problems differently
– Often these multiple perspectives can combine productively
•  E.g. add a “dismiss” button to the answer interface
Richard Nixon
Dolly Madison
Pat Nixon
Watson can
adapt and learn
from its users!

Cut to the chase…..
Watson emerges victorious

Technology marches forward…

Adapt Watson
Models
Answer &
Confidence
Question
Evidence
Sources
Models
Models
Models
Models
Models
Answer
Sources
. . .
Answer
Scoring
Primary
Search
Candidate
Answer
Generation
Hypothesis
Generation
Hypothesis and
Evidence Scoring
Final Confidence
Merging &
Ranking
Synthesis
Question &
Topic
Analysis
Question
Decomposition
Evidence
Retrieval
Deep
Evidence
Scoring
Hypothesis
Generation
Hypothesis and Evidence
Scoring
Learned Models
help combine and
weigh the Evidence
What does it take to use Watson in a new domain?
(medical diagnosis, call centers, etc...)
Gathering significant numbers of
question-answer pairs is proving
to be one of the most significant
challenges for adapting Watson.
Can the social web help?
Community
created!

Integrating Watson in Social Interaction?
Did you hear about Bob?
No
He’s taking a year off to climb
the tallest mountain!
The tallest mountain is
Mount Everest.
Wow.
me
me
Jeff
Watson
Jeff

Privacy – a blessing and a curse
Need to protect our data, but…
Crime on the web, the social web, is very real
Identity theft
Credit card, bank, insurance fraud
Terrorist networks
Medical diagnosis
Monitoring your profile for health-related information
ICT for depression
Calendar, appointments, traffic, spreading disease

The arrival of Cognitive Computing
Learn & improve. The core of Watson is a group of
over 100 independent algorithms that approximate a
solution to the “is this the right answer to the question”
problem. Achieving winning (human expert)
performance, required two hallmarks of cognitive
computing systems: a metric to measure improvements
to the system (the winners cloud), and a significant
ground truth (over 200K Q-A pairs).
Speed&Scale. Watson used big data, as well as a
3000 node cluster for massive computation to get
answering speeds down into the 2s range.
Interact in a natural way. Watson was a significant
step forward in natural language understanding, the
most basic interface for humans. Say goodbye to
your mouse…
Assist & augment human cognition. Watson
depended on primarily a set of background
documents (the corpus). The value of having access
to this kind of fact-finding power over a large (and
possibly changing) corpus provides a clear
augmentation to human abilities.

Learn & improve. The core of Watson is a group of
over 100 independent algorithms that approximate a
solution to the “is this the right answer to the question”
problem. Achieving winning (human expert)
performance, required two hallmarks of cognitive
computing systems: a metric to measure improvements
to the system (the winners cloud), and a significant
ground truth (over 200K Q-A pairs).
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
% Answered

Assist & augment human cognition. Watson
depended on primarily a set of background
documents (the corpus). The value of having access
to this kind of fact-finding power over a large (and
possibly changing) corpus provides a clear
augmentation to human abilities.
UTI
Diabetes
Influenza
hypokalemia
Renal failure
esophogitis
Diagnosis
Models
Conﬁdence

Most
Conﬁdent
Diagnosis:
UTI

Symptoms

Tests/Findings

Medica4ons

Family
History

Notes/Hypotheses

Huge
Volumes
of
Texts,

Journals,
References,
DBs

etc.

Pa4ent
History

Speed&Scale. Watson used big data, as well as a
3000 node cluster for massive computation to get
answering speeds down into the 2s range.

Interact in a natural way. Watson was a significant
step forward in natural language understanding, the
most basic interface for humans. Say goodbye to
your mouse…

…and for Social Web
§  First and foremost, social web analytics (e.g. recommendations) and Social
Computing in general lie clearly in the realm of Cognitive Computing
– Uncertainty, natural language, human intelligence
– Inexact solutions that can improve with time, training
– Problems & solutions need metrics to be solvable
§  All cognitive computing systems require ground truth data
– This data is expensive to collect
– Crowdsourcing is a key new technology/approach
§  The user interface moving closer to people
– Natural language, speech, gestures
– In addition, integrating the collection of training data seamlessly into the interface
is a key development
§  Cognitive computing systems require integration of multiple, disparate, data
sources
– Structured, unstructured, semi-structured
– curated, crowdsourced

Lecture 6: Watson and the Social Web (2014), Chris Welty

Recommended

Recommended

More Related Content

Similar to Lecture 6: Watson and the Social Web (2014), Chris Welty

Similar to Lecture 6: Watson and the Social Web (2014), Chris Welty (20)

More from Lora Aroyo

More from Lora Aroyo (20)

Recently uploaded

Recently uploaded (20)

Lecture 6: Watson and the Social Web (2014), Chris Welty