Yansong Feng from Peking University presented “Towards Building a Cognitive System to Fight for National College Admission Challenge” as part of the Cognitive Systems Institute Speaker Series.
“Towards Building a Cognitive System to Fight for National College Admission Challenge”
1. Towards Building a Cognitive System to
Fight for National College Admission Challenge
Yansong Feng
Joint work with
Kun Xu, Songfang Huang, Dongyan Zhao
Peking University
IBM China Research Lab
December 1, 2016
Feng et al. (PKU) Question Answering December 1, 2016 1 / 25
2. Pass the Exam: A New AI Challenge
The Todai Robot Project
National Institute of Informatics and
collaborators
National Center Test for University
Admissions ( 2016)
Entrance Exam of University of
Tokyo (2021)
Feng et al. (PKU) Question Answering December 1, 2016 2 / 25
3. Pass the Exam: A New AI Challenge
The Todai Robot Project
National Institute of Informatics and
collaborators
National Center Test for University
Admissions ( 2016)
Entrance Exam of University of
Tokyo (2021)
Japanese, Social Science, Math,
Physics
Feng et al. (PKU) Question Answering December 1, 2016 2 / 25
4. Pass the Exam: A New AI Challenge
The Todai Robot Project
National Institute of Informatics and
collaborators
National Center Test for University
Admissions ( 2016)
Entrance Exam of University of
Tokyo (2021)
Japanese, Social Science, Math,
Physics
The Project Aristo and Euclid
The Allen Institute for Artificial
Intelligence
Elementary School:
High School:
Feng et al. (PKU) Question Answering December 1, 2016 2 / 25
5. Pass the Exam: A New AI Challenge
The Todai Robot Project
National Institute of Informatics and
collaborators
National Center Test for University
Admissions ( 2016)
Entrance Exam of University of
Tokyo (2021)
Japanese, Social Science, Math,
Physics
The Project Aristo and Euclid
The Allen Institute for Artificial
Intelligence
Elementary School: Science and
Math
High School: Geometry
Feng et al. (PKU) Question Answering December 1, 2016 2 / 25
6. The GaoKao Challenge
Gaokao in China
National College Entrance Examination
Chinese, Math, English, History, Geography, Politics, Physics,
Chemistry, Biology
over 9,400,000 students in 2016
Feng et al. (PKU) Question Answering December 1, 2016 3 / 25
7. The GaoKao Challenge
Gaokao in China
National College Entrance Examination
Chinese, Math, English, History, Geography, Politics, Physics,
Chemistry, Biology
over 9,400,000 students in 2016
The China Gaokao Challenge
Prompt research in Artificial Intelligence
Team: national research institutes, universities and companies
Real National College Entrance Examinations
Chinese, History, Math, Geography
Feng et al. (PKU) Question Answering December 1, 2016 3 / 25
8. The GaoKao Challenge
Gaokao in China
National College Entrance Examination
Chinese, Math, English, History, Geography, Politics, Physics,
Chemistry, Biology
over 9,400,000 students in 2016
The China Gaokao Challenge
Prompt research in Artificial Intelligence
Team: national research institutes, universities and companies
Real National College Entrance Examinations
Chinese, History, Math, Geography
Feng et al. (PKU) Question Answering December 1, 2016 3 / 25
9. Is that Difficult?
Which is the correct ranking of provinces according to their
average altitutes, from highest to lowest?
Xiang, Liao, Ning
Tai, Lu, Su
Qing, Yue, Jin
Gui, Gan, Yu
Feng et al. (PKU) Question Answering December 1, 2016 4 / 25
10. Is that Difficult?
Which is the correct ranking of provinces according to their
average altitutes, from highest to lowest?
Xiang, Liao, Ning → Hunan, Liaoning, Ningxia
Tai, Lu, Su →Taiwan, Shandong, Jiangsu
Qing, Yue, Jin →Qinghua, Guangdong, Shanxi
Gui, Gan, Yu → Guangxi, Gansu, Henan
1 Knowledge:
short names of provinces
average altitudes of provinces
Feng et al. (PKU) Question Answering December 1, 2016 4 / 25
11. Is that Difficult?
Which is the correct ranking of provinces according to their
average altitutes, from highest to lowest?
Xiang, Liao, Ning → Hunan, Liaoning, Ningxia
Tai, Lu, Su →Taiwan, Shandong, Jiangsu
Qing, Yue, Jin →Qinghua, Guangdong, Shanxi
Gui, Gan, Yu → Guangxi, Gansu, Henan
1 Knowledge:
short names of provinces
average altitudes of provinces
2 Reasoning
relative comparisons of provinces’ altitude
ranking
Feng et al. (PKU) Question Answering December 1, 2016 4 / 25
12. Is that Difficult?
Which is the correct ranking of provinces according to their
average altitutes, from highest to lowest?
Xiang, Liao, Ning → Hunan, Liaoning, Ningxia
Tai, Lu, Su →Taiwan, Shandong, Jiangsu
Qing, Yue, Jin →Qinghua, Guangdong, Shanxi
Gui, Gan, Yu → Guangxi, Gansu, Henan
1 Knowledge:
short names of provinces
average altitudes of provinces
2 Reasoning
relative comparisons of provinces’ altitude
ranking
Not very challenging?
Feng et al. (PKU) Question Answering December 1, 2016 4 / 25
13. What about this one?
Missouri River Valley is an important agricultural area of the
United States. Aerial figure 1 shows the winter of the Missouri
River, where the white part is snow.
1. Why is the farmland on the peninsula shaped as circular?
their watering approach
rugged terrain
their farming approach
lack of farmland
Feng et al. (PKU) Question Answering December 1, 2016 5 / 25
14. What about this one?
2. Could you have a guess what is the main crop in this area?
Winter wheat
Corn
Rice
Potato
Feng et al. (PKU) Question Answering December 1, 2016 6 / 25
15. What about this one?
2. Could you have a guess what is the main crop in this area?
Winter wheat
Corn
Rice
Potato
1 Knowledge:
agriculture
climate
longitude and latitude
read maps and images
Feng et al. (PKU) Question Answering December 1, 2016 6 / 25
16. What about this one?
2. Could you have a guess what is the main crop in this area?
Winter wheat
Corn
Rice
Potato
1 Knowledge:
agriculture
climate
longitude and latitude
read maps and images
2 Reasoning
relationship among those factors
find analogies
common sense knowledge
Feng et al. (PKU) Question Answering December 1, 2016 6 / 25
17. What about this one?
2. Could you have a guess what is the main crop in this area?
Winter wheat
Corn
Rice
Potato
1 Knowledge:
agriculture
climate
longitude and latitude
read maps and images
2 Reasoning
relationship among those factors
find analogies
common sense knowledge
Not very challenging?
Feng et al. (PKU) Question Answering December 1, 2016 6 / 25
18. What We Need?
1 Solid knowledge:
every aspects about the Syllabus
2 Math
3 Reasoning:
logical inference
use common sense knowledge
textual entailment
Feng et al. (PKU) Question Answering December 1, 2016 7 / 25
19. What We Need?
1 Solid knowledge:
every aspects about the Syllabus
2 Math
3 Reasoning:
logical inference
use common sense knowledge
textual entailment
A practical starting point: Answering Factoid Questions with
Knowledge
Feng et al. (PKU) Question Answering December 1, 2016 7 / 25
20. What We Need?
1 Solid knowledge:
every aspects about the Syllabus
2 Math
3 Reasoning:
logical inference
use common sense knowledge
textual entailment
A practical starting point: Answering Factoid Questions with
Knowledge
Information Retrieval Based Question Answering
Answering Factoid Questions with Structured Knowledge Base
Answering Factoid Questions with both Structured and
Unstructured KBs
Feng et al. (PKU) Question Answering December 1, 2016 7 / 25
21. What We Need?
1 Solid knowledge:
every aspects about the Syllabus (Knowledge Bases)
2 Math
3 Reasoning:
logical inference
use common sense knowledge (a little bit)
textual entailment
A practical starting point: Answering Factoid Questions with
Knowledge
Information Retrieval Based Question Answering
Answering Factoid Questions with Structured Knowledge Base
Answering Factoid Questions with both Structured and
Unstructured KBs
Feng et al. (PKU) Question Answering December 1, 2016 7 / 25
22. The Task
What else did the director of the
movie Interstellar direct ?
fb:m.0fkf28 fb:object.type fb:film.film
fb:m.0fkf28 fb:film.film.directed_by ?x
[ ]
select ?y
?x fb:film.director.fim ?y
?y fb:object.type fb:film.film
Inception,
The Dark Knight Rises
The Dark Knight
Batman Begins
…..
Feng et al. (PKU) Question Answering December 1, 2016 8 / 25
23. The Task
What else did the director of the
movie Interstellar direct ?
fb:m.0fkf28 fb:object.type fb:film.film
fb:m.0fkf28 fb:film.film.directed_by ?x
[ ]
select ?y
?x fb:film.director.fim ?y
?y fb:object.type fb:film.film
Inception,
The Dark Knight Rises
The Dark Knight
Batman Begins
…..
Feng et al. (PKU) Question Answering December 1, 2016 8 / 25
24. Question Answering over Structured Knowledge Bases
Goal
Answer Natural Language Questions against Structured Knowledge
Bases
Feng et al. (PKU) Question Answering December 1, 2016 9 / 25
25. Related Work
Information Retrieval Community
Natural Language Processing Community
Semantic Parsing Based
PCCG style: (Zettlemoyer and Collins, 2005, Cai and Yates, 2013;
Kwiatkowski et al. 2013, Reddy et al., 2014)
Syntactic parsing style: (Clarke et al., 2010, Liang et al., 2011,
Berant et al. 2013, Berant and Liang, 2014, Xu et al., 2014)
Information Extraction Based
(Yao and van Durme 2014, Bao et al., 2014, Yih et al., 2015, Dong
et al., 2015)
Deep Learning, End2End style
(Bordes et al., 2014a, 2014b, Yang et al., 2014, Bordes et al.,
2015, Zhang et al., 2016 )
Feng et al. (PKU) Question Answering December 1, 2016 10 / 25
26. Related Work
Information Retrieval Community
Natural Language Processing Community
Semantic Parsing Based
PCCG style: (Zettlemoyer and Collins, 2005, Cai and Yates, 2013;
Kwiatkowski et al. 2013, Reddy et al., 2014)
Syntactic parsing style: (Clarke et al., 2010, Liang et al., 2011,
Berant et al. 2013, Berant and Liang, 2014, Xu et al., 2014)
Information Extraction Based
(Yao and van Durme 2014, Bao et al., 2014, Yih et al., 2015, Dong
et al., 2015)
Deep Learning, End2End style
(Bordes et al., 2014a, 2014b, Yang et al., 2014, Bordes et al.,
2015, Zhang et al., 2016 )
Feng et al. (PKU) Question Answering December 1, 2016 10 / 25
27. Semantic Parsing Based Methods
Challenges:
1 Convert questions into proper meaning representations
2 Ground the meaning representation into a database query
Feng et al. (PKU) Question Answering December 1, 2016 11 / 25
28. Semantic Parsing Based Methods
Challenges:
1 Convert questions into proper meaning representations
2 Ground the meaning representation into a database query
Previously:
1 Search space is huge
2 Difficult to adapt to other KBs
Feng et al. (PKU) Question Answering December 1, 2016 11 / 25
29. Motivation
1 Meaning representation should be KB-independent
[what] did the director of] [the movie] [Interstellar]else movies] [direct]
Feng et al. (PKU) Question Answering December 1, 2016 12 / 25
30. Motivation
1 Meaning representation should be KB-independent
[what] did the director of] [the movie] [Interstellar]else movies] [direct]
2 Separation of meaning representation and instantiation
fb:m.0fkf28 fb:object.type fb:film.film
fb:m.0fkf28 fb:film.film.directed_by ?x
[ ]
select ?y
?x fb:film.director.fim ?y
?y fb:object.type fb:film.film
ns:Interstellar dbo:type dbo:film
dbp:director ?x
[ ]
select ?y
?y
?y
ns:Interstellar
dbp:director ?x
dbo:filmdbo:type
Feng et al. (PKU) Question Answering December 1, 2016 12 / 25
31. Framework
what else movies did the director of the movie Interstellar direct
Feng et al. (PKU) Question Answering December 1, 2016 13 / 25
32. Framework
what else movies did the director of the movie Interstellar direct
Parsing
Feng et al. (PKU) Question Answering December 1, 2016 13 / 25
33. Framework
what else movies did the director of the movie Interstellar direct
Parsing
[what] did the director of] [the movie] [Interstellar]
variable relation category entity
else movies] [direct]
category relation
Feng et al. (PKU) Question Answering December 1, 2016 13 / 25
34. Framework
what else movies did the director of the movie Interstellar direct
Parsing
[what] did the director of] [the movie] [Interstellar]
variable relation category entity
else movies] [direct]
category relation
Instantiation
Feng et al. (PKU) Question Answering December 1, 2016 13 / 25
35. Framework
what else movies did the director of the movie Interstellar direct
Parsing
[what] did the director of] [the movie] [Interstellar]
variable relation category entity
else movies] [direct]
category relation
Instantiation
fb:m.0fkf28 fb:object.type fb:film.film
fb:m.0fkf28 fb:film.film.directed_by ?x
[ ]
select ?y
?x fb:film.director.fim ?y
?y fb:object.type fb:film.film
Feng et al. (PKU) Question Answering December 1, 2016 13 / 25
36. Phrase Dependency Graph
[what] did the director of] [the movie] [Interstellar]
variable relation category entity
else movies] [direct]
category relation
Node
A phrase with a semantic label l ∈ {entity, category, variable, relation}
Edge
A predicate-argument dependency between phrases
unary predicate
binary predicate
Feng et al. (PKU) Question Answering December 1, 2016 14 / 25
37. Structure Prediction
Input: a natural language question
Output: a phrase dependency graph
A pipeline framework to predict the structure
1 Phrase Detection
what did the director of the movie Interstellar
variable relation category entity
else movies direct
category relation
2 Phrase Dependency Parsing
[what] did the director of] [the movie] [Interstellar]
variable relation category entity
else movies] [direct]
category relation
Feng et al. (PKU) Question Answering December 1, 2016 15 / 25
38. Instantiation
[what] did the director of] [the movie] [Interstellar]
variable relation category entity
else movies] [direct]
category relation
1 Converting Phrase Dependency Graph into Structured Queries
2 Instantiating Structured Query against KB
Feng et al. (PKU) Question Answering December 1, 2016 16 / 25
39. Applying Rules
[what] did the director of] [the movie] [Interstellar]
variable relation category entity
else movies] [direct]
category relation
variable category variablecategory
rule#1 rule#1 variablerelationrelationentity
rule#8
?y type movies type ?x
?x ?y
moviesInterstellar Interstellar
direct
the director of
[ ]
select ?y
type moviesInterstellar
?xInterstellar the director of
?x ?ydirect
?y
type movies
Feng et al. (PKU) Question Answering December 1, 2016 17 / 25
40. Probabilistic Model
[ ]
select ?y
type moviesInterstellar
?xInterstellar the director of
?x ?ydirect
?y
type movies
Qind
Qd
ns:Interstellar dbo:type dbo:film
dbp:director ?x
[ ]
select ?y
?y
?y
ns:Interstellar
dbp:director ?x
dbo:filmdbo:type
Q∗
d = arg max P(Qd |Qind )
P(Qd |Qind ) =
n
i=1
P(sdi
|sindi
)P(odi
|oindi
)P(pdi
|pindi
)
Feng et al. (PKU) Question Answering December 1, 2016 18 / 25
41. P(sdi
|sindi
)P(odi
|oindi
)
Freebase Search API ⇒ wikipedia ID ⇒ DBpedia Entity
P(pd |pind )
We construct the co-occurrence matrix from the patty relation
phrase dataset which includes 1,631,530 relation phrases
Feng et al. (PKU) Question Answering December 1, 2016 19 / 25
42. Results on QALDs
Question Answering over Linked Data
Processed Right Partial Recall Precision F-1
2014 40 34 6 0.71 0.72 0.72
2015 42 26 7 0.72 0.74 0.73
Feng et al. (PKU) Question Answering December 1, 2016 20 / 25
43. Results on QALDs
Question Answering over Linked Data
Processed Right Partial Recall Precision F-1
2014 40 34 6 0.71 0.72 0.72
2015 42 26 7 0.72 0.74 0.73
First Place in CLEF QALD 4 and 5
Feng et al. (PKU) Question Answering December 1, 2016 20 / 25
44. Results on QALDs
Question Answering over Linked Data
Processed Right Partial Recall Precision F-1
2014 40 34 6 0.71 0.72 0.72
2015 42 26 7 0.72 0.74 0.73
Nice for longer/complex sentences
Efficient: around 0.33 sec per sentence
Feng et al. (PKU) Question Answering December 1, 2016 20 / 25
45. Results on QALDs
Question Answering over Linked Data
Processed Right Partial Recall Precision F-1
2014 40 34 6 0.71 0.72 0.72
2015 42 26 7 0.72 0.74 0.73
Nice for longer/complex sentences
Efficient: around 0.33 sec per sentence
Consistent performances
0.76 of F-1 on Free917
0.41 of F-1 on WebQuestions
Feng et al. (PKU) Question Answering December 1, 2016 20 / 25
46. Further Improvement
Semantic interpretation for superlatives
Nile is the longest river in the world.
Keys:
the target: Nile
the comparison set: all rivers in the world
the comparison dimension: the length of a river
→/geography/river/length
the ranking order: descending
Feng et al. (PKU) Question Answering December 1, 2016 21 / 25
47. A Little More Extraction
For simple sentences:
Who does michael keaton play in cars
Who
michael keaton cars
Michael Keaton
The Merry Gentleman
Penthouse North
cvt1
Cars
cvt2
Chick Hicks
film
starring
starring
filmdirect direct by
ctv3
spouse_s
Caroline McWilliams
spousespouse_s
spouse
6/5/1982 from
1/29/1990
to
Marriage
type of union
character
portrayer
film
character
starring
charactercharacter
Question:
Star Graph:
Freebase Graph:
cvt3
film
actor
Leona Elizabeth Loftus George A. Douglas
parents
child
Feng et al. (PKU) Question Answering December 1, 2016 22 / 25
48. A Little More Extraction
For simple sentences:
Join Entitly Linking and Relation Extraction
gives 0.49 (+0.06) F1 on WebQuestions.
Feng et al. (PKU) Question Answering December 1, 2016 22 / 25
49. With Hybrid Knowledge Base Resources
a little bit complex...
Where should a visitor see in Germany ?
Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
50. With Hybrid Knowledge Base Resources
a little bit complex...
Where should a visitor see in Germany ?
What is the most popular crop during 1900s in USA ?
Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
51. With Hybrid Knowledge Base Resources
a little bit complex...
Where should a visitor see in Germany ?
What is the most popular crop during 1900s in USA ?
Who did Shaq first play for ?
Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
52. With Hybrid Knowledge Base Resources
Either subjective, or hard to map against existing Structured KBs
Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
53. With Hybrid Knowledge Base Resources
Either subjective, or hard to map against existing Structured KBs
Using both structured knowledge bases and texts, e.g.,
Wikipedia or existing community QA archives.
Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
54. With Hybrid Knowledge Base Resources
who did shaq first play for
KB-QA
Entity Linking Relation Extraction
Joint Inference
shaq: m.012xdf
shaq: m.05n7bp
shaq: m.06_ttvh
sports.pro_athlete.teams..sports.sports_team_roster.team
basketball.player.statistics..basketball.player_stats.team
……
m.012xdf sports.pro_athlete.teams..sports.sports_team_roster.team
Los Angeles Lakers,
Boston Celtics,
Orlando Magic,
Miami Heat
Freebase
Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
55. With Hybrid Knowledge Base Resources
Answer Refinement
Los Angeles Lakers,
Boston Celtics,
Orlando Magic,
Miami Heat
Freebase
Shaquille O'Neal
O'Neal signed
as a free agent with the Los Angeles Lakers
Shaquille O'Neal
O'Neal played for
the Boston Celtics in the 2010-11 season before
retiring
Shaquille O'Neal
O'Neal was drafted
in the 1992 NBA draft
by the Orlando Magic with the first overall pick
Los Angeles Lakers Boston Celtics Orlando Magic
O’Neal was drafted by the Orlando
Magic with the first overall pick in
the 1992 NBA draft
O’Neal played for the Boston Celtics
in the 2010-11 season before retiring
O’Neal signed as a free agent
with the Los Angeles Lakers
Refinement Model
+- -
Orlando Magic
Wikipedia Dump
Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
56. With Hybrid Knowledge Base Resources
Textual
Relation Extraction
Triple SolverTriple Solver
who is the front man of the band that wrote Coffee & TV
Question Decomposition
< ans, is the front man of, var1 >
< var1 , is a , band >
< var1 , wrote , Coffee & TV >
< var1 , wrote, Coffee & TV >
Triple Solver
Entity Linking
Multi-Channel Neural Network
Paraphrase Model
Wikipedia Dump
Textual
KB
KB-based
Relation Extraction
DBpedia
Freebase
DBpedia Lookup
Coffee & TV
Bitter_Coffee_(Iranian_video
_series)
Irish_Coffee_(TV_series)
Coffee & TV
influencedBy
associatedMusicalArtist
associatedBand
writer
front man of
is written by
lead vocalist of
Open
Information Extractor
wrote
is the front man of
Joint Inference
Damon Albarn
Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
57. Conclusion
At this point...
still a lot to do for GaoKao
but, a flexible QA framework
With multiple resources, e.g., structured knowledge bases,
Wikipedia, text books, exercises, even news papers, etc.
Our collaborations with IBM China Research Lab
The Keys
From natural languages to knowledge bases
Inference over structured knowledge
Answer with common-sense knowledge
Understand various images, tables, figures, diagrams...
Feng et al. (PKU) Question Answering December 1, 2016 24 / 25
58. Conclusion
At this point...
still a lot to do for GaoKao
but, a flexible QA framework
With multiple resources, e.g., structured knowledge bases,
Wikipedia, text books, exercises, even news papers, etc.
Our collaborations with IBM China Research Lab
contribute to the Watson Competitions
contribute to a Multi-Modal QA system (with Vision China)
The Keys
From natural languages to knowledge bases → on the way
Inference over structured knowledge
Answer with common-sense knowledge
Understand various images, tables, figures, diagrams...
Feng et al. (PKU) Question Answering December 1, 2016 24 / 25
59. Conclusion
At this point...
still a lot to do for GaoKao
but, a flexible QA framework
With multiple resources, e.g., structured knowledge bases,
Wikipedia, text books, exercises, even news papers, etc.
Our collaborations with IBM China Research Lab
contribute to the Watson Competitions
contribute to a Multi-Modal QA system (with Vision China)
The Keys
From natural languages to knowledge bases → on the way
Inference over structured knowledge → challenging
Answer with common-sense knowledge
Understand various images, tables, figures, diagrams...
Feng et al. (PKU) Question Answering December 1, 2016 24 / 25
60. Conclusion
At this point...
still a lot to do for GaoKao
but, a flexible QA framework
With multiple resources, e.g., structured knowledge bases,
Wikipedia, text books, exercises, even news papers, etc.
Our collaborations with IBM China Research Lab
contribute to the Watson Competitions
contribute to a Multi-Modal QA system (with Vision China)
The Keys
From natural languages to knowledge bases → on the way
Inference over structured knowledge → challenging
Answer with common-sense knowledge → still missing
Understand various images, tables, figures, diagrams... → still
missing
Feng et al. (PKU) Question Answering December 1, 2016 24 / 25