The second study focuses on a new way to increase user engagement around news articles. We propose to introduce questions for the user to answer, which are related to the viewed article. We argue that in order to serve a diverse set of questions fast to the flow of incoming news articles, it would be beneficial to utilize automatic question generation. Therefore, in this work we study the design of an automatic algorithm that generates high quality question suggestions for a given news article.

1. - 1 -
GENERATING SYNTHETIC
COMPARABLE QUESTIONS
FOR NEWS ARTICLES
Joint work with Idan Szpektor

2. - 2 -
Motivation
• Increase user engagement on content pages
– Recommend additional content or activities
– Traditional engaging content:
• Related articles
• Updates from user’s social neighborhood
• Votes or comments on videos, blogs etc.
• Personalized stories recommended to the user
• Our approach:
– Introduce questions for the user to answer, which are related to
the viewed article
– Focus on comparative questions:
• “Is Beyonce a better singer than Madonna?”
• “Who is better looking, Brad Pitt or George Clooney?”,
• “Who is faster: Superman or Flash?”

3. - 3 -
Why the problem is challengeable?
Question Correct Relevant
X
V
X
X
VV
“Who is faster, Will Smith or David Beckham ?”
“Who is better looking: Will Smith or Angelina
Jolie ?”
“Who is better looking: Will Smith or Brad Pitt ?”

4. - 4 -
Yahoo!
Answers
questions
corpus
Comparable
relation
mining
Comparable
relation
unification
Topic
distribution
Single
entity
model
Entity
pair
model
Synthetic
comparable
questions
News
articles
Comparable
relation
DB
Training models
Named
Entity
extraction
Context
model
inference
Single
entity
filtering
Entity
pair
filtering
Template
instantiation
Relevant
relation
ranking
Context
model
Specific relation

5. - 5 -
Comparable question mining (offline stage)
1. Comparable Relation Extraction
– CRF tagger based
– Who do you think is the greatest Nascar driver ever?
2. Relation Unification
– Many relations are only syntactic variations of the same
underlying semantic relation
– the worst dancer = a better dancer
3. Comparable Template Extraction
– Mine repeated syntactic structure
– E.g. “is <ne1> <rel> than <ne2>?” can be instantiated with
“is Angelina Jolie prettier than Katie Holmes?”
– Keep most frequent ones

6. - 6 -
Online question generation
1. Ranking relevant relations
– Use LDA to infer topic distribution for each relation
– Compute similarity score between the article’s topic
distribution and each relations topic distribution

7. - 7 -
Online question generation
1. Ranking relevant relations
– Use LDA to infer topic distribution for each relation
– Compute similarity score between the article’s topic
distribution and each relations topic distribution
2. Instantiating relevant relations
a. Filter out all entities that do not seem to be valid participants in
the relation, independently of paired entity
b. Scan all possible pairings of the remaining entities and keep
only the pairs that are likely to be matched to each another
under the specific relation
male celebrities should not instantiate
the relation ‘better actress’
‘is faster’ should only be instantiated with athletes,
for whom this comparison is of interest to the reader
filter out a comparison between male and female celebs under ‘is prettier’

8. - 8 -
Evaluation
5000 OMG!
News articles
from 2011
1016 articles
Filter by length
100 articles
Sample
top 3
comparable
relations
Algorithm
3
comparable
questions
Instantiation
of best pair

9. - 9 -
Results
• We compared the performance of our algorithm to two
baselines:
1. Random baseline chooses a relation randomly out of all
possible relations in the database and then instantiate it with
a random pair out of all possible pairing of entities in the
article
2. Relevance baseline chooses the most relevant relation to the
article based on our algorithm, but still instantiates it with a
random pair
Relevance Correctness
Random baseline 29% 43%
Relevance baseline 37% 53%
Full algorithm 54% 77%

10. - 10 -
Example
Ron Livingston is teaming up with Tom
Hanks and HBO again after their
successful 2001 collaboration on Band
of Brothers. The actor has been cast in
HBO’s upcoming film Game Change
that centers on the 2008 presidential
campaign, Deadline reports.
He joins Ed Harris, Julianne Moore and Woody Harrelson. The Jay Roach-
directed movie follows John McCain (Harris) as he selects Alaska Gov. Sarah
Palin (Moore) as his running mate, throughout the campaign and to their
ultimate defeat to Barack Obama. Livingston will play Mark Wallace, one of
the campaign’s senior advisors and the man who prepped Palin for her
debate. Harrelson will play campaign strategist Steve Schmidt …

11. - 11 -
Example (contd.)
Algorithm Question
Random baseline
Who is a better singer, Sarah Palin or Barack
Obama ?
Relevance baseline
Would Ron Livingston be a better president
than Julianne Moore ?
Full algorithm
Who has the best movies, Tom Hanks or
Julianne Moore ?
Is John McCain a better leader than Barack
Obama ?
Would Sarah Palin be a better president than
John McCain ?

12. - 12 -
Good and bad examples
Who is a worse actress , Angelina Jolie or Sarah
Jessica Parker ?
Who is more attractive, Jennifer Aniston or
Angelina Jolie ?

13. - 13 -
Summary
• We introduced the novel task of automatically generating
synthetic comparable questions that are relevant to a
given news article but do not necessarily appear in it
• We assessed the performance of our algorithm via a
Mechanical Turk experiment
• The full algorithm outperformed this baseline by 45% on
question correctness, but surprisingly also by 46% on
question relevance
– These results show that our supervised filtering methods are
successful in keeping only correct pairs, but they also serve as
an additional filtering for relevant relations, on top of context
matching.