Abstract: This paper addresses the problem of determining the best answer in Community-based Question Answering (CQA) websites by focussing on the content. In particular, we present a system, ACQUA1, that can be installed onto the majority of browsers as a plugin. The service o↵ers a seamless and accurate prediction of the answer to be accepted. Previous research on this topic relies on the exploitation of community feedback on the answers, which involves rating of either users (e.g., reputation) or answers (e.g. scores manually assigned to answers). We propose a new technique that leverages the content/textual features of answers in a novel way. Our approach delivers better results than related linguistics-based solutions and manages to match rating-based approaches. More specifically, the gain in performance is achieved by rendering the values of these features into a discretised form. We also show how our technique manages to deliver equally good results in real-time settings, as opposed to having to rely on information not always readily available, such as user ratings and answer scores. We ran an evaluation on 21 StackExchange websites covering around 4 million questions and more than 8 million answers. We obtain 84% average precision and 70% recall, which shows that our technique is robust, effective, and widely applicable.

1. GEORGE GKOTSIS 1, MARIA LIAKATA 2,
CARLOS PEDRINACI 3, JOHN DOMINGUE 3
Leveraging Textual Features for Best
Answer Prediction in Community-based
Question Answering
1King’s College London
2Department of Computer Science, University of Warwick
3Knowledge Media Institute, The Open University

2. Outline
8-11June 2015ICCSS 2015
 Motivation
 Problem description
 Proposed solution
 Evaluation
 ACQUA

3. 8-11June 2015ICCSS 2015
Motivation

4. Questions on social networking sites
8-11June 2015ICCSS 2015
Recommendations &
opinions
Authoritative
responses
Expert &
Empirical
knowledge

5. Queries on CQA
8-11June 2015ICCSS 2015

6. 8-11June 2015ICCSS 2015
Problem description

7. 8-11June 2015ICCSS 2015

8. Reputation based Answer Rating based
8-11June 2015ICCSS 2015
“…we observe significant
assortativity in the reputations of
co-answerers, relationships
between reputation and answer
speed, and that the probability of
an answer being chosen as the
best one strongly depends on
temporal characteristics of answer
arrivals.”
Ashton Anderson, Daniel Huttenlocher,
Jon Kleinberg, Jure Leskovec
Discovering Value from Community
Activity on Focused Question Answering
Sites: A Case Study of Stack Overflow.
KDD 2012
“When available, scoring (or
rating) features improve
prediction results significantly,
which demonstrates the value of
community feedback and
reputation for identifying valuable
answers.”
Grégoire Burel, Yulan He, Harith Alani.
Automatic Identification of Best Answers
in Online Enquiry Communities
ESWC 2012
State of the art solutions

9. Best answer prediction in Social Q&A
8-11June 2015ICCSS 2015
 Binary classification problem
 Is it solved?
 Yes, partially
 Current solutions depend on:
Answer Ratings
• Score, #comments
Knowledge is Future & Unknown
User Ratings
• User Reputation
• UpVotes etc
• Preferential attachment
Knowledge is Past & Not
always available

10. State of the art solutions
Summary
8-11June 2015ICCSS 2015
Our solution
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Linguistic User Ratings Answer ratings
Average Precision

11. StackExchange network
8-11June 2015ICCSS 2015
SE “is all about getting answers, it’s not a
discussion forum, there’s no chit-chat”
 123 Q&A sites
 5,622,330 users
 9.5 million questions
 16.3 million answers
 9.3 million visits per day
20 June 2014:

12. 8-11June 2015ICCSS 2015
StackOverflow
91%
The Rest
9%
3,375,817
3,795,276
0
1,000,000
2,000,000
3,000,000
4,000,000
5,000,000
6,000,000
7,000,000
8,000,000
stackoverflow
Non Accepted
Answers
Accepted
Answers
September 2013 dump
Questions with Accepted Answers

13. Shallow Linguistic features
8-11June 2015ICCSS 2015
 Long history, coming from studies on readability
1. Average number of characters per word
2. Average number of words per sentence
3. Number of words in the longest sentence
4. Answer length
5. Log Likehood:
Pitler &
Nenkova, 2008

14. StackOverflow
Overview of shallow features’ evolution
8-11June 2015ICCSS 2015

15. Shallow features: Observations
8-11June 2015ICCSS 2015
 Accepted answers tend to be:
 Longer
 Differ more from the community vocabulary
 Contain shorter words
 Have longer longest sentences
 Have more words per sentence
But how good are shallow features?

16. But how good are shallow features?
8-11June 2015ICCSS 2015
 58% macro precision (our baseline)
 Possible reasons
1. Evolution of language characteristics
 Language becomes more eloquent
2. Variance is huge
3. Universal classifier looks unreachable, e.g.:
 SuperUser average length is 577
 Skeptics average length is 2,154
Bad
Good

17. StackOverflow vrs. SuperUser
8-11June 2015ICCSS 2015

18. 8-11June 2015ICCSS 2015
Proposed solution

19. Objectives
8-11June 2015ICCSS 2015
 Build a classifier which is:
1. Based on linguistic features solely
2. Robust
 Performs equally well to other classifiers that use user ratings
(past knowledge) or answer ratings (future knowledge)
3. Universal
 Same classifier applicable to as many SE websites possible
(domain agnostic)

20. Feature discretisation
Example for Length
8-11June 2015ICCSS 2015
Group by question
Question Id
1
5
Answer Id
6
7
Length
2 200
3 150
4 250
150
100
Sort by Length in descending order
Rank
LengthD
1
2
3
1
2

21. Feature discretisation
8-11June 2015ICCSS 2015
Category Name Information Gain
Linguistic
Length 0.0226
LongestSentence 0.0121
LL 0.0053
WordsPerSentence 0.0048
CharactersPerWord 0.0052
Linguistic
Discretisation
LengthD 0.2168
LongestSentenceD 0.1750
LLD 0.1180
WordsPerSentenceD 0.1404
CharactersPerWordD 0.1162
20x increase

22. User and answer rating features
8-11June 2015ICCSS 2015
Category Name
Other
Age
CreationDateD
AnswerCount
User Rating
UserReputation
UserUpVotes
UserDownVotes
UserViews
UserUpDownVotes
Answer
rating
Score
CommentCount
ScoreRatio

23. 8-11June 2015ICCSS 2015
Evaluation

24. Evaluation Comparison
8-11June 2015ICCSS 2015
Case Features Used P R FM AUC
1 Linguistic 0.58 0.60 0.56 0.60
2 Linguistic & Discretisation 0.81 0.70 0.74 0.84
3 Linguistic & Discretisation &
Other
0.84 0.7 0.76 0.87
4 Linguistic & Other & User
Rating
(no discretisation)
0.82 0.69 0.75 0.86
5 Linguistic & Other & User
Rating
(with discretisation)
0.82 0.72 0.77 0.88
6 All features
(Answer and User Rating
with discretisation)
0.88 0.85 0.86 0.94

25. 8-11June 2015ICCSS 2015
ACQUA
Automatic Community-based Question Answering
https://acqua.kmi.open.ac.uk/

26. 8-11June 2015ICCSS 2015
ACQUA - Architecture

27. ACQUA - Screenshot
8-11June 2015ICCSS 2015

28. Read more about our work
8-11June 2015ICCSS 2015
 It’s All in the Content: State of the Art Best
Answer Prediction based on Discretisation of
Shallow Linguistic Features. WebSci ’14
 ACQUA: Automated Community-based
Question Answering through the
Discretisation of Shallow Linguistic Features.
The Journal of Web Science, 1(1) (preprint available)

29. Thank you
8-11June 2015ICCSS 2015
http://xkcd.com/386/