O SlideShare utiliza cookies para otimizar a funcionalidade e o desempenho do site, assim como para apresentar publicidade mais relevante aos nossos usuários. Se você continuar a navegar o site, você aceita o uso de cookies. Leia nosso Contrato do Usuário e nossa Política de Privacidade.
O SlideShare utiliza cookies para otimizar a funcionalidade e o desempenho do site, assim como para apresentar publicidade mais relevante aos nossos usuários. Se você continuar a utilizar o site, você aceita o uso de cookies. Leia nossa Política de Privacidade e nosso Contrato do Usuário para obter mais detalhes.
Peer judge: Praise and Criticism Detection in F1000Research reviews
PeerJudge: Praise and
Criticism Detection in
University of Wolverhampton
• Based on a dictionary of review sentiment terms
and phrases from F1000Research reviews
• Each dictionary term or phrase has a praise or
• Well written: +2
• Flawed: -4
• Reviews given the maximum positive and negative
scores of words or phrases found in each sentence.
• -1: no criticism … -5: very strong criticism
• 1: no praise …. 5: very strong praise
• Also 12 linguistic rules to cope with negation,
booster words (very, slightly)
• The paper is well written but the study is poorly
• Praise: 2; Criticism: -4.
• Try: online
Part of the
• Java jar program
• Dictionaries are external plain text files
• Easily customizable
• 14,000 reviews per second
• Explains its judgement
• So is transparent and the owner can adjust the dictionary for
• Agrees above random chance with reviewer scores
• Because based on a dictionary, does not “cheat” by
identifying hot topics, fields, affiliations or jargon
Where is the dictionary from?
• Human evaluation of a development dataset of
• Machine learning to suggest extra terms and
• Designed for F1000Research decisions – needs
dictionary modification for good performance on other
• F1000Research reviews are unbalanced – few negative
• F1000Research reviews have standard concluding text
that had to be removed – so referees might not
• Referees often give judgements in field-specialist
languages, avoiding general conclusions
• More substantial modifications may be needed for
• Difficult to do this in advance because very few outlets
publish reviews and scores
• Warning reviewers if their judgements are
apparently out of line with their scores?
• Warning reviewers if they have not given any
• As above for editors
• On a larger scale, allow publishers to check for
anomalies in the reviewer process, such as by
identifying journals with uncritical referees (low
average criticism scores).