Peer judge: Praise and Criticism Detection in F1000Research reviews
PeerJudge: Praise and
Criticism Detection in
University of Wolverhampton
• Based on a dictionary of review sentiment terms
and phrases from F1000Research reviews
• Each dictionary term or phrase has a praise or
• Well written: +2
• Flawed: -4
• Reviews given the maximum positive and negative
scores of words or phrases found in each sentence.
• -1: no criticism … -5: very strong criticism
• 1: no praise …. 5: very strong praise
• Also 12 linguistic rules to cope with negation,
booster words (very, slightly)
• The paper is well written but the study is poorly
• Praise: 2; Criticism: -4.
• Try: online
Part of the
• Java jar program
• Dictionaries are external plain text files
• Easily customizable
• 14,000 reviews per second
• Explains its judgement
• So is transparent and the owner can adjust the dictionary for
• Agrees above random chance with reviewer scores
• Because based on a dictionary, does not “cheat” by
identifying hot topics, fields, affiliations or jargon
Where is the dictionary from?
• Human evaluation of a development dataset of
• Machine learning to suggest extra terms and
• Designed for F1000Research decisions – needs
dictionary modification for good performance on other
• F1000Research reviews are unbalanced – few negative
• F1000Research reviews have standard concluding text
that had to be removed – so referees might not
• Referees often give judgements in field-specialist
languages, avoiding general conclusions
• More substantial modifications may be needed for
• Difficult to do this in advance because very few outlets
publish reviews and scores
• Warning reviewers if their judgements are
apparently out of line with their scores?
• Warning reviewers if they have not given any
• As above for editors
• On a larger scale, allow publishers to check for
anomalies in the reviewer process, such as by
identifying journals with uncritical referees (low
average criticism scores).
Parece que tem um bloqueador de anúncios ativo. Ao listar o SlideShare no seu bloqueador de anúncios, está a apoiar a nossa comunidade de criadores de conteúdo.
Atualizámos a nossa política de privacidade.
Atualizámos a nossa política de privacidade de modo a estarmos em conformidade com os regulamentos de privacidade em constante mutação a nível mundial e para lhe fornecer uma visão sobre as formas limitadas de utilização dos seus dados.
Pode ler os detalhes abaixo. Ao aceitar, está a concordar com a política de privacidade atualizada.