This is a presentation of the paper in which we focus on the analysis of peer reviews and reviewers behavior in conference review processes. We report on the development, definition and rationale of a theoretical model for peer review processes to support the identification of appropriate metrics to assess the processes main properties. We then apply the proposed model and analysis framework to data sets about reviews of conference papers. We discuss in details results, implications and their eventual use toward improving the analyzed peer review processes.
Is peer review any good? A quantitative analysis of peer review
1. Is peer review any good? A quan4ta4ve
analysis of peer review
Fabio Casa), Maurizio Marchese, Azzurra Ragone, Ma6eo Turrini
University of Trento
h6p://eprints.biblio.unitn.it/archive/00001654/01/techRep045.pdf
2. Ini)al Goals
• Understand how well
peer review works
• Understand how to
improve the process
• Metrics + Analysis
– (refer to liquid doc)
• Focus only on
gatekeeping aspect “Not everything that can be counted counts,
and not everything that counts can
be counted.” -- Albert Einstein
12. Quality‐related Metrics: Sta)s)cs
Distribu4on of marks (integer marks)
0.18
0.16
0.14
0.12
Probability
0.1
0.08
0.06
0.04
0.02
0
0 1 2 3 4 5 6 7 8 9 10
Marks
12
13. Disagreement
• Measure the difference between the marks
given by the reviewers on the same
contribu4on.
• The ra)onale behind this metric is that in a
review process we expect some kind of
agreement between reviewers.
15. Robustness
• Sensi)vity to small varia)on in the marks
– Tries to assess the impact of small indecisions in
giving the mark (e.g., 6 vs 7…..)
• Measures divergence afer applying an ε‐
varia)on to the mark
• Results: reasonably robust except for the
conference managed by young researchers
15
17. Fairness
• Defini)on: A review process is fair if and only
of the acceptance of a contribu)on does not
depend on the par)cular set of PC members
that reviews it
• The key is in the assignment of a paper to
reviewers: A paper assignment is unfair if the
specific assignment influences (makes more
predictable) the fate of the paper.