4. • The importance of understanding the peer
review (PR) process
In general
Avoid silly mistakes
Adjust the level of expectancy of your submission
To not be upset with some reviews
Targeted submissions
Know how your paper will be treated
UNDERSTANDING THE PEER REVIEW
PROCESS
5. Peer review is a very old process
Ishaq al-Rahwi (854-931) of Syria first described the PR process
The Philosophical Transactions of the Royal Society is the first journal
to formalize the peer review process (300 years ago)
Validation by peers and publication in a scientific journal
the method through which authors register, validate, disseminate, and
archive their discoveries and results.
process and the speed at which articles are peer reviewed and
published are key elements in the appropriate accreditation of
scientific findings.
Recent Survey
85% of scientists agree that PR greatly helps scientific
communication
93% of them believe peer review is necessary.
6. Single Blind Review
The names of the reviewers are hidden from the author.
Traditional method of reviewing (the most common type)
Advantage
Reviewer anonymity allows for impartial decisions free from influence by the
authors
Disadvantages:
Authors fear the risk that reviewers working in the same field may
withhold submission of the review in order to delay publication,
thereby giving the reviewer the opportunity to publish first
Reviewers may use their anonymity as justification for being
unnecessarily critical or harsh when commenting on the author’s
work.
Prejudice. Yes, it happens all the time!
TYPES OF PEER REVIEW (1/3)
8. Double Blind Review
Both the reviewer and the author remain anonymous
Advantages:
Author anonymity prevents any reviewer bias based on, for
example, an author’s country of origin or previous controversial
work.
Articles written by ‘prestigious’ or renowned authors are
considered on the basis of the content of their papers, rather than
on the author’s reputation.
Disadvantage:
It is uncertain whether a paper can ever truly be ‘blind’ –
especially in specialty ‘niche’ areas. Reviewers can often identify
the author through the paper’s style, subject matter or self-
citation.
TYPES OF PEER REVIEW (2/3)
10. Open Review
Reviewer and author are known to each other
Advantage
best way to prevent malicious comments, stop plagiarism,
prevent reviewers from drawing upon their own ‘agenda’ and
encourage open, honest reviewing.
Disadvantage
the opposite view:
a less honest process in which politeness or fear of retribution may
cause a reviewer to withhold or tone down criticism.
Junior reviewers may hesitate to criticize more esteemed authors for
fear of damaging their prospects
TYPES OF PEER REVIEW (3/3)
12. Again: Understand the Purpose of Peer Review
Peer review is a critical element of scholarly publication, and one of the
major cornerstones of the scientific process.
Peer Review serves two key functions
Acts as a filter: Ensures research is properly verified before being published
Improves the quality of the research: rigorous review by other experts helps to
improve key points and correct inadvertent errors
On Being Asked To Review
Does the article you are being asked to review truly match your
expertise?
Only accept an invitation if you are competent to review the article
Do you have time to review the paper?
an article will take about 5 hours to review properly
Are there any potential conflicts of interest?
A conflict of interest will not necessarily eliminate you from reviewing an
article
REVIEWER GUIDELINES – FIRST STEPS
17. Referee
2007 – 2007 - IEEE Transactions on Multimedia
2007 – 2007 - Journal of Internet Engineering
2009 – Atual - Computer Communications
2009 – Atual - Peer-to-Peer Networking and
Applications Journal
2009 – Atual - Journal of Network and Systems
Management
2009 – Atual - Journal of Network and Computer
Applications
2008 – Atual - Ad Hoc Networks
2010 – Atual - Journal of The Brazilian Computer
Society
2011 – Atual - Wireless Communications and
Mobile Computing
2012 – Atual - Wireless Networks
2009 – Atual - Computer Networks
2012 – Atual - Journal of Communications and
Networks
Editorial Board
2012 – Atual - International Journal on Advances
in Intelligent Systems
2012 - Atual - Journal of Mobile and Ubiquitous
Computing Technology
Referee
IEEE ICC, GLOBECOM,
INFOCOM, WCNC, GHTCE,
ICNS, CCNC, HPSR, BCN
TPC Member
IEEE WCNC, CCNC,
INFOCOM TMA WORKSHOP
CHAIR
IEEE TRICANS, ACM HPN
CONNEPI 2010 (interesting
stuff, long history)
GENERAL GUIDELINES FROM
EDITORS AND TPC CHAIRS
18. the article should not be disclosed to a third party
you would be expected to evaluate the article according to the
following
Originality
Is the article sufficiently novel and interesting to warrant publication?
Is the research question an important one?
Structure
Are all the key elements present: abstract, introduction, methodology, results,
related work, discussion (lessons learned), conclusions?
Title: Does it clearly describe the article?
Abstract: Does it reflect the content of the article?
Introduction: Does it describe what the author hoped to achieve accurately,
and clearly state the problem being investigated?
Method: Is there sufficient information present for you to replicate the
research?
Results: has appropriate analysis been conducted? Are the statistics correct?
Conclusion/Discussion: Are the claims in this section supported by the results,
do they seem reasonable?
CONDUCTING THE REVIEW
19. Commentary should be courteous and constructive
You should indicate whether your comments are your own
opinion or are reflected by the data
If the review is not blinded (yes, it’s true!)
Do the authors have a "track record" of working in this area?
Are they from a reputable institution?
Think about the consequences of this? (my question)
Subjectively:
do you believe what the authors are telling you?
do you suspect some consistent error in the hypothesis, methods,
analysis of data, etc.?
Is there some chance that there is scientific fraud or plagiarism
involved in this manuscript?
CONDUCTING THE REVIEW
20. Finally, on balance, when considering the whole article, do the
figures and tables inform the reader, are they an important
part of the story?
Previous Research:
If the article builds upon previous research does it reference that
work appropriately?
Are there any important works that have been omitted?
Are the references accurate?
CONDUCTING THE REVIEW
21. RELEVANCE
Is the topic of the paper relevant to the journal?
Is the paper aimed at the readership?
Is the technical depth of the paper relevant to the journal?
CONTENT
Is the paper needed (does it fill a gap)?
Are the references up to date?
Is the content (theories, concepts, methodologies) sound?
Does the originality of the material warrant publication?
PRESENTATION
Is the paper well organised?
Do any parts require further explanation of clarification?
REVIEWER CONFIDENTIAL COMMENTS TO
EDITOR - ELSEVIER
22. Section I. Overview
A. Reader Interest
1. How relevant is this manuscript to the readers of this journal?
B. Content
Is the manuscript technically sound?
What do you see as this manuscript's contribution to the literature in
this field?
What do you see as the strongest aspect of this manuscript?
What do you see as the weakest aspect of this manuscript?
C. Presentation
Are title, abstract, and/or keywords satisfactory?
REVIEWER CONFIDENTIAL COMMENTS TO
EDITOR - SPRINGER
23. Please identify and discuss the contribution of this manuscript.
Please include in your discussion:
Does the paper have significant tutorial content?
Is there enough background provided so that the generalist in
communications can understand its main contributions? Elaborate.
Does the paper contain original contributions? What is the nature
of the contributions?
Is there a description of lessons learned that are given to the
reader to help the reader avoid pitfalls in his own work?
Is there a need for a paper such as this in the communications
community?
For example, are there articles that are already available which cover
more or less the same topic at about the same depth?
Please discuss the quality of the citations in this manuscript
REVIEWER CONFIDENTIAL COMMENTS TO
EDITOR - IEEE
24. Please comment on the organization of the paper, and offer any
suggestions that you think will improve the paper and its
readability
Please comment on the technical correctness of the manuscript
in general
identify any specific technical inaccuracies that you find, and make
suggestions for correcting those
If the manuscript does not require major revision, please provide
a list of minor changes
Please provide a summary comment on the overall suitability of
the paper for publication in IEEE *
REVIEWER CONFIDENTIAL COMMENTS TO
EDITOR - IEEE
27. I now realize that the community generates a large amount of
“junk”
I don't mean bad ideas
Each paper I reviewed for SIGCOMM had at least a nugget of an interesting
idea.
Definition of Junk: "it took me twice as long to read this as it should
have"
In other words, the paper was sloppy
I had a very difficult time trying to figure out what the paper was
proposing
What the experiments were really showing.
The following is a short list of *easy* ways to improve your
research papers
Most comments are general
Last comment is directed at the internetworking research community
0 - LARGE AMOUNT OF “JUNK”
28. It seems to me that everyone has access to a good spell
checker these days.
Not using the spell checker just indicates that either you are
sloppy or you do not care.
If you're sloppy with your writing why should I believe you are
not sloppy with your experiments/analysis as well?
If you don't care about your paper, why should I?
1 - FIRE UP THE SPELL CHECKER,
PLEASE!
29. Proofread your paper
I see lots of mistakes that indicate the paper was not looked over
before it was submitted (e.g., "The foo algorithm works better
than the the bar algorithm.").
Again, if you do not take enough care in presenting your work,
why should I be compelled to recommend accepting it?
Sloppy papers give the indication of sloppy research
As well as reflecting poorly on your work, such papers are often very
difficult to read and understand
The point of writing a paper is to convey new information to the
world
Therefore, there is major value in clear writing.
If a referee can't get the point of the paper, or has a very difficult time
getting the main idea, there is a high likelihood that the paper will be
rejected
2 - SLOPPY PAPERS ARE LONG SHOTS
30. I print out all papers I review. I read them in my lazy boy. I
scribble notes on them.
Do not assume that I am looking at color plots!
Color plots come out of my black and white printer very badly in most
cases.
Besides, most conferences/journals publish black and white papers
Make it easy to tell the difference between the lines on your plots.
Do not plot on a grey background.
Make the font on the plots readable without a magnifying glass
Help me out! Please! Often the plots are key to understanding the
paper
If I cannot understand the paper I will recommend rejecting it
3 - I READ SUBMISSIONS IN MY LAZYBOY
31. You do not need to give an in-depth tutorial of all
background material.
How much background material to include?
It depends on the conference to which you are submitting,
the maturity of the background work, etc.
There are some topics (e.g., TCP congestion control)
that are well understood by people attending any
networking conference.
You do not need to describe the algorithms in painful
detail.
A short paragraph re-capping the main ideas and
providing references to the original works should
suffice.
4 - DON'T WASTE MY TIME (I)
32. You do not need to provide me with an example of every
kind of plotting strategy you use
When you first show a plot with notation that I might
not understand, explain the notation
But, using half a page to show me a plot that adds nothing to
my understanding of the topic is a less than compelling use
of real estate
5 - DON'T WASTE MY TIME (II)
33. Roughly stick to the page limit and the requested format.
Don't make me try to guess how long your paper really is
because you 1.75-spaced the paper rather than double-
spacing it as requested in the call for papers.
Editors usually check this
Authors have been relying on technical reports to “extend” the paper
6 - DON'T WASTE MY TIME III
34. If you need a constant for the foo algorithm and you define
c=17, I want to know why you chose 17.
Also, some items in papers jump out at reviewers as odd.
For instance, I was recently reviewing a paper that simulated
a network with a bottleneck bandwidth of "about T1 rate". The
actual rate of the link turned out to be something like 1.3
Mbps.
why the authors used 1.3 Mbps rather than using T1 rate (~1.5
Mbps)?
Even if the explanation you end up with is rough, it is better than no
explanation at all in the vast majority of the cases.
7 - ANSWER ALL THE EASY QUESTIONS
35. 8. Make My Life Easy
It is not a super-big deal, but
it is always nice to get a paper that is easy to read
This includes things like using a very clear font that is of readable
size and numbering the pages
8 - MAKE MY LIFE EASY
36. Your paper does not solve the entire world's networking
problems. Really.
Many papers rub reviewers the wrong way by making bold
claims about solving all sorts of problems without any sort of
evidence (or only weak evidence)
It is much better to show some restraint and perspective when
discussing your results.
It is alright to solve small problems, or to only show
preliminary results that *may indicate* a solution.
Just don't over claim what you have shown.
9 - A LITTLE RESTRAINT
37. 10. Use Units
"The length of our data transfers is 1" means nothing. 1 what? 1
packet? 1 KB? 1 minute?
Sometimes it is obvious from the context -- sometimes it isn't. If you
use units you don't have to worry about it.
10 - USE UNITS
38. Theorems are quite a necessary part of many papers.
However, it is not necessary to give *only* a theorem with no
summary or context.
Summaries aid me as a reviewer to gain an intuitive
understanding before trying to dig into the math to make sure
what you have it correct
thus making my task easier and less error prone, I believe.
As a reader of a paper, I often just want the high-order bit
without slogging into the theorems and proofs to get said bit.
11 - WE'RE ALL NOT MATHEMATICIANS
39. One common mistake that authors make is failing to include
relevant references.
It is better to err on the side of citing too much related work
Next, when previous work must be criticized, do so in a
constructive manner.
In other words, do not say something like "Zevon's [1] work is easily
shown to be less effective than the work presented in this paper" --
Mr. Zevon might be reviewing your paper!
Something more along the lines of: "Zevon [1] first introduced the foo
algorithm … We show a more robust algorithm”
Finally, do not overstate what a paper you are citing says
It is often better to stick with the author's conclusions
12 - ON REFERENCES
42. 13 technical submissions
All of them were rejected by the Area Editors on the advice of the
reviewers
One could ask: were all the papers so terrible?
Some submissions were too broad, others too narrow,
Some too incremental, some too radical,
Some were just not interesting enough
I feel that this attitude has permeated our community at large
A similar spirit of harsh criticism is used to judge papers at SIGCOMM,
MOBICOM, CoNEXT, and probably every other top-tier computer science
conference
Reviewers seem only to want to find fault with papers
rather than appreciate insights
Despite
inevitable errors
lack of technical completeness
THE ACM SIGCOMM’S CCR - JULY 2011
43. 1. Subconscious desire to get one’s back
1. if my paper has been rejected from a venue due to sharp criticism…
2. Why not pay this back with sharp criticism of my own?
2. A desire to prove one’s expertise
1. if I can show that a paper is not perfect, that shows how clever I am
3. A biased view of what papers in a particular area should look
like
1. I’m the expert in my field so I think I know what every paper in my
field should look like
4. unrealistic expectations
1. I may not write perfect papers but I expect to read only perfect ones
5. Subconscious attitudes are exacerbated by
1. a ballooning of reviewer workloads
2. journals in computer science languishing in their roles, conference
papers being held to archival standard
44. One negative consequence is the stifling of innovation
they scale the walls by adding epsilon to delta until the
incrementality threshold is breached
well-studied areas are further overstudied to the detriment of others
A second negative consequence is that it turns some
researchers off
do not want to take part in a game where they cannot respect the
winners or the system
PC chairs and Area Editors have to change our mental attitude
from finding reasons to reject a paper
to finding reasons to accept a paper
We will certainly be trying to do this from now on at CCR
45. As a reviewer of a paper, it is your duty to critique a paper and
point out its flaws.
But can you overlook minor flaws ?
Can you find the greater good?
With this small change, the system will also change.
One review at a time
53. • Title: be creative
• Abstract: act as a marketing person
• Introduction: be clear. State scenarios, need
(motivation), topic importance, a quick view on
previous work/your solution/results, paper
organization
• Methodology: make it reproducible (at least, clear)
• Results: be creative on presentation (not much),
interpret data, infer things (not much), discuss the
lessons learned
• Related work: show both broad and close research
papers; state the difference to yours (during and
at the end of the section)
• Conclusion: do not add new material. Summarize
main findings and make it coherent with
title/abstract/intro
“Convince
the referee
that it is
worthwhile
reading
the next
section”
(me)
