When you prepare your reviews, please consider that they serve two critical purposes:
Importantly, please consider who reads your reviews: e.g., young PhD students who try to find their way on how to become researchers, as well as junior researchers who recently graduated with a PhD and are starting their career. Regardless of outcomes, our goal is to encourage people to keep improving their work and to keep contributing to our research community.
We would like the reviewers to try to find the good in each paper and promote it. That is, do not go out trying to find reasons to reject a paper, but rather find reasons why the particular work might be promising and propose concrete ways on how the paper can be further improved.
We will have a Review Quality Week. The week after the review submission deadline and before the reviews are released to the authors the Associate Editors will lead an effort to work with the reviewers to make sure that all reviews are:
Regardless of whether a review marks a paper as accept or reject, it should be thoughtful, civil, and provide constructive feedback. Reviews that do not satisfy these principles will be flagged and the AEs will work together with the reviewers to propose improvements. Flagged reviews that are not updated to satisfy review quality metrics during Review Quality Week, will be dropped. The week before the review submission deadline we will send a reminder so that reviewers check their own reviews just before submission.
Novelty can manifest in many ways that go beyond the traditional requirement of having a new algorithm or new system design. When reviewing we would like to consider many additional aspects of novelty such as having a novel result, novel engineering, or a novel benchmark. That is, even if the algorithms used and the problem itself are not dramatically new, a paper might still be providing new and useful insights.
A good review should cover at least the following aspects of the paper in detail: problem focus and motivation, technical contributions, experimental evaluation, related work, and overall presentation. Please elaborate with regards to strong and weak points on all these fronts with detailed comments. We expect an ideal review to span at least 4-5 paragraphs, discussing these aspects. A very short review is very likely a bad review. Recall, that one of the purposes of your review is to give authors constructive feedback, and a short review does not accomplish that.
A good review should evaluate the importance and relevance of the problem that the authors list. The authors may not agree with your conclusion, but it is vital that they understand your rationale. For this reason, criticisms on the motivation of the problem should be clearly explained and justified.
A big part of your review is to evaluate the technical contributions of the work. Examine the claims made in the paper, judge if they are applicable to the problem, and evaluate their expected importance, applicability, and usefulness. If you believe that some assumptions of the paper are too restrictive and make the problem unrealistic or narrow, explain that. As a reviewer, we rely on your expertise to evaluate the novelty of a paper's contributions. However, you should avoid tainting your evaluation with personal opinions. Judge the work that the authors are presenting, rather than the work that you have preferred to see.
It is great to bring up additional contributions that you would like to see and ideas that could extend the presented work. But write them as suggestions, rather than a basis of criticism. Your review should judge the contributions that the paper makes, rather than the contributions it does not make. As a rule of thumb, if your criticism could apply to any paper (e.g., "the paper does A very well, but what about B") then it is not a fair criticism.
Your review should evaluate whether the experiments validate the claims of the paper. If the paper claims that the proposed algorithm improves on the state-of-the-art, discuss whether the experiments compare the proposed approach with the appropriate methods, and against all expected metrics (e.g., efficiency, accuracy, scalability). Your review should note whether there are any inconsistencies in the results and whether there are observations that are not explained, justified, or discussed. We also rely on you to judge whether the experimental methods and datasets are appropriate, or if they introduce biases in the results.
You may often find the presentation of a paper unsatisfactory. Please list specific points of possible improvement such as repetitiveness, poor structure, confusing notation, missing examples, or lack of formalism. However, make sure that you provide concrete examples and pointers for your criticisms. For example, explain what about the notation confused you, which terms were not properly defined, and which statements should be made more rigorous. Communicating vague discontent about the readability of the paper is not constructive, because it doesn't explain to the authors how they can improve it.
You should ensure that the paper includes sufficient discussion of the related work, explains similarities and differences, and does not omit obvious connections. Keep in mind however that, unless you are reviewing a survey paper, there will always be some citations that could have been added but weren't. Do not criticize a paper for the omission of a particular citation if it already includes sufficient discussion of the related work in the particular area. Do point out though glaring omissions and important connections that were missed.
We expect you to write thoughtful and professional reviews. Refrain from using language that is demeaning and ensure that your criticisms are constructive and clearly supported. Keep a friendly tone and avoid being condescending. A thoughtful and courteous review communicates to the authors that you read their paper with a positive attitude, and not with a mind to find faults with it. Well-structured arguments in your reviews help assure the authors that their papers were treated fairly and professionally.
In keeping with JDS’s efforts on Diversity and Inclusion, please check the paper for language that may further the marginalization, stereotyping, or erasure of any group of people, especially historically marginalized and/or under-represented groups (URGs) in computing. Authors have been given detailed guidelines and examples on this front. Please read the Diversity and Inclusion page under the Author menu for the specific instructions we have given to authors for this: If you find exclusionary or hurtful language or examples (even if unintentional), point them out in your review in the appropriate review field and ensure the authors change the text accordingly. Please also be mindful of using inclusive language in your own review text, as well as during the discussion phase.