Instructions and Policies

Thank you very much for serving on the JDS Review Board.

When you prepare your reviews, please consider that they serve two critical purposes:

• 1. To provide authors with feedback on how to improve their paper and proceed with their work, regardless of the paper decision.
• 2. To justify and document final decisions regarding publication of this version of the paper.

Importantly, please consider who reads your reviews: e.g., young PhD students who try to find their way on how to become researchers, as well as junior researchers who recently graduated with a PhD and are starting their career. Regardless of outcomes, our goal is to encourage people to keep improving their work and to keep contributing to our research community.

Review Constructively

We would like the reviewers to try to find the good in each paper and promote it. That is, do not go out trying to find reasons to reject a paper, but rather find reasons why the particular work might be promising and propose concrete ways on how the paper can be further improved.

Review Quality Week

We will have a Review Quality Week. The week after the review submission deadline and before the reviews are released to the authors the Associate Editors will lead an effort to work with the reviewers to make sure that all reviews are:

• 1. civil and respectful
• 2. detailed
• 3. all remarks are based on technical observations and facts

Regardless of whether a review marks a paper as accept or reject, it should be thoughtful, civil, and provide constructive feedback. Reviews that do not satisfy these principles will be flagged and the AEs will work together with the reviewers to propose improvements. Flagged reviews that are not updated to satisfy review quality metrics during Review Quality Week, will be dropped. The week before the review submission deadline we will send a reminder so that reviewers check their own reviews just before submission.

Novelty

Novelty can manifest in many ways that go beyond the traditional requirement of having a new algorithm or new system design. When reviewing we would like to consider many additional aspects of novelty such as having a novel result, novel engineering, or a novel benchmark. That is, even if the algorithms used and the problem itself are not dramatically new, a paper might still be providing new and useful insights.

Common issues

• 1. When proposing changes for a paper, such as new experiments, please also suggest concrete directions on how to make space for the new material. When we ask for revisions, the meta-review form will specifically ask for these suggestions.
• 2. Papers that have not gone through peer-review and are not formally published can be brought up if a reviewer feels it is relevant to inform the authors, but such papers cannot be considered in a way that diminishes the novelty of the submitted work or in order to ask the authors to compare explicitly against them, e.g., with experiments. This includes papers on arXiv and white papers.

Good reviewing practices

1. Review length

A good review should cover at least the following aspects of the paper in detail: problem focus and motivation, technical contributions, experimental evaluation, related work, and overall presentation. Please elaborate with regards to strong and weak points on all these fronts with detailed comments. We expect an ideal review to span at least 4-5 paragraphs, discussing these aspects. A very short review is very likely a bad review. Recall, that one of the purposes of your review is to give authors constructive feedback, and a short review does not accomplish that.

2. Critiquing the paper's motivation and focus

A good review should evaluate the importance and relevance of the problem that the authors list. The authors may not agree with your conclusion, but it is vital that they understand your rationale. For this reason, criticisms on the motivation of the problem should be clearly explained and justified.

Examples to avoid:

• "The motivation of the paper is weak": It is not clear to the authors whether you were dissatisfied with their presentation or with the problem they focus on.
• "This problem is not interesting" "I don't think anyone will use this": Avoid basing your critique on personal opinions and biases. Perhaps the paper's focus is not interesting to you, but that does not mean that it is not important. Give specific arguments to explain why you believe it is not practical. It is also not sufficient to simply state that you didn't find the problem motivation convincing. You should explain why you believe the paper's focus is unrealistic / too narrow / too broad / no longer relevant.

More constructive:

• "The introduction tries to establish the problem motivation by citing prior work, but the paper should stand more on its own. The motivation would improve by discussing real-world examples of this problem and the challenges of current technology."
• "The introduction of the paper attempts to explain the problem the paper intends to address. However, the introduction makes several self-contradicting statements (such as....) and thus the current presentation makes it sheer impossible to adequately understand the actual problem the paper addresses...."

3. Critiquing the technical contributions

A big part of your review is to evaluate the technical contributions of the work. Examine the claims made in the paper, judge if they are applicable to the problem, and evaluate their expected importance, applicability, and usefulness. If you believe that some assumptions of the paper are too restrictive and make the problem unrealistic or narrow, explain that. As a reviewer, we rely on your expertise to evaluate the novelty of a paper's contributions. However, you should avoid tainting your evaluation with personal opinions. Judge the work that the authors are presenting, rather than the work that you have preferred to see.

Examples to avoid:

• "The paper makes strong theoretical contributions in this domain, but it doesn't test them in a real-world, large-scale deployment"
• "Algorithm A is really interesting and insightful, but what about algorithm B that I just came up with?"
• "The paper solves the problem for 4 dimensions, but what about 5 dimensions?"

It is great to bring up additional contributions that you would like to see and ideas that could extend the presented work. But write them as suggestions, rather than a basis of criticism. Your review should judge the contributions that the paper makes, rather than the contributions it does not make. As a rule of thumb, if your criticism could apply to any paper (e.g., "the paper does A very well, but what about B") then it is not a fair criticism.

More constructive:

• "Algorithm A is non-trivial and has low time complexity. However, it requires maintaining a very large data structure in memory, which makes it impractical for large datasets."

4. Critiquing the evaluation

Your review should evaluate whether the experiments validate the claims of the paper. If the paper claims that the proposed algorithm improves on the state-of-the-art, discuss whether the experiments compare the proposed approach with the appropriate methods, and against all expected metrics (e.g., efficiency, accuracy, scalability). Your review should note whether there are any inconsistencies in the results and whether there are observations that are not explained, justified, or discussed. We also rely on you to judge whether the experimental methods and datasets are appropriate, or if they introduce biases in the results.

Examples to avoid:

• "The experiments compare the proposed approach with methods A, B, and C, but method D could also be applicable if one made these parameters flexible": This is more often than not an unfair criticism. If a method needs non-trivial adjustments to apply in this work, and the paper already compares with suitable techniques, this should not be a basis of criticism.
• "Larger-scale experiments are needed": This comment by itself is not sufficient criticism. Communicate to the author what scale of experiments you would consider satisfactory and why.

More constructive:

• "The experiments do not include comparisons with other methods, because this is the first work that addresses problem X. However, I believe that Algorithm ABC for problem Y could be applicable here with the following minor modifications:..."
• In addition, as you consider such remarks, please consider space restrictions. For example, are there any of the existing experiments that you think are less useful and can be dropped?

5. Critiquing the writing and presentation

You may often find the presentation of a paper unsatisfactory. Please list specific points of possible improvement such as repetitiveness, poor structure, confusing notation, missing examples, or lack of formalism. However, make sure that you provide concrete examples and pointers for your criticisms. For example, explain what about the notation confused you, which terms were not properly defined, and which statements should be made more rigorous. Communicating vague discontent about the readability of the paper is not constructive, because it doesn't explain to the authors how they can improve it.

Examples to avoid:

• "The paper is unreadable": Only OK if the paper is actually written in a foreign language. Otherwise, give some examples of the readability issues that you experienced.
• "The notation is confusing": Be more explicit about what you found problematic, and what you think might help.

More constructive:

• "I found section 4.1 hard to follow. I believe that it would help if the authors moved the discussion at the end of section 4.3 to the beginning of section 4.1, as it gives the intuition behind the definitions"
• "There is a clash of notation: sometimes \alpha is used to represent a query answer, but it is also the approximation factor in Algorithm 1."

6. Critiquing the related work

You should ensure that the paper includes sufficient discussion of the related work, explains similarities and differences, and does not omit obvious connections. Keep in mind however that, unless you are reviewing a survey paper, there will always be some citations that could have been added but weren't. Do not criticize a paper for the omission of a particular citation if it already includes sufficient discussion of the related work in the particular area. Do point out though glaring omissions and important connections that were missed.

Examples to avoid:

• "This work seems very similar to [X]": This claim is vague and unsubstantiated. Provide clear discussion on whether particular claims have been demonstrated before in prior work.
• "There is a large body of related work on problem X that the paper does not discuss": You should make such a criticism more concrete by including particular references that you expected to see discussed. Suggest to the authors a few specific citations that they should include and explain why.

More constructive:

• "The authors mention connections to [X] in the related work, but they do not explain why [X] is not directly applicable and why a new approach is needed."

7. Professionalism

We expect you to write thoughtful and professional reviews. Refrain from using language that is demeaning and ensure that your criticisms are constructive and clearly supported. Keep a friendly tone and avoid being condescending. A thoughtful and courteous review communicates to the authors that you read their paper with a positive attitude, and not with a mind to find faults with it. Well-structured arguments in your reviews help assure the authors that their papers were treated fairly and professionally.

8. Inclusive language

In keeping with JDS’s efforts on Diversity and Inclusion, please check the paper for language that may further the marginalization, stereotyping, or erasure of any group of people, especially historically marginalized and/or under-represented groups (URGs) in computing. Authors have been given detailed guidelines and examples on this front. Please read the Diversity and Inclusion page under the Author menu for the specific instructions we have given to authors for this: If you find exclusionary or hurtful language or examples (even if unintentional), point them out in your review in the appropriate review field and ensure the authors change the text accordingly. Please also be mindful of using inclusive language in your own review text, as well as during the discussion phase.