Skip to yearly menu bar Skip to main content


ICML 2020 Reviewer Guidelines

How to review?

The intent of the review process is twofold. First, to identify papers which offer significant contributions to the field of machine learning, for attendees and readers. Second, to provide constructive feedback to authors that they can use to improve their work. Your role as a reviewer is critical to both goals. When reviewing a paper, always think about the impact the work may have on the community in the long run --- out-of-the-box ideas, novel problems and “bridging fields” contributions are crucial for the successful development of the field so do not neglect high-level picture in favor of technical correctness, which is also important. 

Keep in mind: Novel and/or interdisciplinary works (e.g., which are not incremental extensions of previously studied problems but instead perhaps formulate a new problem of interest) are often very easy to criticize, because, for example, the assumptions they make and the models they use are not yet widely accepted by the community (due to novelty). However, such work may be of high importance for the progress of the field in the long run, so please try to be aware of this bias, and avoid dismissive criticism. 

In this document, we go through the reviewer form and (Part 1) discuss what we expect from you in each of its sections and (Part 2) give some examples of good reviews.   

Acknowledgments: The examples of strong reviews we use in this document are partially adopted from the NeurIPS 2019 reviewer guidelines, which in turn utilize reviews written for some NeurIPS and ICLR papers. We thank the NeurIPS program committee and reviewers who write such remarkable reviews.

 

Part 1. Theory of a good review

1. Summary of main claim(s)

Your review begins with a summary of the main ideas of the submission. Although this part of the review may not provide much new information to authors, it is invaluable to the meta-reviewers and program chairs, and it can help the authors determine whether there are misunderstandings that need to be addressed in their author response. Please be brief, but specific.

2. Merits of the paper

In this field, you should list the main contributions of the paper. Contributions may be theoretical, methodological, algorithmic, empirical, connecting ideas in disparate fields (“bridge papers”), formulating a novel problem or providing a critical analysis (e.g., principled justifications of why the community is going after the wrong outcome or using the wrong types of approaches).  One measure of the significance of a contribution is (your belief about) the level to which researchers or practitioners will build off of or use the proposed ideas. Papers that explore new territory or point out new directions for research are preferable to papers that advance the state of the art only incrementally.  

Please remain polite and avoid writing “This paper did not contribute any new ideas.”  Instead, try to find at least one contribution (for example, something along the lines of “This paper proposes a model that primarily combines models in [cite A] and [cite B]”) and then comment on its significance in subsequent sections.

3. Overall evaluation

You should evaluate how strong this paper is, weighing its strengths against any potential flaws. Your evaluation should reflect absolute judgments of the contributions made by each paper. To understand the typical level of the ICML papers, you may browse last year ICML proceedings http://proceedings.mlr.press/v97/. 

You should not assume that you have received an unbiased sample of papers, nor should you adjust your answers to achieve an artificial balance of positive and negative recommendations across your batch of papers.

4. Justification of your recommendation

In this section, you need to write concrete arguments that explain why you gave a particular recommendation to the submission. Your arguments should be objective, specific, concise and polite. Please avoid vague, subjective complaints. Remember that you are not reviewing your personal level of interest in the submission, but its scientific contribution to the field. For each argument, try to briefly explain its significance. If some argument requires a long explanation, then it is ok to write that explanation in the the next section on detailed comments to authors, but make sure you do not make any unsubstantiated claims (for instance, if you assert that some aspect of the paper has been done before, please provide relevant citations to that claim). 

5. Detailed comments for authors

In order to provide the most useful feedback to authors, write this section in a top-down manner and start from the most important aspects. For example, if you want to support some of the arguments you made in the previous section, then this justification should come at the beginning of this section. Beyond clarification of the arguments, please comment on the following criteria, as appropriate for the paper: 

 - The significance and novelty of the paper's contributions.
 - The paper's potential impact on the field of machine learning.
 - The degree to which the paper substantiates its main claims.
 - Constructive criticism and feedback that could help improve the work or its presentation.
 - The degree to which the results in the paper are reproducible.
 - Missing references, presentation suggestions, and typos or grammar improvements.

Again, begin with the comments you find the most significant and put the least significant comments (which can be nevertheless very useful for authors) at the end of this section.

 

 

Part 2. Examples of a good review

1. Merits of the paper

The following are examples of contributions a paper might make. This list is not exhaustive, and a single paper may make multiple contributions.

  • “The paper provides a thorough experimental validation of the proposed algorithm, demonstrating much faster runtimes without loss in performance compared to strong baselines.”

  • “The paper proposes an algorithm for [insert] with sample complexity [or computational complexity] scaling linearly in the observed dimensions; in contrast, existing algorithms scale cubicly.”

  • “The paper presents a method for robustly handling covariate shift in cases where [insert assumptions], and demonstrated the impact on [insert application].”

  • “The paper provides a framework that unifies [insert field A] and [insert field B], two previously disparate research areas.”

  • “This paper demonstrates how the previously popular approach of [insert] has serious limitations when applied to [insert].”

  •  “This paper formulates a novel problem [insert brief description]  and clearly shows that it is of interest to the community because [insert brief explanation].”

 

2. Justification of your score and detailed comments for authors

When justifying your score and writing detailed comments to authors, consider the following (non-exclusive) list of criteria: significance, novelty, potential impact, technical quality, presentation/clarity and reproducibility. Below are examples of good comments with respect to each of these criteria. 

a) Significance

Try to answer the following questions: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?

  • Example: “This article answers a very natural question: algorithm A is an extremely classical, and very simple algorithm, yet we do not fully understand its convergence rate. This paper provides a novel proof that is conceptually simple and elegant, and I found its presentation very clear.”

  • Example: ”This paper seems to be a useful contribution to the literature on [topic], showing a modest improvement over the state of the art. However, the paper could be strengthened by demonstrating and analyzing the efficacy of the approach in [situation X].”

b) Novelty

Try to answer the following questions: Are the tasks or methods new? Is the work a novel combination of well-known techniques? Is it clear how this work differs from previous contributions? Is related work adequately cited? Does the work formulate a novel problem?

  • Example: “The main contribution of this paper is to offer a convergence proof for minimizing sum fi(x) + g(x) where fi(x) is smooth, and g is nonsmooth, in an asynchronous setting. The problem is well-motivated; there is indeed no known work on this, in my knowledge .… There are two main theoretical results. Theorem 1 gives a convergence rate for the proposed algorithm, which is incrementally better than a previous result. Theorem 2 gives the rate for an asynchronous setting, which is more groundbreaking.”

  • Example: “The paper is missing a related work section and also does not cite several related works, particularly regarding [topic 1] (list of citations), [topic 2] (list of citations) and [topic 3] (list of citations). The proposed model is similar to that of (citation), though the [specific detail 1] makes the proposed method sufficiently original. The study of [specific detail 2] is also fairly novel.”

c) Potential Impact

Try to answer the following questions: Will the ideas in this paper have an impact on the community in the long run? Does the paper bridge previously disconnected fields? Does the paper push the community in a new, interesting/important direction?

  • Example: “In order to prove Theorem 2, the paper presents a novel proof technique based on [key idea]. This technique could potentially be used for a much broader class of problems such as [X], [Y] and [Z], and would lead to improved rates in settings where [assumption holds].”
     

  • Example: “The approach presented in this paper is well-evaluated in [domain], but potentially useful in many other settings. Because the approach is somewhat complex, it could have even more potential impact if the authors released an implementation.”

d) Technical Quality

Try to answer the following questions: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Is this a complete piece of work or work in progress? Are the authors careful about evaluating both the strengths and limitations of their work? Does the evaluation justify the main claims of the paper?

  • Example: “The technical content of the paper appears to be correct albeit some small mistakes that I believe are typos instead of technical flaw (see #4 below).…
    4. The equation in line 125 appears to be wrong. Shouldn't there be a line break before the last equal sign, and shouldn't the last expression be equal to [equation]?”

  • Example: “The idea of having a bound for [X] is certainly good. While the paper did demonstrate that the bound does indeed contain [X] as expected, it is not entirely clear that this bound will be useful for model selection. This is not demonstrated in the experiments reported in the paper, despite being one of the main claims of the paper.”

e) Presentation/Clarity 

Try to answer the following questions: For a reader with the appropriate background knowledge, is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improvement.)

  • Example: “The paper is generally well-written and structured clearly. The notation could be improved in a couple of places. In the inference model (equations between ll. 82-83), I would suggest adding a frame superscript to clarify that inference is occurring within each frame, e.g. [equation]. In addition, in Section 3 it was not immediately clear that a frame is defined to itself be a sub-sequence.”

  • Example: “While the paper is fairly readable; there is substantial room for improvements in the clarity. There were several variables that were used in equations before they were defined, such as [example 1] and [example 2]. Moreover, in the statement of Theorem 2, it was unclear whether the same assumptions were being made as in Theorem 1. Finally, I was sometimes confused because \ell appears to be overloaded and used to mean both [X] and [Y].”

f) Reproducibility

Try to answer the following question: does the paper contain enough details to reproduce the results? If the submission has supplementary code and you managed to run it and reproduce the results [this is optional for reviewers], please do mention it in your review.

  • Example: “The paper describes all the algorithms in full detail and provides enough information for an expert reader to reproduce its results. However, it seems that Theorem 1 requires an additional assumption of d>3 which is not specified..” 

  • Example: “Neither main text nor the supplementary code explain how the hyperparameters are selected for the synthetic experiment. The paper should explain the specific procedure used to alleviate reproducibility concerns”.