Reviewer FAQ

 

General Questions

 

-- I discovered the identities of authors of some papers I review, what to do now?

 

-- I have a very good guess who the authors are, but I don't know for sure. How shall I answer Questions 13 and 14?

 

-- Some of my papers violate ICML style/anonymity guidelines, shall I review them?

 

-- Do other reviewers and the meta-reviewer know who I am?

 

-- Do authors know who I am?

 

-- Am I required to review supplementary materials?

 

-- What if I don’t have enough expertise to write a strong review for some paper?

 

-- I know that the paper I review is under submission to a different conference, what to do?

 

-- Do you provide any accessibility support?

 

-- What if something is unclear in the paper and I have questions for the authors?

 

-- What happens after my review is submitted?

  

Author Response and Discussion

 

-- What should I do now that author response is in and discussion has begun?

 

-- When does the discussion period end?

 

-- What can the author response include? Links? Author identity? Should authors have responded to every issue brought up in reviews?

 

-- Does ICML allow "revise and resubmit" decisions on papers?

 

-- What is the Reproducibility Checklist?

 

-- Do other reviewers know my identity? Does the meta-reviewer?

 

-- There was a paper I reviewed but it is not there anymore! What happened?

 

-- I am asked if I read the author response, but there is no author response. What should I do?

 

-- Oh no, my name shows up in the discussion when I posted a comment! I thought I was supposed to be anonymous!

  

Code Submission

 

-- What is the usage policy for code?

 

-- Is there anything I need to be aware of before I access the code?

 

-- What do we expect from the authors by way of code submission?

 

-- How do I use the code in the decision process?

 

-- What if the submitted code is not runnable? What if it is just pseudo-code?

 

-- What if the submitted code is not anonymous?

 

-- What if I find that the submitted code is empty or the link does not work?

  

Specific Questions

 

-- A very recent paper is not cited, is it a problem?

 

-- Not all baselines are considered due to limited computational resources, is it a problem?

 

-- How should I review papers for which a new dataset is a significant contribution?

 

General Questions

 

-- I discovered identities of the authors of some papers I review, what to do now?

Under no circumstances should you attempt to find out the identities of the authors for any of your assigned submissions (e.g., by searching on Google or arXiv). If you accidentally find out, please do not divulge the identities to anyone, but do tell the meta-reviewer assigned to that paper that this has happened. You should not let the authors’ identities influence your decision in any way.

Additionally, the review form will contain a question asking you if you have discovered the authors' identities and how; please respond honestly. The answers to this question to estimate the degree of anonymity preserved in the review process, and try to understand the factors that influence it.

 

 -- I have a very good guess who the authors are, but I don't know for sure. How shall I answer Questions 13 and 14?

Please answer yes if you know the identity of at least one author (e.g. the research group this paper is coming from); also answer yes if you believe you have guessed the authors and you are confident about this guess. The next question will allow you to give us more details.

 

 -- Some of my papers violate ICML style/anonymity guidelines, shall I review them?

Please contact your AC if you find that a paper violates ICML style/anonymity. All ICML submissions have been automatically checked but this does not guarantee that all violations have been detected.

 

-- Do other reviewers and the Meta-reviewer know who I am?

Your identity is visible to other reviewers starting with the discussion period. It is also visible to the meta-reviewer in charge of a submission, as well as to the senior area chars and the program chairs.

 

-- Do authors know who I am?

No, the review process is double-blind and your identity, as well as the identity of meta-reviewers, is hidden from authors.

 

-- Am I required to review supplementary materials?

Your responsibility as a reviewer is to read and review the submission itself; looking at supplementary material is at your discretion. That said, you may wish to look at supplementary material before criticizing a submission for insufficient details, proofs, or experimental results.

A small number of ICML submissions have by mistake included the supplementary material along with the main submission. We have not desk rejected these submission, although technically, they are above the allowed page limit. As a reviewer, you are asked to review the main paper. Please treat the material after the references as supplementary material; that is, it is at your discretion to read it or not.

Note that the Supplementary Material deadline was one week after the main paper submission deadline. Many authors have taken the opportunity to submit improved/updated versions of their main paper as supplements. Please treat these versions as you would treat any other supplementary material: as supporting information, that you can use at your discretion to better evaluate the submission. In your evaluation, for consistency and fairness, please always refer by default to the main paper. If you have comments on the supplement, be explicit about that. For example, the main paper is unclear about a point, but this is clarified in the supplement. Then, answer question 7. [Clarity] w.r.t main paper, but feel free to explain in the Detailed comments section that after reading the supplement you believe the final version,  if accepted, would be better in this aspect.

Note that Author Instructions require  "material critical to the evaluation of your paper, [...] needs to be included in the paper".

 

-- What if I don’t have enough expertise to write a thoughtful review for some paper?

Please inform the meta-reviewer for that paper immediately (by Feb 19, 2021).

 

-- I know that the paper I review is under submission to a different conference or journal; what to do?

Please bring it to the attention of the meta-reviewer for that paper immediately. Please do the same with other instances of misconduct that you suspect, but please avoid discovering the identities of the authors in trying to verify your surmises.

 

-- What if something is unclear in the paper and I have questions for the authors?

If you have specific questions for the authors, please list them clearly in the either in "Detailed comments for Authors" or in "Questions for Authors" section of your review. Authors will have a one-time opportunity to provide a response to the reviews, at which time they may answer your specific question. They will be instructed to give priority to the questions listed in "Questions for Authors" if they do not have space to respond to all your comments and questions. After the feedback period, you will be able to update your review and score.

 

-- What happens after my review is submitted?

From the begining of the Discussion period, you will be able to see review from all the other reviewers. Additionally, if needed, you will be asked to participate in discussion amongst the reviewers and meta-reviewer. Authors will have had the opportunity to respond to questions and concerns raised in reviews during author feedback period (see previous question). The discussion period ends on April 12, 2021, at which point meta-reviewers will make acceptance recommendations and provide meta-reviews for their papers.

 

Author Response and Discussion

 

-- What should I do now that author response is in and discussion has begun?

1.  Please read the author feedback, and let the authors know that you have done so (by at least checking the "I read the response" button on the review form).
2. Please update your scores and your review in light of the feedback, if applicable. Please take author feedback into account. Authors put a lot of time into providing feedback, and it is disheartening when it feels like it was ignored. If the authors have satisfactorily addressed your criticism(s), then please update your score accordingly. If not, then explain what remains problematic.
3. If your scores disagree substantially from the other reviewers, then please begin discussion with the other reviewers and the Area Chair in charge of the paper, to ensure that all relevant aspects of the paper have been discussed (if consensus can be reached, that's great, but it may not always happen).

 

-- When does the discussion period end?

Discussion must conclude by April 12, and all reviews must be updated by that date.

 

-- What can the author response include? Links? Author identity? Should authors have responded to every issue brought up in reviews?

As the length of the author response is limited, and as authors cannot reveal their identities in the response, please keep in mind that not all issues raised by a reviewer can be addressed; a paper need not be flawless to be accepted, so please evaluate the degree to which the feedback addresses the *important* points in a review, rather than whether it addresses *all* points. If you observe any author feedback that violates the rules (e.g., include external links, or de-anonymize authorship), please bring them to our attention.

Author responses are allowed to contain embedded figures, tables, etc. as long as the authors anonymity is preserved.

 

-- Does ICML allow "revise and resubmit" decisions on papers?

ICML does not allow "revise and resubmit," so if the authors cannot satisfactorily deal with the reviewer criticisms without a major revision (which would need to be checked in a second review), then the paper should be rejected.

 

-- What is the Reproducibility Checklist?

Authors have been asked to fill out a "Reproducibility Checklist" as part of their submission, in which they declare what aspects of their paper are reproducible and how. You may wish to check the author's responses to the Reproducibility Checklist when answering Question 8 in the review form.

 

-- Do other reviewers know my identity? Does the meta-reviewer?

Yes, starting with the discussion phase, other reviewers will know your identity.

 

-- There was a paper I reviewed but it is not there anymore! What happened?

Several authors have withdrawn their papers following the review process, in which case you will no longer be able to see their paper on CMT.

 

-- I am asked if I read the author response, but there is no author response. What should I do?

In this case, please just select "yes."

 

Code Submission

 

-- What is the usage policy for code?

Any submitted code is confidential, just like any paper submission. You are only allowed to use it for the purpose of ICML reviewing simply for the duration of the review period. Please remember that code is proprietary. Just like the paper submission, you are required to keep the code confidential and to delete it at the end of the review period to comply with the need for confidentiality.

 

-- Is there anything I need to be aware of before I access the code?

Please remember that just like any other untrusted code, any submitted code may contain security vulnerabilities. To guard against these, we recommend that either you look at the code, or you run any downloaded code on a virtual machine or a docker that is not connected to the internet. See https://docs.docker.com/get-started/ for an example.

Some authors may also submit an anonymous link to a code repository (like github). In such cases, please make sure to either manually download the code or to clone (and NOT fork) the repository as the authors will be able to see who forks their repository and thus infer your identity. Please make sure you log out of your own github account before accessing such links, as an extra precaution.

You may wish to also be cautious about accessing other web links provided from the paper, as these may contain vulnerabilities or may log visitor IP addresses.

 

-- What do we expect from the authors by way of code submission?

Authors who choose to submit code are supposed to submit it in anonymized form, either as a zip file or as part of an anonymous github link. The code submitted may or may not be standalone or runnable by a third party; and submission of detailed pseudocode that can help you verify the credibility of the results is also allowed. We are not asking authors to submit any private data; however, they can choose to submit toy or simulated data for testing code. Also, if the authors are introducing a new dataset, then they are encouraged to make it available in an appropriate data repository.  

 

-- How do I use the code in the decision process?

Just like supplementary material, submitted code provides additional evidence to add credibility to the results in a paper. High quality papers whose results appear credible should be accepted, even if code is not submitted. However, if something is unclear in the paper, then please use code, if submitted, to clarify the details of the results.

 

-- What if the submitted code is not runnable? What if it is just pseudo-code?

Authors are permitted to submit code that is not runnable as well as detailed pseudocode. Our advice in this case is to look over the code to validate the results as much as you can. If the validity is still unclear, please mention this in your review.  

 

-- What if the submitted code is not anonymous?

If the code reveals the authors or the organization name, please bring it to the attention of the meta-reviewer for this paper immediately.

 

-- What if I find that the submitted code is empty or the link does not work?

Please put this in your review.

 

Specific Questions

 

-- A very recent paper is not cited, is it a problem?

Submissions should not be penalized for not comparing with or citing papers which show up on arxiv up to a month before the abstract submission deadline (i.e., any paper appearing in calendar year 2021 should not be grounds for penalty).

 

-- Not all baselines are considered due to limited computational resources, is it a problem?

Submissions should not be automatically penalized for not providing comparisons to baselines that require extensive computing resources (e.g. tens or hundreds of GPUs),

if comparisons to baselines that require extensive computing are not essential to the significance of a paper. For instance, if the paper would benefit other researchers who do not have access to extensive computing resources.

 

-- How should I review papers for which a new dataset is a significant contribution?

Some papers may release a dataset. We have encouraged authors to follow several norms for dataset release (e.g., providing links, DOIs, licenses, etc.). As part of the review, you are asked which of these norms is followed. Such information is not required and should not impact the score you give to a paper.