Thorsten Joachims & Johannes
Fürnkranz
ICML 2010 Program Chairs
This summary describes the reviewing process of the 27th International Conference on Machine Learning (ICML 2010), and it presents the results of a survey in which 560 members of the ICML community participated. ICML is the annual conference of the International Machine Learning Society (IMLS) and provides a venue for the presentation and discussion of current research in the field of machine learning. The proceedings of recent ICML conferences can be found on-line at http://www.machinelearning.org, those of ICML 2010 are currently still available at http://www.icml2010.org/proceedings.html.
ICML 2010 was held June – June 25 in Haifa, Israel, making it the first ICML to be located in Israel. The conference was immediately followed by the 23rd Annual Conference on Computational Learning Theory (COLT 2010), both conferences sharing a joint workshop day and a leisure day with an organized tour to Jerusalem.
The technical program consisted of 152 contributed papers selected via the reviewing process described below. As the plenary invited speakers, we are excited to have won Tom Mitchell, Nir Friedman, and Duncan Watts, as well as Nobel Prize winner Robert Aumann for a special guest lecture. We also had 7 invited application papers, which were presented in special parallel sessions.
The conference program was flanked by an exciting tutorial and workshop program, starting off with 7 tutorials on the first day of the conference, and finishing with 9 workshops on its last day.
We received 594 manuscripts for review, which roughly matches the number of submissions in the previous year. Of those submissions, eventually 152 papers were accepted using a thorough two-phase reviewing process with a program committee consisting of 46 Area Chairs and 517 Reviewers. The reviewing process was double blind, meaning that Reviewers could not see the author names. Area Chairs and Program Chairs, however, were aware of the author identities. All accepted papers received both an oral and a poster presentation at the conference. The proceedings are evidence of the resulting high-quality and exciting program.
In the call for papers, we specified 64 area keywords. These were used for selecting 42 Area Chairs in a strictly area-driven process. We tried to account not only for area popularity but also for breadth of coverage, so that we had suitable chairs even for areas that had low submission rates in the past. Four additional Area Chairs were nominated after the submission deadline to better match the actual area distribution of the submissions. Each of the 46 Area Chairs was responsible for 2 – 4 areas, and most areas were covered by more than one chair. Authors could mark their submissions with up to six areas (one primary and five secondary), and Area Chairs were asked to bid on papers in their areas.
This information was then used to automatically generate candidate assignments of papers to Area Chairs. The final assignment was made by the Program Chairs. Deliberately loosening the constraint of a balanced work-load throughout the reviewing process, together with the above-mentioned redundancy in the area-to-area-chair mapping gave us more flexibility to assign papers to appropriate chairs and diminished the risk of the proverbial "odd paper" in the batch. Eventually, each Area Chair was responsible for 10 – 15 papers.
Initially, more than 600 Reviewers were nominated by the Area Chairs, and about 80% of them accepted our invitations. More Reviewers were later added by the Area Chairs throughout the reviewing process.
Papers were reviewed in two phases. For Phase I, Reviewers bid on papers and each Reviewer was assigned 2–3 papers based on these bids. At the end of Phase I, every paper had 2 reviews. These two reviews were the basis of the Author Rebuttal. Authors could respond to the Reviewers, but could also make confidential comments to the Area Chair. In several cases, the confidential comments identified shortcomings and problems with the review assignments of Phase I, which the Area Chair could act upon for the assignments in Phase II.
If, on the basis of the two reviews from Phase I and the Author Rebuttal, the Area Chair identified substantial problems that left no hope for eventually accepting the paper, the Area Chair could reject a paper at this point. This early reject option was used for 78 submissions (including a few submissions that were withdrawn by the authors after seeing the first two reviews).
In Phase II, the Area Chairs were able to manually assign additional Reviewers to each of their papers. All papers that were not early rejects received at least one further review, but the Area Chair could assign as many additional Reviewers as necessary to make an confident decision. All papers that made it into Phase II received at least 3 reviews in total, some receiving up to 6 reviews. Because of time constraints, there was no Author Rebuttal for Phase II reviews.
One guiding principle in designing this two-phase review process was to give all involved parties some influence in the most crucial decisions within the reviewing process, namely the assignment of Reviewers and Area Chairs to papers.
We believe that the joint expertise of all parties provided good assignments of Reviewers to papers, and hope that nobody felt that their voice was not heard.
During and after Phase II, Reviewers who had already submitted their review were able to discuss the paper. Based on all reviews, the Author Rebuttal of Phase-I reviews, the discussion, and possibly reading the paper themselves, the Area Chair made a recommendation for acceptance/rejection as argued in their meta-review. Each Area Chair also had a "Buddy" Area Chair, who had read-access to all of their reviewing data and could be asked for a second opinion on difficult papers.
All recommendations were reviewed by the Program Chairs. In several cases the Program Chairs assigned additional Reviewers or Area Chairs to come to confident conclusions on all papers.
The Best Paper Committee selected the Best Paper, the 10-Year Best Paper Award, the Best Student Paper, the Best Application Paper, and their runner-ups. The award-winning papers as well as a detailed description of the selection process can be found at http://www.icml2010.org/awards.html.
We thank Springer, publisher of Machine Learning, for sponsoring our student paper awards.
In addition to the submitted papers, we introduced an Invited Application Track this year. To bring expertise and interesting research questions from related fields to ICML, we invited 7 papers from interesting application areas to the conference. These papers were not reviewed at ICML, but have already been reviewed and published at recent conferences in these application areas. The papers to invite were identified by a committee, particularly focusing on papers that opened interesting new questions and applications for machine learning research. The committee was asked to select one paper in each of the following subject areas: Robotics, Computer Systems, Computer Vision, Natural Language Processing, Computer Music and Entertainment, Games and Planning, and Computational Biology. The Invited Application Papers received a regular slot in a parallel session, as well as the opportunity to publish a write-up of their work in the proceedings. We hope that this Invited Applications Track leads to an interesting exchange of ideas between fields and further extends the importance of machine learning approaches.
To get feedback on the reviewing process and identify further ideas for improvement, we conducted a survey during the two weeks before ICML 2010. All Authors that submitted papers to ICML 2010, as well as all Reviewers and Area Chairs were invited to participate, and we were delighted to get 560 responses. The survey contained a total of 38 questions in the following topic areas - not only about the reviewing process, but also other questions about the current and future ICMLs.
The main results from the survey - as well as additional statistics about ICML 2010 - are summarized in the Presentation of the Program Chairs and the Presentation of the Local Chairs for the business meeting. However, not everything could be covered in the presentations. Therefore, we decided to publish the full survey results here. Below are the responses partitioned into the 4 main constituencies:
Many people worked together in organizing ICML 2010 and ensuring its success. All of them are named in the proceedings and on the ICML 2010 web-site, including the members of the Invited Applications Committee, the members of the Best Paper Awards Committee, the officers and board members of the IMLS, the Area Chairs, and the Reviewers. A big thank you goes to Stefan Wrobel for providing his experience and sound judgment in difficult decisions, to John Langford for organizing an exciting workshop program, and to Ben Taskar for an excellent selection of tutorials. To Alan Fern we owe a great list of sponsors, whom we thank as well. Jennifer Dy made sure travel scholarships were fairly awarded among the student applicants, and Hal Daume III expertly produced these proceedings and the program booklet. Last but not least we would like to thank all authors who submitted their work to this conference.
Our special thanks go to the Local Chairs, Shai Fine and Dan Pelleg, for handling the budget and organizing everything from conference rooms, to hotels, to meals, to pins for putting up the posters. Noam Slonim organized the many volunteers, Oded Margalit handled visa issues, Efrat Maimon and Ettie Gilead took care of publicity and the Web-Site, and Vered Aharon provided local and technical assistance. We are grateful to IBM Research in Haifa for providing the many resources that went into the local organization of ICML 2010.
Many more people spent many hours on ICML 2010 – many more than we could possibly acknowledge in this short summary. But we hope that they found their involvement just as interesting and fun as we did.
Thorsten Joachims
Johannes
Fürnkranz
ICML 2010 ProgramChairs