ICML 2010 Feedback
1. Did you serve as an area chair at ICML 2010?
 answered question560
 
skipped question0
 Response
Percent
Response
Count
Yes
4.8%27
No
95.2%533
2. Did you serve as a reviewer at ICML 2010?
 answered question560
 
skipped question0
 Response
Percent
Response
Count
Yes
34.8%195
No
65.2%365
3. Are you an author of a *rejected* ICML 2010 paper?
 answered question560
 
skipped question0
 Response
Percent
Response
Count
Yes
51.6%289
No
48.4%271
4. Are you an author of an *accepted* ICML 2010 paper?
 answered question560
 
skipped question0
 Response
Percent
Response
Count
Yes
30.5%171
No
69.5%389
5. Are you going to attend ICML 2010 in Haifa?
 answered question560
 
skipped question0
 Response
Percent
Response
Count
Yes
27.3%153
No
72.7%407
viewMy main reason for not attending is:305
6. What model do you prefer for assigning area chairs to papers? (multiple selections possible)
 answered question460
 
skipped question100
 Response
Percent
Response
Count
Authors supply area keywords, and area chairs bid on papers primarily in the areas they agreed to cover. The assignment of area chairs to areas is public to the authors. (this year's model)
75.4%347
Authors supply area keywords, and the system matches area chairs to papers by keyword overlap.
18.7%86
Authors bid on area chairs. (last year's model)
17.0%78
viewI have the following alternative proposal:
3.3%15
7. What model do you prefer for assigning reviewers to papers? (multiple selections possible)
 answered question470
 
skipped question90
 Response
Percent
Response
Count
Two-phase model mixing bidding and manual assignment. In Phase I, reviewer bidding is used to assign two reviewers to each paper. In Phase II, area chairs manually assign additional reviewers based on the Phase I reviews and the author rebuttal. (this year's model)
60.6%285
Two-phase model like above, but area chairs manually assign reviewers in both phases.
17.7%83
Single-phase model, where reviewers are assigned to papers based on their bids.
16.8%79
Single-phase model, where reviewers are manually assigned by the area chairs.
11.7%55
viewI have the following alternative proposal:
5.7%27
8. What reviewing model do you prefer?
 answered question473
 
skipped question87
 Response
Percent
Response
Count
double-blind (reviewers do not know the authors, and vice versa) (as this year)
81.2%384
single-blind (reviewers know the authors, but authors don't know the reviewers)
12.1%57
open (reviewers and authors know each other's identities)
6.8%32
9. Do you think the Phase I reviews were different in quality from the Phase II reviews?
 answered question439
 
skipped question121
 Response
Percent
Response
Count
Phase I substantially better
10.7%47
Phase I slightly better
15.5%68
Both were the same
45.1%198
Phase II slightly better
19.1%84
Phase II substantially better
9.6%42
10. Do you think the ICML2010 reviews were different in quality from the reviews at previous ICMLs?
 answered question369
 
skipped question191
 Response
Percent
Response
Count
ICML2010 substantially better
4.6%17
ICML2010 slightly better
14.1%52
About the same
67.2%248
Previous ICMLs slightly better
9.2%34
Previous ICMLs substantially better
4.9%18
viewAdditional comments:57
11. Do you think the ICML2010 reviews were different in quality from those at related conferences?
 answered question427
 
skipped question133
 Response
Percent
Response
Count
ICML2010 substantially better
11.5%49
ICML2010 slightly better
26.7%114
About the same
45.2%193
Other conferences slightly better
12.2%52
Other conferences substantially better
4.4%19
viewWhat conference are you comparing to and what is the subject area?98
12. What do you think of early rejects by area chairs (i.e., rejecting a paper based on two reviews and author rebuttal)?
 answered question467
 
skipped question93
 Response
Percent
Response
Count
I object to early rejects. Rejected papers should get at least three reviews.
17.6%82
I support early rejects, but only in very clear cases (approximately 13% of submissions this year).
72.2%337
Area chairs should make more liberal use of early rejects.
10.3%48
13. What do you think of early accepts (i.e., accepting a paper after only two reviews and author rebuttal)?
 answered question466
 
skipped question94
 Response
Percent
Response
Count
We should not allow this (as it was this year).
59.7%278
We should allow this for papers with exceptionally good reviews.
40.3%188
14. Do you think the author feedback has an influence on the decision of acceptance/rejection?
 answered question467
 
skipped question93
 Response
Percent
Response
Count
Very strong influence.
3.0%14
Substantial influence.
17.6%82
Marginal influence.
59.3%277
No influence.
20.1%94
15. When do you think should authors be asked to provide their feedback?
 answered question466
 
skipped question94
 Response
Percent
Response
Count
After Phase I of the reviewing process, so it can inform the selection of Phase II reviewers and is immediately available during the discussion. (this year's model)
51.9%242
After Phase II of the reviewing process, so authors can respond to all reviews, not just the Phase I reviews.
19.5%91
Ask for author feedback both after Phase I and Phase II, even though the paper submission deadline would need to be moved to January 15.
14.4%67
Do not ask for author feedback at all.
7.3%34
viewI have the following alternative proposal:
6.9%32
13. Compared to other conferences, how much effort was it to be a reviewer or area chair for ICML2010? (Area Chairs only)
 answered question27
 
skipped question0
 Response
Percent
Response
Count
Substantially more effort
3,7%1
Slightly more effort
33,3%9
About average
51,9%14
Slightly less effort
7,4%2
Substantially less effort
3,7%1
viewComments:7
14. Throughout the reviewing process, was it always clear to you what needed to be done? (Area Chairs only)
 answered question27
 
skipped question0
 Response
Percent
Response
Count
Yes.
63,0%17
Most of the time.
33,3%9
Mostly not clear.
3,7%1
viewComments:5
15. How well do you think did the Phase I reviews and the author feedback inform a good selection of Phase II reviewers? (Area Chairs only)
 answered question27
 
skipped question0
 very importantimportantsome influenceno influenceRating
Average
Response
Count
Phase I reviews25,9% (7)22,2% (6)44,4% (12)7,4% (2)2,3327
Author feedback0,0% (0)18,5% (5)44,4% (12)37,0% (10)3,1927
13. How did you like your assignments in the first reviewing phase? (Reviewers only)
 answered question164
 
skipped question31
 Response
Percent
Response
Count
very interesting
9,8%16
good fit with my expertise
84,1%138
mostly not my area
6,1%10
terrible 0,0%0
14. How did you like your assignments in the second reviewing phase?  (Reviewers only)
 answered question141
 
skipped question54
 Response
Percent
Response
Count
very interesting
12,8%18
good fit with my expertise
80,9%114
mostly not my area
6,4%9
terrible 0,0%0
15. How would you like to receive your assignment?  (Reviewers only)
 answered question170
 
skipped question25
 Response
Percent
Response
Count
An area chair who knows me should assign them to me
12,9%22
I want to bid for my papers
32,9%56
We should try a mix of both models
51,2%87
Don't care
2,9%5
16. Did you look at the supplementary materials of the papers?  (Reviewers only)
 answered question171
 
skipped question24
 Response
Percent
Response
Count
always
22,2%38
sometimes
33,9%58
never
8,8%15
my papers didn't have any
28,7%49
I don't know whether my papers had any
6,4%11
17. As a reviewer, how did you rate the author feedback of your papers?  (Reviewers only)
 answered question168
 
skipped question27
 Response
Percent
Response
Count
it changed my mind about a paper
7,7%13
it helped to clarify a few questions, but did not change my mind
75,0%126
it was not useful
14,9%25
I did not see it
2,4%4
18. Throughout the reviewing process, was it always clear to you what needed to be done? (Reviewers only)
 answered question170
 
skipped question25
 Response
Percent
Response
Count
Yes.
66,5%113
Most of the time.
32,9%56
Mostly not clear.
0,6%1
viewComments:7
25. Do you think the majority of the reviews of your paper were (Authors only)
 answered question341
 
skipped question95
 Response
Percent
Response
Count
right to the point
13,5%46
high quality
53,1%181
low quality
24,0%82
the reviewers did not understand my paper
9,4%32
26. How do you rate the length of the majority of the reviews? (Authors only)
 answered question343
 
skipped question93
 Response
Percent
Response
Count
too long
2,3%8
just right
77,8%267
too short
19,8%68
27. Do you think the reviewers read your feedback? (Authors only)
 answered question348
 
skipped question88
 Response
Percent
Response
Count
yes, and they took it into consideration
14,7%51
yes, but they ignored it
18,4%64
I'm not sure
59,2%206
certainly not
7,8%27
28. Do you think that the meta reviewer (area chair) adequately summarized the reviewers' opinions and made a right decision? (Authors only)
 answered question336
 
skipped question100
 Response
Percent
Response
Count
yes
39,3%132
somewhat, but mostly OK
38,7%130
no
17,0%57
there was no justification of the decision
5,1%17
29. How do you rate the length of the meta review(s)? (Authors only)
 answered question332
 
skipped question104
 Response
Percent
Response
Count
too long
0,9%3
just right
72,6%241
too short
26,5%88
30. Will you submit again to a future ICML conference? (Authors only)
 answered question348
 
skipped question88
 Response
Percent
Response
Count
yes
76,7%267
no
2,3%8
don't know yet
21,0%73
31. Which conference tool do you like best?
  answered question 288
 
skipped question 272
  Response
Percent
Response
Count
CMT (this year)
83.0% 239
CyberChair
12.5% 36
START
4.5% 13
viewOther (please specify) 36
32. Did you find CMT
 answered question411
 
skipped question149
 Response
Percent
Response
Count
great
9.2%38
comfortable
39.9%164
o.k.
42.3%174
clumsy
6.6%27
awful
1.9%8
33. Do you have any suggestions for improving CMT that we should pass on to its developers?
 answered question53
 
skipped question507
 Response
Count
view53
34. With ca. 160 accepted and invited papers, the program currently fills 5 parallel tracks on three days to the limit. Should we allow the conference to grow further?
 answered question401
 
skipped question159
 Response
Percent
Response
Count
No, we already accept too many papers.
15.7%63
No, we should not accept more papers even if we get substantially more submissions.
25.7%103
Yes, but include accepted papers that will only be presented as posters.
31.9%128
Yes, add another parallel track.
11.0%44
Yes, add another day of sessions to the program.
15.7%63
viewOther suggestions?17
35. This year, we invited 7 application papers, which got a free entry to the program and a slot in the proceedings. Do you think this is a good idea?
 answered question424
 
skipped question136
 Response
Percent
Response
Count
Yes, we should keep that.
15.8%67
Sounds good in principle, but I have not yet seen the papers.
52.6%223
No, application papers are fine, but we do not need additional incentives for attracting them
18.6%79
No, we should find another way for attracting application papers.
13.0%55
viewOther comments?24
36. At this year´s conference, proceedings on paper, USB or CD will *not* be available, but papers can be downloaded from the Web page. Does this work for you?
 answered question436
 
skipped question124
 Response
Percent
Response
Count
Yes.
89.0%388
Yes, but I think it's a bad idea.
7.6%33
No.
3.4%15
37. As in previous years, printed proceedings will be available from Omnipress (this year only after the conference). This service is not free (about 6$ per registration). Do you think we should still produce printed proceedings for archival purposes?
 answered question430
 
skipped question130
 Response
Percent
Response
Count
Yes.
35.1%151
Yes, but try to find a cheaper way.
11.6%50
No.
53.3%229
38. What other conferences do you regularly attend?
 answered question337
 
skipped question223
 Response
Percent
Response
Count
NIPS
68.2%230
AIStats
19.3%65
UAI
19.3%65
COLT
11.6%39
ECML/PKDD
31.5%106
SIGKDD
18.4%62
ICDM
12.5%42
IJCAI
20.8%70
AAAI
22.8%77
ECAI
6.8%23
EMNLP
6.2%21
CoNLL
1.2%4
SIGIR
7.7%26
viewOther (please specify)61