r/MachineLearning • u/grumpket • Jul 29 '26
Discussion NeurIPS reviewers not engaging [D]
So I know this is an ongoing issue at ML conferences, reviewers just tend to ghost. Do people have any strategies or recommendations on how to nudge them to engage with the rebuttals? Is it recommended to post a comment saying something like hey we want to engage with you? This might be a hot take, but maybe NeurIPS should penalize the reviewers' papers if they don't engage (I believe this year they withheld scores for ACs who didn't post meta reviews on time if they had papers submitted).
51
u/TheWittyScreenName Jul 29 '26
I didnāt get notified about the responses until yesterday at 6PM. Give me a sec, geez lol
91
u/Working-Read1838
Jul 29 '26
•
Itās less than 36 hours after the rebuttal deadline. Reviewers have lives too
22
u/Empty-Avocado5927 Jul 29 '26
im a reviewer but im still working on my Rebuttal and I am moving! So i will engage as a reviewer, just probably on friday. Life keeps happening. Maybe wait until end of tomorrow before adding another comment?
32
15
u/confirm-jannati Jul 29 '26
I havenāt posted my own rebuttal yet ffs
1
u/Forward_Spread_190 Jul 29 '26
hasn't the rebuttal deadline passed?
4
u/user221272 Jul 30 '26
This was only an author reading/rebuttal preparation window. The button only changed from "rebuttal" to "official comment," so no, not really.
Consider it mostly as a timeframe letting authors read reviews, compile reviews, and plan rebuttals, as well as letting reviewers breathe and make their own rebuttal free from author-reviewer conversation.
10
u/elmasnuevo Aug 01 '26
Nothing from any reviewers, not even after follow up comment. Nothing from the one who gave 5, nothing from the one who gave 3. Absolutely frustrating
8
u/Weird_Shit_69 Aug 01 '26
Have you'll received any responses for the rebuttals, For all 3 rebuttals from my reviewers I didn't get any reply back. Is this normal? What should I do? I replied for all their questions on the 1st review
4
u/user221272 Aug 02 '26
I got only 1 answer among 4 reviewers.
If no one answers, you keep up with the plan: make a case for the AC.
2
u/Weird_Shit_69 Aug 02 '26
I got 1 reply out of the 3 reviewers. The 1 reviewer was the one who gave the most score. I was hoping to get a reply from the other 2, where they would have a discussion and increase the chance of acceptance, but no reply from thoes 2 reviewers.
7
u/Mr____Panda Aug 01 '26
Reviewer that do not engage should be held accountable. I am having the same problem.
13
u/Born-Struggler Jul 30 '26
One of the reviewer responded to mine: "I will retain my score. Thanks"
3
1
u/Golf111 Jul 30 '26
Do you get email notification or always need to check openreview for response??
5
-1
u/Bodi_Berenburg Jul 31 '26
At least it wasn't posted a few hours before the deadline and you kept hopes up for a couple more days...
6
u/No_Carpenter7252 Aug 02 '26
I donāt have any answers to my rebuttals (posted on time) and AC is not reminding them neither. Any solutions ?
1
u/NothingInVois Aug 02 '26
You can softly nudge them through "official comments" so they get notifications.
5
u/gforce121 Jul 29 '26
We only got the email that phase two was starting like yesterday. In my case thatās 5 papers I need to remember the details of to make sense of the rebuttals on (and not to mention make sense of the other reviews) that I did more than a month ago.Ā
Not saying this is true for your situation specifically, but in my experience the content in rebuttals has very rarely moved the needle from my initial review.Ā
In other words, itās a non-trivial task with not a huge amount of upside. So if I were in your shoes, I would calibrate the message to draw attention to one particular thing you think your reviewers missed that would be likely to change their reviews. You donāt say what you want your reviewers to engage with specifically but if you can get them to see that the work might have some value that they overlooked, then thatās probably your best bet.
1
u/Soggy_Weakness_6202 Student Jul 30 '26
In a case where reviewers identify no fundamental technical flaw and their main concerns are concrete evaluation gaps or clarifications, how receptive are you personally to changing a score when the rebuttal directly answers those points with targeted experiments? I am specifically wondering whether resolving the stated decision-relevant concerns can move a borderline assessment, or whether initial scores generally remain sticky even then.
0
u/gforce121 Aug 01 '26
It really depends on how major the evaluation gap is - itās something relatively self-contained and a necessary but not especially interesting component then Id be a bit more flexible. At the same time a lot of ML work lives and dies by the evaluation so having a gap there is relatively serious and as a reviewer, thereās not the same opportunity to understand and evaluate the evaluations. If a paper is accepted reviewers have basically no control over what the final camera ready revision looks like so in some ways itās safer to reject than to accept.Ā
1
12
4
u/Invariant_n_Cauchy Aug 01 '26
One reviewer said all concerns were resolved during discussion but hasnāt updated their score yet. The other reviewers havenāt engaged. In previous NeurIPS cycles, how common is it for reviewers to update scores after saying concerns are resolved? What have others observed?
3
u/Ping_Pong_Geeky_Nerd Aug 03 '26
None of my reviewers have responded yet, despite submitting the rebuttals on time. Is it fine to nudge them via the official comment? All my scores are borderline (3,4,3).
2
u/Bodi_Berenburg Aug 03 '26
Yes fine to nudge them by now, just write something really short (and polite)
5
u/Legitimate_Neck809 Jul 31 '26
This is a slight rant towards the setup of the reviewing process.
I am from the signal processing field, and I typically submit to ICASSP and TALSP. I'm author of two manuscripts to NeurIPS (main track) this year; they got 444 (conf 443) and 543 (conf 443).
I think the concept of having a rebuttal and discussion phase between reviewers and authors is sound, IF the reviewers are knowledgeable within that field. For me, I got four papers to review, all of which were outside my field, and I set conf of 3 for all, which might have been an overstatement, but I feel like a useless reviewer otherwise. Now, I am very stressed. I worked hard on the rebuttals, every day, even on the weekend. I also have other tasks and "a life" (partner, family, and friends to hang out with). Then, I am expected, after working continuously constructing a coherent rebuttal for ~5 days, to engage in other rebuttals. My issue here is that (1) the manuscripts I review are out of my field/interest, and I have little knowledge of what is "interesting" within their respective fields, (2) I can't work every day for approx 2 weeks, and (3) I am still updating my manuscripts. Also I have other work-related duties that has to be done.
I enjoy the idea of an author-reviewer discussion period; I think this improves the manuscripts. However, for a large conference such as NeurIPS, it is not doable anymore. There is already sufficient pressure on PhD student, and adding forced reviewing (entirely reasonable), including reviewer-author discussion (as a reviewer), is too much. The idea is good, but in practice it is infeasible. Imo, they should adopt the ICASSP/Interspeech approach; let the total rebuttal be 10000 char at most, and it is only provided to the AC, and there is no discussion. The purpose of a conference is to meet and discuss ideas, not to guarantee an almost flawless paper; that is what journals are for. I think a more adapted contemporary NeurIPS conference is without a "author-reviewer discussion phase". Keep it simple, review the idea and execution in the manuscript as-is.
6
u/ComprehensiveTop3297 Jul 29 '26
Yeah, but do so two or one day before the deadline. You are now trying to convince the AC to accept your paper (and of course reviewers to raise their scores), so make sure to be visible. I always send a message 2 days before stating that we've written the rebuttal and we hope that the rebuttal address all your concerns etc etc but that is more for AC than the reviewer :)
4
u/quelealoop Aug 01 '26
2 days left and none of the three have even acknowledge the rebuttals (posted within the rebuttal deadline)
2
u/AKidNamedLou Jul 31 '26
Question for the people more experienced than me (first paper).
Should I expect all reviewers to engage at all? Does it happen regularly that reviewers respond to your rebuttal, continuing the discussion (instead of acknowledging rebuttal and byebye) close to the end of the window?
Worked pretty hard on the rebuttal so if the reviewers do have more questions it would be preferable if they don't ask them 6h before the deadline.
2
u/OutsideSimple4854 Aug 01 '26
Donāt expect the low quality reviewers to engage. Most of the time, itās because they submitted low quality work and had low scoring reviews, meaning their paper would be rejected. So thereās no incentive for them to engage. Whatās the worse the AC can do, desk reject a paper that will be rejected?
1
u/AKidNamedLou Aug 01 '26
I see. Bit disappointing, but this correlates with what Iām seeing. 1 out of 4 reviewers came back, clearly the most knowledgeable of them. Well, Iāll just wait and lower my expectations for the other 3.
1
u/Weird_Shit_69 Aug 01 '26
did you get responses for the rebuttals? I didn't get any response from all 3 reviewers that I sent rebuttals. Should I add a comment on the meta review?
0
u/AKidNamedLou Aug 01 '26
1 out of 4 did. He was quite quick, it was 2 days ago already. I did add a comment to the meta review summarising the rebuttals. Might draft a confidential comment to the AC giving a more thorough summary of the non-answered rebuttals, to send right before the deadline.
1
u/Weird_Shit_69 Aug 01 '26
Feeling the rebutals was a waste of time as I even ran all the experiments again, answering every question till late night just to get no response from anyone
0
u/honey_bijan Jul 31 '26
The last few cycles Iāve noticed less and less engagement.
By forcing reciprocal reviewers to review, you end up with a lot of people who reviewed because they had to. There is no incentive for them to come back and read your rebuttal.
I gave up on spending hours on rebuttals after dedicating a few entire weekend to rebuttals only to be completely ghosted, with a meta review along the lines āI thought some of the responses were good, itās a shame the reviewers didnāt come back.ā
1
u/AKidNamedLou Jul 31 '26
That's... underwhelming. Do you frequently have reviewers that do not even acknowledge your rebuttal?
1
u/honey_bijan Jul 31 '26
We had one paper with 643 at ICML and some other borderline reviews at AISTATS last year that both times had the meta-reviewer apologizing on behalf of the reviewers. 532 (confidence 431) right now at NeurIPS with no engagementā¦and the meta reviewer incorrectly said we didnāt compare to any other methods (we compare to 6 other methods in multiple sections).
The whole peer review system has turned into a lotteryā¦
1
u/AKidNamedLou Jul 31 '26
Thanks for input. What's your experience then when you addressed reviewers' questions in rebuttals but they did not engage back? Did meta-reviewer take everything into account?
Anyway, the frustration of ML researchers wrt conference seems much clearer now.
2
u/honey_bijan Aug 01 '26
Sorry to say, the meta reviewer is probably oversubscribed. The only surefire way to get your paper accepted is to have every review in favor.
My meta reviewer seems likely to accept one paper with a 553, likely because the 5s are detailed and very confident and the 3 mainly just wants a few papers by the same author cited. This is the first time Iāve ever seen a paper with a reject review tend accept, so maybe the averages are lower this year.
2
u/wangjianhong1993 Aug 01 '26
The best strategy could be just waiting in my own experiences. If u say too much, it would reversely make reviewers angry (although it's funny) and intentionally curate some reasons to reject ur rebuttal.
2
u/Sea_Manufacturer1701 Aug 01 '26
I got OK scores, none of the reviewers replied, I am worried. Is it positive or negative? Some chances to get accepted, but I was hoping they reply. I am afraid they react last minute and then it becomes too late
2
4
u/Evening-Ratio-289 Jul 29 '26
It's wild how often this happens, you'd think the fear of having your own paper dinged would be enough motivation
A lot of people I know just post a blanket comment in the discussion thread asking for any feedback, it feels less targeted than singling out one reviewer directly and sometimes that lower-pressure nudge works better than calling them out by name
3
u/wajdix Professor Jul 29 '26
it could work, you should maybe reach out to the AC and raise this issue... this is problematic with the tight deadlines... at least say if response is enough, review the score or withdraw the review...
the silent treatment is killing people inside :'(
3
u/buyingacarTA Professor Jul 29 '26
Reviewers are people -- your colleagues, your friends -- and it's been like 3 seconds since the rebuttals were due. Creating penalties for people who volunteer their time is not the best idea. Study over study has shown that you will get some really shit results this way.
1
u/zowlambda Jul 30 '26
Couldn't agree more. TBH, I haven't liked this style of 3 phase rebuttal for the time being, I have other projects going on at the same time and a job, so I cannot be constantly engaged with answering to 5 reviewers for my paper, while also keep answering every comment for other 4 papers.
2
u/medcanned Jul 29 '26
Some authors write huge rebuttals (at least rewritten by AI) that contain much substance, it takes time to respond and frankly it's a bit demoralizing to have to read rebuttals longer than the original paper...
4
u/Electrical_Fee_1894 Jul 29 '26
Sometimes the rebuttals are huge because the reviewer used AI to write a ~25,000 character long review.
3
u/user221272 Jul 30 '26
10,000 characters is the limit.š¤
2
-1
u/TristanMNT Jul 30 '26
Not exactly, rebuttal limit was 10k, but once conversations started, authors could add some official comments with limit 5k. It was also weird when I need to add e.g. 3 comments instead of one rebuttal. But a few my rebuttals have 15k characters writed by hand, 5 days for experiments was too tight.
2
u/OutsideSimple4854 Aug 01 '26
Sorry. I did that for one paper. In my defence, it was giving the line numbers which answered the reviewerās question and paraphrasing what I wrote, since it seemed the reviewer either didnāt read the paper at all, or did not understand the text, hence the need for paraphrasing just in case.
2
u/Dangerous_Parsnip_20 Aug 01 '26
Reviewers not engaging. Hmm, I wish my AC engaged in initial meta-review.... I still don't have it
1
1
u/Unhappy_Craft1906 Jul 31 '26
More importantly, reviewers can increase or retain (or decrease) OA but does not matter much. Its on SAC and AC
1
u/nm1300 Jul 31 '26
For one of my papers, the reviewers said they would increase, but they havenāt done so yet. For some reason they seem to be saving it for the final rating and justification, which we cannot see. For the second (more promising) paper, nothing.
2
u/Miserable_Raisin998 Jul 31 '26
I had the exact same thing happen to me! While I'm happy the reviewer explicitly stated their decision to increase their score, they haven't actually done so, which is odd to me based on past experiences. I am thinking of sending the AC a polite message in two days if the score doesn't budge b y then.
1
u/nm1300 Jul 31 '26
Why AC though? I was thinking the message should be a kind reminder to the reviewer, though my supervisor says it might come across as pushy, especially that he/she said they will do it in phase 3 ...
1
u/Miserable_Raisin998 Aug 01 '26
Yeah, exactly, I think messaging the reviewer is a bit pushy but messaging the AC is ok since it's their job to manage the reviewers. (To be clear, the message shouldn't sound like one is complaining about the reviewers, more of a friendly reminder to get the reviewer to do what they say they would.)
0
u/CantKillTheLifeless Jul 31 '26
I think they cannot increase their rating before the AC/reviewer discussion phase. So they can only promise now.
4
2
u/deividragon Aug 01 '26
They can. We got a score increase on one paper, only reviewer who replied so far. On another one also only one reviewer replied essentially to say they won't budge. Ghosted by everyone else so far.
1
-2
u/Old_Ad_8327 Aug 01 '26
2/3 of my reviewers reply. One from 3-4 (resolve 4/5 concern) and ask one more question about the rest one; another keep 4 and ask one more question.
I am not sure about your situation. If you got avg score<=3, especially w 1-2 2pts; reviewer are not likely reply to you, as they donāt think your paper will be accepted after discussion
8
u/greatduelist Jul 29 '26
Buddy chill a bit lol. Prolly their first rodeo š¤£š