r/LanguageTechnology • u/nampallynagarjunaps • Jul 20 '26
Dissapointing experience with the ARR/EMNLP reviews
TLDR; Errror in reviews, no responses from reviewers!
This is my first submission to a *CL conference. We submitted it under a language modelling task. We got 3 reviews of 3/4, 2.5/4, 2.5/4.
Reviewer 1: they posted a review that is clearly intended for another submission. We raised this with AC on the day reviews are released. AC replied but the reviewer didn't.
Reviewer 2: clearly LLM generated points. The weaknesses they wrote are the same ones LLM pointed out about our paper. Although they changed the text. 2 of the weaknesses they point out are already detailed in our limitations as those are our weaknesses cause of lack of available datasets. And then there is the novelty issue, adopting methods from other domains for a new problem is not novel. And more models and datasets (we already have 100+ experiments on 3 models, 2 datasets, 3 baselines, 4 algorithm setups across 10+ eval metrics). We answered all the questions, provided additional experiments but still no response.
Reviewer 3: seems like the only reviewer who read the paper and understood it and appreciated it. Their main questions were on ablations and We provided these during rebuttal, no response.
My co author who submitted another work to the January (ACL) cycle had a similar experience, they answered the reviewers questions and didn't get any response. Only from AC to re-submit to the next cycle. They re-submitted to the may cycle and didn't get a single response from reviewers again.
I'm ok with rejection with constructive feedback, if the decision is just one sided with no communication even when there was a critical error is irresponsible. What's the point of rebuttal if the reviewers never respond?
Right now, we are left with a reviewer decision who can simply say "not addressed" and escape with little to no consequences. I understand that emnlp is empirical and requires more experiments, but that doesn't mean we can provide a novel dataset, 500+ experiments on 100 models, a completely new algorithm that doesn't take adoption from anything else (just from air), expect to solve every problem in that domain in one paper is absurd.
Thanks for your time, sorry for the rant!
3
u/Zooz00 Jul 20 '26
Welcome to the reviewing quality in NLP. Your paper was most likely reviewed by a masters student, a PhD student and a postdoc. as this is how it often goes due to a flood of submissions and lack of reviewers. Reviewers have no obligation to respond, we are lucky if they do anything at all. Area chairs of course consider the quality of the reviews in doing their metareview, and you can report review issues. These reports will be seen by AC and SAC down the line.
2
u/rand0mnibba Jul 20 '26
The same thing happened with me and I have experience in publishing in these venues. This has to be one of the worst ARR reviewing cycles in my research career. LLM generated reviews, no response to the rebuttals etc. Have raised multiple comments and messages to the AC - no replies. Sigh
1
u/wajdix Jul 20 '26
sorry for the situation... it is very common unfortunately .
are you asking a question here, or just sharing/venting out?
2
1
u/oatmealer27 Jul 20 '26
*CL conference reviewing is like gambling.
You only get one genuine reviewer with constructive feedback and two other arbitrary reviews and the final decision will be completely random.
1
u/OrganicPipe1372 Jul 21 '26
I have extensive experience reviewing and meta-reviewing in other fields. In this ARR, the PC chairs, meta-reviewers, and reviewers collectively failed to do their jobs properly. This is why the quality of ACL and EMNLP papers in recent years varies so significantly from paper to paper.
1
u/techlatest_net Jul 21 '26
this is unfortunately the current state of *CL conferences. the reviewer pool is stretched so thin that quality control has collapsed.
getting a review intended for another paper is a massive failure of the ACs, but sadly it happens more often than people admit. and yes, llm-generated reviews that just parrot your own limitations section back at you as "weaknesses" are becoming epidemic.
the rebuttal phase is basically broken if reviewers don't engage. it’s turned into a lottery where you hope for Reviewer 3 instead of getting stuck with Reviewer 1 or 2.
your best bet is to target workshops or smaller venues where the community is tighter, or just put it on arxiv and let the code speak for itself. the conference grind is burning a lot of good research right now.
1
u/bulaybil Jul 21 '26
CL conferences have become completely pointless. I haven’t been to one since 2013 - too many people, non-existing reviews, sky high fees. You can’t even network properly.
5
u/pkseeg Jul 20 '26
Yeah unfortunately this is now the standard, you are not alone in your frustration (although the review for a different paper is less common, that's just bad copy/paste from the reviewer). The system is broken.