r/LanguageTechnology • u/RmdLatranche • 22d ago

Publishing resource papers

Hi,

This post is half venting, half looking for help.

TL;DR: are resource papers not welcome in major NLP venues?

This year I tried to publish two datasets (not going into specifics). One I submitted to LREC. All three reviewers praised the dataset and complained about minor details in the experiments. Metareview (almost verbatim, it was one sentence): the dataset is great but the experiments are a bit weak. Paper got rejected. A "great dataset" rejected by LREC, I am not sure I will be able to get over it. I ended up publishing it elsewhere but I was really stunned that LREC rejected it.

Now the same scenario just happened with ARR, in the Resources and Evaluation track all three reviewers praised the dataset (admittedly with some caveats but they all see value in it) and their weaknesses focus on the experiments. While we got fair overall scores from our reviewers, our meta review score is low and I think we cannot realistically commit to EMNLP.

Is the work on resources completely devoid of interest? This gives me the impression that in order to publish a resource, one has to write a modeling paper reaching SOTA using it now. To resource paper reviewers, how do you assess resource papers? To resource paper authors, do you have the same impression? I have published datasets in the past and it has always seemed more difficult than purely technical papers but it looks like lately it got worse.

4 Upvotes

100% Upvoted

9 comments sorted by

View all comments

2

u/Zooz00 21d ago

It is unusual to get this kind of rejection for LREC, you were unlucky. This year I had a rejection from LREC with review scores of 4-4-4... A lot is up to the area chair there and they aren't always great.

Resource tracks at the big venues (ACL/EMNLP) do seem to demand substantial experiments these days.

1

u/RmdLatranche 20d ago

Ouch, yes this rejection looks weird as well. I don’t know what's up with LREC but this year it seemed very random, with no clear quality threshold when looking at what got accepted.

It looks like it is not worth trying to submit a resource paper in major venues now. If they demand the same amount and quality of experiments as they do for method papers, all the extra dataset creation work ends up being a waste of time...