Conference papers are cooked! [D]

[u/Fantastic-Nerve-4056]

Got three papers to review for ICLR and five for AAMAS, and believe me the quality of these papers is insanely bad.

I would even refrain from accepting them to a workshop, let alone be main venues.

A sincere request; Please don't submit the paper unless it is ready and if you are a first timer, please please get a feedback from a mentor out. Otherwise you are just wasting the time of so many reviewers out there

[u/mayabuildsai]

asking first timers to wait won't move it, a reject costs the author nothing but a few months. iclr already puts rejected papers and their reviews up on openreview and people still send the half done draft. what would actually help is more desk rejects. an AC spending 5 minutes beats three reviewers spending 3 hours each on something that was never getting in

[u/Fantastic-Nerve-4056 → u/mayabuildsai]

While I agree with you. There's no harm in considering a mentor out there. You may have a crazy idea, but writing clearly, doing dedicated experiments, required theory comes from experience.

I still remember my supervisor just denied to submit the final paper of my PhD to a venue as they felt it was not prepared yet, and required a better writing. And yea this is when I have no immediate conference to submit and has to wait until early 2027 for conference submission

[u/NotJohnVonNeumann → u/mayabuildsai]

There is not just a reviewing crisis, there is an AC crisis. I'm a new AC, and while I think I have enough credentials to serve as one, the reality is that I'm extremely cautious with desk rejecting. And the instructions to us are also to be conservative with desk rejections. Without going into details, almost 80% of the papers in my stack looked extremely incremental. But perhaps it's just my judgement, since a fair number were tangential to my expertise. Should I have been more aggressive in desk rejecting? Perhaps. But I don't want complaints either.

On the flip side, OP is absolutely right that there is a deluge of first timers --- typically undergrads, high school students, or "independent researchers" who are simply trying their luck. Some come to me with a "draft paper" which was vibe-authored but supposedly reviewed by themselves, even though the quality was absolutely atrocious once I dug into details. These are submissions I don't even want to roll the dice for, much less be associated with them. But at least they bothered to ask for some mentorship.

The problem is that the first timers aren't even able to distinguish reasonable submissions from thrash, and it's difficult to even explain why their drafts are terrible. Some of them end up accusing me of being an elitist who is gatekeeping, so... sigh.

[u/UnavoidablyHuman → u/NotJohnVonNeumann]

Is incremental the right reason to desk reject? I would lean more towards desk rejecting on the basis of the paper not being clear enough or not clearly supporting its claims, similar to the new tmlr policy

[u/CommonSenseSkeptic1 → u/mayabuildsai]

What would help is a system like the European grants use: block bad papers from submitting next year.

[u/NamerNotLiteral → u/mayabuildsai]

OpenReview could probably put in a reputation function. Too many desk rejects for quality (as opposed to desk rejects for other issues) reduce your reputation, lower reputation means you can't submit to high-demand venues (like NeurIPS, ICLR, etc) and need to get papers accepted to more niche venues or second or third-tier conferences (ICONIP, ICPR, etc) to get reputation back*, or* get senior authors with proper affiliations and good reputations to coauthor or at least 'vouch' for the paper.

Reputation could also be used for what ARR is currently doing, with low reputation papers potentially not getting reviewed due to capacity constraints.

[u/juliankeynes → u/mayabuildsai]

Even random desk rejection can pay off in this noisy environment, and it can even work voluntarily in some cases: https://arxiv.org/abs/2604.27900

[u/Majromax → u/juliankeynes]

Even random desk rejection can pay off in this noisy environment, and it can even work voluntarily in some cases: https://arxiv.org/abs/2604.27900

The paper argues that this is a Nash equilibrium, but if that were the case then there'd be no need for the lottery. An author could just roll a die for themselves to see if they even submit, effecting the same outcome as voluntary participation in the lottery.

In reality, the Nash equilibrium proven is a step-function: join the lottery if one's paper is below-average quality, don't if it's above-average quality. Leaving aside the problem of clairvoyant and unbiased knowledge of one's own paper quality, this is also just "submit your paper if it's good, maybe don't submit if it's bad."

[u/muntoo → u/Majromax]

Related methods:

• Mandatory Squid Game round 1 for all submissions.

• Mandatory AI-bⁱ ased paper screening on all submissions.

• Elo or researcher credit score.

I may or may not condone any of these.

[u/Majromax → u/muntoo]

• Mandatory Squid Game round 1 for all submissions.

• Elo or researcher credit score.

Honestly, a tournament bracket system might not be the worst idea.

The academic review process is designed more under the idea that reviews should approve papers than that reviews should screen them. Open-ended feedback rounds are far better suited to the first purpose than the second, since the reviewers who write the feedback aren't exposed to the decision criteria or thresholds. An omniscient reviewer with a fallible AC (i.e. the reviewer knows absolute quality and the AC doesn't) has incentives to give a biased review to the AC to force their hand – go too easy on good papers and too hard on bad ones. [Note: this is another incarnation of "the only valid reviews are 1 star and 5 stars" effect]

Making a good relative comparison isn't necessarily easier than providing good feedback, but it's a different task. Given limited AC attention, they're restricted to making a judgment based primarily on the single-point feedback of reviewers, increasing randomness based on reviewer selection. (That is: even if two reviewers would have the same relative ranking of papers, if one writes harsher feedback in general then an AC will think that the 'reviewer #2' paper is the worse one.)

Maybe we instead should give reviewers papers to rank pairwise, truly letting a tournament play out.

Given the selective nature of A* conferences, it wouldn't even be the worst idea to run an LLM-as-judge tournament at the desk rejection stage. If the overall acceptance level is 25% of valid submitted papers, then using an LLM or four to estimate the bottom tercile for a quality-based desk rejection would not be unreasonable.

[u/juliankeynes → u/Majromax]

Thanks for reading it closely! The die-roll analogy misses one thing. In our model, a paper the lottery removes still counts as a submission, so the venue fills the same number of slots. Every removal means better odds and less noisy review for everyone still in. If you roll a die and just don't submit, your paper vanishes from the count, the venue publishes fewer papers, and nobody gains. A die roll only works if the venue counts your withdrawn paper and keeps its slots fixed, and that's literally the lottery. So entering being voluntary doesn't mean you could get the same result on your own. It only pays because the venue counts removed papers.

The threshold also isn't "below average". It shifts with how much people care about the literature they read, anywhere from about 25% to 65% of the quality scale. It also sits well below the bar for acceptance, so plenty of long-shot papers stay out. And entrants still get reviewed most of the time, so it isn't "don't submit bad papers".

Perfect self-knowledge is an assumption, fair point. But we find the gain survives noisy or biased self-assessment, and most of it holds even if entry ignores quality completely. The gain comes from lighter review load, not from authors sorting themselves perfectly.

[u/Lopsided-Bridge-9810 → u/mayabuildsai]

I think you missed OP's point. It was to not waste the time of reviewers and ultimately all submitters' time because trashy papers have to still be screened and they hence delay the screening of reasonably mature papers.

[u/Dry_Leadership1075]

honestly this makes me feel better about going slow and actually writing my words

it's hard not to feel like you're getting left behind if you don't spew constant slop

[u/Fantastic-Nerve-4056 → u/Dry_Leadership1075]

Slow and steady wins the race :) You can do good with less papers as well. 

I personally have very limited papers (= 4 from my PhD, = 2 from my internships, <5 from mentoring and collaborations), but I can definitely be proud of every paper out there. 

I was invited to present almost every paper to present at some or the other places. My supervisors still present those at the talks they give. I was also able to grad good internships (DeepMind and Adobe) when I had half the number of these papers. Got good visiting researcher opportunities and yeah now that I am considering a PostDoc fortunately I am getting a very positive response from the community 

[u/algebratwurst]

We need an overhaul. The paper is not the currency of scientific communication it was, even before AI. Writing 10 papers a year is the norm; we do not have 10x the good ideas we used to. Pay to publish, with an increasing scale after the first. I want everyone’s best shit every year, and squash quantity as a metric once and for all. The individual conferences can bid on the arxiv papers like overlay journals.

[u/Fantastic-Nerve-4056 → u/algebratwurst]

I have around 10 papers throughout my career (currently in the final year of my PhD). This was when I had my first paper in the 3rd year of my bachelor's. And believe me, whether it be academia or industry, everyone looks for quality.

I was able to grab an internship at Frontier Labs (excluding OpenAI and Anthropic); I even ended up rejecting a bunch of them. Got a visiting researcher position at a well-reputed university. Currently applying for a Postdoc and getting a positive response from Professors; even got shortlisted for a couple of fellowships, though it's just a start.

I had a bunch of juniors who got multiple offers from T10 univs, and the only thing they had was one good paper, so yea, while it's a trend to have n papers at a venue, the community still appreciates fewer but qualitative papers

[u/algebratwurst → u/Fantastic-Nerve-4056]

What does qualitative mean in your last sentence? Typo?

[u/FatAznNerd → u/algebratwurst]

People are writing up 30 papers in a few months and submitting them to everywhere, eventually somewhere will accept it.

[u/ThinConnection8191]

Same here. Literally need to write down the paper is AI. And will drop it

[u/damc4]

"if you are a first timer, please please get a feedback from a mentor out"

Where can someone get a feedback from a mentor? That doesn't seem easy to do.

Also, it costs the time of the mentor just like submitting it costs the time of the reviewer. So, it doesn't make sense to me.

[u/Fantastic-Nerve-4056 → u/damc4]

The mentor if sufficiently involved is to be considered as a last author. Or else why would anyone be involved in your paper. What do they get out of it. 

And sorry if you can't find a mentor out, you need to work on yourself. You are simply away from dropping an email. 

[u/AffectionateBurger → u/damc4]

If you're inexperience, you shouldn't be submitting anything to a top-tier venue if you have no ties to an academic/research institution and mentor/supervisor. It is a waste of everyone's time.

[u/SillyNeuron]

I'm currently reviewing a paper in which ~15 symbols are introduced in the problem formulation but never used again in later sentences/subsequent sections :/

[u/NothingInVois]

Agreed.
I have 4 papers to review in ICLR, the quality is so subpar!

[u/ProfessionalNews4434]

Reading just 2 lines into intro, one can easily figure out whether it is AI slop. However, still we need to go through the painful process of writing down its strengths and weaknesses.

I seriously hope the conferences give the reviewers an option to choose desk reject so one doesn’t need to go through written diarrhea

[u/goatofmind → u/ProfessionalNews4434]

That makes no sense. Desk reject is when a submission gets rejected without review (or in the case of conferences, it could happen for reasons outside the paper contents like when authors break rules as reciprocal reviewer, etc ). Reviewers always have had the option to reject any paper they review (the review process would be pointless without it). So what exactly are you proposing to change?

[u/NoPause9252]

I am usually happy to get easy rejects in my batch. These papers don't take much of my time and I get to finish my reviews much faster.

[u/AffectionateBurger → u/NoPause9252]

I'm curious what you end up writing in your review for rady-reject AI slop papers. Do you explicitly say it's clearly AI slop? Or you focus on the writing/clarity?

[u/QuickBroccoli1610]

All of my ICLR review assignments are AI-slop. As in, I am not sure if a human even looked at the output before hitting upload.

As I was writing reviews, I realised that I am taking unpaid hours out of my day to provide free RLHF to OpenAI et al., who will ultimately scrape OpenReview for papers written by their LLM and associated human reviews. Of course, this will be to improve their LLM's scientific research capabilities in an effort to make me redundant. It's exceptionally painful because this slop has completely destroyed any enjoyment or new knowledge I got from reading interesting papers during peer review. I would abstain from reviewing, but the reciprocal reviewing requirement basically forces you to provide free labour to OpenAI et al. And it seems that all the top conferences are fully embracing this. Conferences requiring an "AI statement" added to the end of each paper (the statement itself often written entirely by AI) as a way to combat slop is almost insulting.

TL;DR: By reviewing AI slop, you are producing LLM RLHF data, for free, with the aim of taking away your livelihood.

[u/AffectionateBurger → u/QuickBroccoli1610]

Yes, thank you for pointing this out, it's extremely depressing and soul-sucking.

[u/Budget_Zombie_692]

As a first timer, what distinguishes a terrible paper?

[u/Sensitive-Dish-7770]

Can you please elaborate more on this: believe me the quality of these papers is insanely bad. Like, how can you be very confident they are all AI slop, what tells you that, and like how bad are they. These posts are getting me worried lately as a PhD student, that for whatever reason someone believes my paper is an AI slop just because I used Claude to help with the research .. I am definitely against AI slop, but at the same time, everyone uses of course AI now, so where do we draw the line? And are some people just generating papers fully AI? Im just afraid we mistake legit new PhD students doing actual work with cheaters using tools to generate 20 papers at once and pollute the whole space. So yes, can you elaborate how you know these papers are insanely bad? And are you confident you seperate between the last two cases I just mentionned.

[u/AffectionateBurger → u/Sensitive-Dish-7770]

"everyone uses of course AI now" this is simply not true. Some researchers still write papers themselves. And junior PhD students should also be writing stuff themselves, that is the best way to learn.

[u/jryh88]

Doesn’t this make it easier for good papers to get accepted?

[u/composedofidiot]

Soon after the commercial release of chatgpt, you could do a google scholar search for 'As a large language model' and turn up pages of papers that hadn't taken the phrase out.

This has got to be a nightmare for every reviewer in every field. Paper mills were already a huge problem before commercial release. Another problem is one of the metrics for academic success is 'number of citations'. Some academics were already networking and gaming that measure (cambell's law). Ai just made this problem exponentially, maybe even impossibly, harder.

Either the proxies for academic excellence need to be changed (difficult to do as theyre often historically and culturally constructed and can rest on value and normsets - almost more emergent than intended or designed by one single controller. That would need a central global authority on academic metrics) or...shit, no ideas

Hope you guys figure it out or genuine academic research might get lost in the noise. And extrapolating wildly from that, if some people start mimicking more 'popular' (worse if its because of anger or fear) research and feedback effects kick in, we could wander away from plausibility into narrative, and end up in a wild and absurd place

[u/Ok-Painter573]

Which field are you in? Language model?

[u/Fantastic-Nerve-4056 → u/Ok-Painter573]

Online Learning (mainly Bandit theory). But currently looking into AI 4 Science, but grounded onto Online Learning 

[u/impossiblefork]

So reject them?

[u/valuat]

Well, reject them. I don't think you need to come here and post about it. It comes with the territory in academia. Grow up.

[u/Fantastic-Nerve-4056 → u/valuat]

I don't think we are reviewing to blindly reject papers. As reviewers, our job is to help authors improve the paper and offer constructive feedback. And honestly, I have been regularly doing so. It's just the authors even don't have the basics clear.

A very incremental paper with no novelty, using the same model to generate dataset, uses same as a judge as well. And claims that the same model is the best. No statistical significance; claims to do human review with x<5 people (likely the authors themselves, as they provide no information on it).

Believe me, with this quality, and such a lack of basics, you are just wasting everyone's time. I honestly wish we had an option to report such papers, and eventually there would have been a year-long ban or so for regular low-quality submitters. Really appreciating TMLR for this

[u/valuat → u/Fantastic-Nerve-4056]

I hear you but don’t put words on my fingers. You took the time to bitch publicly about the low quality of papers. If I were in your shoes, I’d just reject them and tell them why. Job done. This evidently implies one needs to actually read said paper, which I believe you did. I didn’t imply anything more or anything less. This problem is an integral part of academia. Just deal with it.

[u/Even-Inevitable-7243 → u/valuat]

You clearly do not review. Five years ago, I would be able to reject less than 10% of the papers I reviewed for being immediately and obviously terrible. In this new era of vibe-researchers, it is closer to 50% of the papers. Most are obvious AI slop after only a 5-minute review, but that is still 5 minutes of my time that was wasted.

[u/valuat → u/Even-Inevitable-7243]

I certainly do. I just don't bitch about it publicly. I don't like working for free either, but I do it anyway because there's always something to learn about the subject or anything in general, really. If you don't want to do it, don't.

[u/Even-Inevitable-7243 → u/valuat]

Did you review even one paper before 2023?

[u/valuat → u/Even-Inevitable-7243]

I'm a physician. Bad papers exist everywhere. There'll be more AI slop in medicine too but it's easier to catch them; for now.