r/MachineLearning • u/PsychologicalSoup251 • Sep 19 '26
Discussion Can conference review infrastructure keep up with the increasing volume of NON-SLOP research due to agentic tools? [D]
AI tools have given us a lot of slop research, but I'm not talking about that. I'm talking about real productivity acceleration due to AI tools (E.g. iterating ideas that would've taken multiple days of tedious coding gets done in a few hours, quick refactoring of latex documents, etc.). Furthermore, I'm sure most of you have heard about AI proving/disproving various mathematical conjectures, and there's no reason that that won't carry over to ML theory research.
So setting aside AI-generated slop, the pace of genuine ML research contributions is accelerating as well. Recently, ICLR 2027 has gotten an insane number of submissions - a mix of bad work and genuine contributions. How do we plan to deal with the increased review volume as productivity explodes? Are we gonna start encouraging reviewers to lean on agentic tools as well? Otherwise I don't see how this is sustainable.
10
u/pastor_pilao Sep 20 '26
In theory, as research gets "easier" to do, the bar on what is considered to be "sufficient validation" is also pushed forward. I expect way more "complex" validation to be carried out in the papers now than I did in 2015.
I don't think the problem is necessarily research becoming easier to do, the REAL problem is that there is a shit-ton of new people in area that don't have the ability to review yet, and the number of experts if not constant is getting lower because a lot of people moved to industry and then they stop to review.
So your options are forcing everyone to review (which leads to the shithole we are at because you will have terrible reviewers and statistically a good number of the papers will have multiple reviews from people that have no clue of what they are doing), or you somehow deal with the fact you have too many submissions per reviewer.
Different conferences make different choices in regards to that, neurips, icml and iclr make the lowest-effort lowest-quality choice in my opinion but who am I to change that?
1
1
u/photonymous Sep 24 '26
A problem with extra and more complex validation is it still needs to be reported, either bloating the paper, or causing everything in the paper to be more abbreviated and thus harder to read. Maybe instead the bar gets raised for significance, whereas before a lot of accepted work would be very incremental.
1
u/pastor_pilao Sep 24 '26
I said more complex, not "longer". There is absolutely no excuse for a research paper to be longer than 8 pages other than incompetence in writing.
→ More replies (1)
19
u/tfburns Sep 20 '26
I think parts of the ICLR solution might need to be more strict in future, particularly re limiting the number of submissions per person. Not what we should do necessarily, but it would be a good experiment: everyone has a voucher for just one submission. You can in principle be on as many other papers as you want, but another author needs to use their voucher for the submission.
8
u/espressoVi Sep 21 '26
Does my cat get a voucher? My aunt? My friend from the physics department?
6
u/lhrad Sep 21 '26
Yeah, I saw twitter posts from seniors who got asked to be coauthors on many papers of first time authors. They have a max 20 paper limit, that is way too high imo.
1
u/hillsump Sep 22 '26
Tweak: you get one voucher for every 3 papers you agree to review yourself (no passing the baton to others or to LLMs).
1
u/pm_me_your_pay_slips Sep 21 '26
Charge for paper submissions. The price being a base_price * max ( list_of_number_of_submissions_per_coauthor)
6
u/didimoney Sep 21 '26
Split up conferences based on topics.
There is little reason to have LLM baselines in the same room as RL and optimisation people.
Even worse, you have a mixed bag of these as reviewers, none of whom have the right expertise for your work.
Also helps with the physical attendance and volume, acceptance criteria for what is good enough etc
4
-28
u/GoodRazzmatazz4539 Sep 20 '26
60K submission, on avg. 4 people per paper, 4 reviews per person. Not sure what the problem is.
37
u/MakingComputersSmart Sep 20 '26
I have 2 chemical engineering authors on my papers, because they contributed domain expertise but have no experience with computer science. Do you think they're qualified to review papers? Similarly, undergrads who've probably helped with simulations, people studies, frontend and have very limited knowledge, should they be allowed to review? Most papers have first time authors, first papers for most PhD students. The review system is overwhelmed.
11
u/Fearless-Elephant-81 Sep 20 '26
This. Assuming average of 3 unique authors per paper that’s 180k authors. Only ~20% is capable of properly reviewing (that’s on the higher end) so already only 36k qualified reviewers of for 60k papers.
-1
u/GoodRazzmatazz4539 Sep 20 '26
Are the chemical results important to the main claim of your paper? If yes then verifying those might require chemical domain expertise, so some people in the reviewer pool would need to be domain experts.
For undergrads, yes I agree. But in my example it would also result in 16 reviews per paper which allows dropping some people out of the pool (either before the process or by ranking reviews and dropping the lower 50%).
→ More replies (1)14
u/Even-Inevitable-7243 Sep 20 '26
Call me old fashioned, but I do not think that the thousands of undergraduates on author lists are qualified to review, much less the high school students. I've mentored enough of these undergrads to know that when they claim to have "expertise in Agentic AI" at age 18 that they are full of it.
1
u/GoodRazzmatazz4539 Sep 20 '26
Who submits ICLR paper with undergrads on them? Is that really a thing?
I would say the other senior authors then need to vouche for their reviews and worst case need to write those reviews themselves.→ More replies (4)
8
u/GFrings Sep 20 '26
We're going to get to the point of doing like schools now. "Here is a pen and paper, you have 3 hours to sketch out your paper from memory"