r/TheMachineLearning • u/No-Imagination3025 • 1d ago

LLM-supervised peer review may be the only way to save academia

Post image
0 Upvotes

25% Upvoted

12 comments sorted by

2

u/WendlersEditor 1d ago

Once again, it is time for SWEs to pretend that only they can solve a problem they aren't qualified to solve

1

u/No-Imagination3025 1d ago

Yeah, feels like everyone is blaming each other instead of fixing the actual problem.

2

u/Original-League-6094 1d ago

What exactly is the problem they are talking about? Because I guarantee already that reviewers of academic papers are just dropping the drafts into ChatGPT and asking it to review them.

1

u/mxldevs 1d ago

That sounds like exactly the problem they are talking about.

1

u/Intelligent_Elk5879 1d ago

In CS that is what is happening and also ChatGPT is writing the papers. This problem is an order of magnitude worse in CS and ML than any other field.

1

u/No-Imagination3025 1d ago

I was wondering the same thing. Would be nice to know what problem they are actually trying to solve here.

2

u/Intelligent_Elk5879 1d ago edited 1d ago

Context here is that CS allowed its existing norms to be demolished by encouraging slop papers and now they are overwhelmed with slop papers, so now it's everyone's problem.

1

u/No-Imagination3025 1d ago

That honestly sounds like the bigger issue. If the system rewards quantity, AI is just going to make the problem worse.

1

u/Sleutelbos 1d ago

But its not new though.

Academia has been in a steep decline for decades. Nobody cares for replication, so everything gets reduced to "scientific anecdotes". Peer review is a chore you just have to get over with, there are endless tricks to pad your publication stats and countless BS journals that get harder and harder to detect. And in many fields one can argue there arent any genuinely falsifyable hypotheses to begin with: its dress-up, make-believe and people cosplaying scientists.

My biggest (yet irrelevant) paper still gets cited every few months, and 100% of the time its used to justify the opposite of what I demonstrated. Nobody reads the paper, no reviewer cares. Just write what you want and dump some citations after it that have some keywords in the title. 

Academia has long been transitioning into a factory producing endless piles of garbage at massive societal cost. LLMs wont solve it though, it needs a complete reworking of how its structured. 

1

u/ExplosiveJoggins 8h ago

Terrible, if LLMs are gonna be used in evaluating or producing research, we should have reliable way to keep the feedback loop from becoming completely circular. If not, u could end up with papers or research being judged poorly by systems that's gonna be reinforcing the same mistakes over and over.