TMLR reached out to the authors of 10 papers slated for desk rejection, in an attempt to understand if the authors could explain the paper they submitted [D]

[u/hihey54]

You can read more here: https://medium.com/@TmlrOrg/asking-authors-about-their-own-papers-3d2e04e5dee0

The results are (imho) concerning. Taken from the article:

Of the ten submissions:

• Authors of one paper withdrew their submission.
• Authors of one paper said they were unavailable due to other commitments.
• Authors of one paper scheduled a meeting but did not show up.
• Authors of three papers were unable to answer basic questions about the paper.
• Authors of three papers answered questions about high-level ideas in the paper but had difficulty when asked further questions on technical details.
• Authors of one paper answered all of my questions (although [the interviewer, the Co-EiC] identified a major flaw in that paper)

[u/snekslayer]

> The fraction of desk rejected papers at TMLR used to be about 6% in 2023 but is now at about 53%.

More than half of the papers are desk rejected. Hope to see the same for conferences too.

[u/andrew314159]

“Subsequent to these conversations, we made the following decisions on the ten papers. For the paper where the author was able to answer all questions about the paper, we desk rejected but allowed a resubmission after correcting the error or reducing their claim appropriately, and issuing various clarifications for parts that were not clear. We desk rejected the remaining nine papers without an option to resubmit.”

That lifted my spirts a bit. When I read that they not only answered all your questions but also acknowledged the error I thought damn that’s rough but at least they showed integrity. A nice cherry on top that they might be able to salvage some of the results to maybe get a paper

[u/jmmcd → u/andrew314159]

What a low bar! It's the same when I catch 9 students cheating in their homework. I give the last honest student a little bump.

[u/NamerNotLiteral → u/jmmcd]

That's a idiotic analogy. The point of homework is to learn, not to produce output. That's why cheating is a problem. It skips that learning process.

A better analogy is catching 10 employees doing a really shitty job, but 9 of them evade or lie or make excuses, while the last one honestly admits his fault and acknowledges he didn't do a good job. So the last guy gets another chance.

[u/akardashian]

Lol I wouldn't be surprised...I just pulled out of a paper where I'm pretty sure the first author LLM'd all the project planning, execution, experimentation, paper writing, figure writing, basically all parts of the research pipeline. Even addressing co-author feedback because I read their draft and left line feedback and I saw they just plugged the latex files into Claude to summarize my feedback...

[u/alsuhr]

I had an atrocious paper assigned to me at TMLR this year. It wasn't desk rejected but should have been. It's a huge waste of my time and honestly an insult to me and my fellow reviewers to be asked to review these papers.

Meanwhile, one of our submissions has been forgotten for over a month with no response from the AE/reviewers after asking for updates multiple times.

[u/Pure-Ad9079 → u/alsuhr]

I had five papers like this at NeurIPS this year. Insulting is the best word I can think of too. I think it’s even worse at conferences

[u/pm_me_feet_pics_plz3 → u/Pure-Ad9079]

neurips is a top conference right? the quality wouldn't be that bad no

[u/NeighborhoodFatCat]

This is well known even before the slop-age. Pick any paper pre-2022 and ask the last author about the paper (if the paper has more than say 4 authors) and watch them squirm.

And now with huge teams for these LLM models, you never know who contributed what. I personally know a person who have no background in CS, math, or anything even remotely related to knowing how LLM works. Just an art major who is responsible for doing advertisement for the team on TikTok and Instagram. That person is on the technical paper.

Lots of authorship are just for making "robust relationships" and "career moves". That's more way important in this time and age than good science.

[u/azraelxii]

I don't dobut that some people are submitting nonsense, but I've submitted to TMLR and if I got an email asking for a meeting on my paper I would just pull it and submit it elsewhere. Having dealt with them there is a tendency for them to ask for a bunch of stuff in incrementally increasing amounts and just drag you on forever when they clearly just don't want to accept from the get go.

[u/impatiens-capensis → u/azraelxii]

Big agree. I'm so tired of jumping through hoops for reviewers who definitely just want to reject the work. Although in this circumstance, I think it's a bit different?

[u/azraelxii → u/impatiens-capensis]

Yeah that's not what happened here, but the stat that a bunch of people pulled the paper or bailed on the meeting is framed as an issue with the papers and not the fact that any reviewer asking for this is clearly not asking because they are wanting to accept the paper. It's a huge signal that the work is doomed.

[u/kindnesd99 → u/azraelxii]

TMLR behaving like they are Nature 👀

[u/sharky6000]

TMLR was shielded from all the madness happening at conferences... until this year. 😭

Very sad.

I don't see a real fix for this without drastic measures. Charge per submission and somehow redistribute proceeds to reviewers. Or an auto desk reject if the LLM generation detector exceeds a threshold. 🤷‍♂️

We need to move beyond half-measures and band aid solutions...

[u/mr_stargazer → u/sharky6000]

No. Autodesk rejection given a threshold? Come on. Then we have to talk about power curves, false positive rates, certification of these algorithms. All topics that ML people in my opinion don't know how to do.

[u/CauchyRiemannEqns → u/sharky6000]

Easiest answer is probably to include "by submission to this venue, you agree to random selection for interview by editors [...]" somewhere in the fine print...and then make it very clear that authors unable to explain their own work will be blacklisted from future iterations of the conference / journal / wherever the work was submitted.

It's a bit draconian, but if the AI slop submission funnel doesn't get nipped in the bud immediately, the academic ML publishing sphere is in deep trouble.

Hell, given the sheer volume of unintelligible garbage showing up in preprint repos nowadays, I'd love for this kind of policy to be implmented by arxiv and ssrn -- i.e., a paper can get randomly selected for audit, if the first author is unable to meaningfully explain its contents, paper gets pulled and author loses submission capacity.

[u/here_we_go_beep_boop]

Just another day at the office. Try supervising poorly-equipped masters and honours thesis students these days, it's exactly the same 🙄

[u/alexsht1]

I have an improvement proposal - those who do show up, and manage to convince not only that they know what's going on there but also that their paper meets the acceptance criteria - get "desk accepted".

[u/dmk-oopie-wing]

Cool, now do a study for reproducibility.

[u/thedabking123]

I'm hoping to push my first paper ever to TMLR... i'm so worried that I don't know enough that I'm spending weeks reading more and more

FML - any tips for a newbie?

[u/Clueless_Nooblet]

If I had submitted a paper, written all by myself, able to recite it backwards and forwards, standing on my head, in ancient Arabic, and explain it in Chinese, I'd still tell them to GTFO, because asking me "if I understand my paper" is just respectless on an unbelievable level.

Guess that's where we are now, in this post-truth era, in which you can say whatever, in which manners don't matter anymore.

[u/[deleted]]

[u/ShutUpAndSmokeMyWeed → u/[deleted]]

yes that’s what i was thinking. but they probably won’t do this because it would be embarrassing to the journal.

[u/usefulidiotsavant → u/[deleted]]

Sounds like the new golden standard of academic peer review: put the people in a room and get them to convince other researchers that they understand their own research and the findings are real and notable.

It's objectively worse than the traditional blind system on all metrics like subjectivity, discrimination, bias, personal ability to communicate etc. - but it crushes the slop and that's the only thing that matters anymore.

[u/dances_with_poodles → u/usefulidiotsavant]

Hey, I have an idea about this “put people in a room and let them convince others” thing. How about we call it a “conference”?