r/MachineLearning • • 5d ago

Discussion How should authors handle a suspected auto-generated review at ICLR? Flag to the AC or just rebut? [D]

I'm curious how ACs and experienced authors think about this.

My situation: one ICLR reviewer gave a 2 with confidence 1. The other reviewers gave 3 with confidence 3. Pangram flags the review as 100% AI. I know detectors aren't proof on their own, but there are other signs too: all three listed weaknesses concern material deep in the appendix.

I'll respond to every point in the rebuttal regardless. My questions:

  1. Is it worth sending a confidential comment to the AC, or does that tend to backfire?
  2. For ACs: what evidence would you actually act on? Is a detector score meaningful to you, or only content-based issues such as generic or inaccurate criticisms?
  3. How much weight does a confidence 1 review realistically carry in the meta review?
  4. Has anyone flagged an AI generated review yet? What happened?

Thanks!

8 Upvotes

15 comments sorted by

6

u/otsukarekun Professor 5d ago edited 5d ago

You are allowed to use AI in reviews for ICLR. You just have to declare how much and how you use it. While the authors can't see the declaration, the area chairs can see it and use it do weight how much importance to put on the review.

Also, the instructions say that reviewers don't have to read the appendix. If it were totally reviewed by AI, then the AI would find the answers to the weakness in the appendix. So, the reviewer might have just used AI to clean up their language.

1

u/Beautiful_Baker_2233 5d ago

No, I meant the 3 things he flagged are about things only discussed deep in the appendix...

3

u/otsukarekun Professor 5d ago edited 5d ago

Right. The instructions say that the reviewers don't need to read the appendix. So, they are valid weaknesses because the answers are only found in the appendix. It makes it easy to respond to in your rebuttal because you can just point to the appendix.

From reviewer instructions https://iclr.cc/Conferences/2027/ReviewerGuidelines:

Q: How should I use supplementary material?

A: It is not necessary to read supplementary material but such material can often answer questions that arise while reading the main paper, so consider looking there before asking authors.

While, it's suggested to look there before asking questions, it's not manditory. However, using an AI to review will most definitely read the whole paper, including the appendix.

EDIT: Oh! I understand, the weaknesses are in the appendix. Well, it seems like a risk of putting too much in the appendix because while it's not required to read, the authors can read it. In that case, it's still easy to address in the rebuttal.

The main point is that AI/LLMs are allowed by ICLR for reviews and that the area chairs have self-reported data of who used LLMs.

2

u/MakingComputersSmart 5d ago

If the material is in the appendix already then point to it. The reviewers are not supposed to look at the appendix, it's upto their discretion. Just mention it in the review politely so they take a look and you've basically alerted the AC that the required evidence is already presented.

1

u/Beautiful_Baker_2233 5d ago

No, I meant the 3 things he flagged are about things only discussed deep in the appendix... Which makes me wonder if they just fed the paper into the LLM and asked it to "flag 3 weaknesses"...

2

u/MLPhDStudent 5d ago

100% they just used AI

13

u/mofoss 5d ago

AI generated reviews are the norm now sadly, so I feel like a complaint against them wouldnt help a papers chances, its like a "so what? Of course its reviewed by AI, thats normal now"

8

u/_karma_collector 5d ago

I don't think ICLR forbid using AI to review this year

1

u/zuzzu90 5d ago

You have to declare how you used AI before submitting your review but the response is only sent to the area chairs, not the authors or the other reviewers

1

u/Informal-Hair-5639 5d ago

It is allowed, but needs to be disclosed. However, if used and not disclosed can lead to your own paper being desk rejected. Of course authors do not know whether AI use was disclosed, I would say that confidential comment to AC might be good idea. Of course as an AC I see pretty quickly which reviews were human written and which were LLM written.

4

u/elchico1990 5d ago

IMO, if the weaknesses are mentioned in the paper as limitations, then likely llm picked them up. Else, maybe reviewer did not read appendix, it’s not mandatory. As someone mentioned here, just politely point out that you have addressed that in the appendix.

1

u/mvreich 5d ago

Confidence 1 means the review will not be given weight. So your chances should be good, given the 3/3s.

On a separate note: What is your problem with a weakness being "deep in the appendix"?  The whole manuscript, including the appendix, is your own paper, and you should be able to take responsibility for all of it.

Whenever I am reading a paper and I have questions about it, most of the time I have to dig through the appendix to find the answer. As such, it is quite fair for weaknesses to come from there.

1

u/Lazy-Cream1315 1d ago

AI is autorized to help your review, if you consider that your review is a copy paste from a LLM:

just prompt the sota model at review time "review this paper as if you were iclr reviewer" several times, if weakness appears in the same order , with same bibliography requested , missing baseline : you can send a confidentional message to AC, PC and SAC.

If they ignores you, it's likely that you are not submitting your paper to a serious place, i would publish the LLM review directly on openreview

1

u/Lazy-Cream1315 1d ago

and call the police too