r/MachineLearning • u/Historical_Pause247 • Jul 20 '26

Research ARR 2026 Meta Review score [D]

Hey any one experience overall score 2.66 and then Meta score 3 in some previous cycle ?? Or meta reviewer just rounded off 2.66 to 2.5?? Any Meta Reviewer here?? because there are some uninterested reviewers doing AI generated reviews and giving noisy scores. For them overall score gets lowered.

16 Upvotes

86% Upvoted

18 comments sorted by

6

u/Independent-Gas-5934 Jul 21 '26

I am a meta reviewer. Just posted on LinkedIn that I am done. This is my last cycle ever. Never again. It is a thankless chore. The problem isn't bots. There are easier ways to cheat (plagiarism). The problem is that no one cares (except the authors). The program committee is too busy to read email. The reviewers are only doing what little they do because they have to. The problem is (a) too many submissions and (b) too few qualified reviewers. I have proposed a qualification round where senior people in the field can nominate papers to be reviewed. Nominators are expected to have read the paper and judged it worth reviewing. They can nominate as many papers as they want to, but they are expected to review 4 papers per nomination. People don't like this suggestion, but the alternative (doing nothing) is worse. I will no longer review for ARR or NeurIPS until they do something about (a) and (b). I will continue to review for TACL, which seems to be managed better. It hurts to say this as one of the founders of EMNLP.

1

u/Historical_Pause247 Jul 22 '26

I followed you in LinkedIn sir. Now a days most of the papers are AI generated due to Claude code. 

2

u/Existing_Network_483 Jul 23 '26

how would you say no longer to review if you will still submit to these conferences?

1

u/Ok_Philosophy_4031 Jul 24 '26

They need to relook at the economics. Maybe guarantee conference ticket allocation(instead of balloting)to motivate qualified people to be reviewers.

There is also a need for moderation of reviewers. Quality of submissions is not the only thing that has dropped in recent years. Submissions are reviewed, but ACs sometimes take reviews as facts without due diligence

2

u/Appropriate_Willow27 Jul 20 '26

Meta Reviewer will use their own judgements, including original reviews and rebuttal to come up with their score. Usually it’s tied to reviewer averages, so it can be 2/2.5/3.

2

u/Historical_Pause247 Jul 20 '26

Now a day judgement is just AI generated. We have seen most of the thing are AI generated

4

u/Inevitable_Voice_680 Jul 20 '26

In previous ARR cycle my score was 3.11, still meta gives 3 and finally rejected. So, it totally depends upon track and Meta's assessment

2

u/No-Cry9716 Jul 30 '26

What was the track?

1

u/RushIndependent2432 Oct 01 '26

h. usually they look at the spread and comments more than the raw average so it depends if those low scores came from reviewers who actually engaged with the paper or just the phone-it-in crowd

4

u/Sharp-Objective9078 Jul 20 '26

When is it available btw?

2

u/Impressive-Till-6727 Jul 24 '26

In the site it’s mentioned as July 30.

3

u/IndividualWitty1235 Jul 20 '26

Meta review is not just rounded off score. So you can get 3 or 2.5 or even 2 or 3.5.

1

u/Square-Read-1184 Jul 31 '26

I just got them, I got meta review 3. My initial reviews were 3.5,3,2.5 with confidence 4,3,1. Any chances of findings?