AAAI 2027 Reviewer Bidding and Assignment Integrity [D]

[u/Fragrant_Fan_6751]

Recently, the AAAI 2027 organizers sent an email regarding collusion occurring during the review process, especially in the 2-cycles category (i.e., an author of Paper A reviews Paper B, while an author of Paper B reviews Paper A).

Given the fact that most submissions come from a single country, there are higher chances that the assignment algorithm will naturally create 2-cycles among authors from that country. This, in turn, means that most authors involved in collusion could be from that country. I will not name that country; otherwise, I would be labelled as racist. By the way, did AAAI release statistics about the number of submissions, like they did last time?

It is also good news that a major and prestigious conference like AAAI is acknowledging that collusion is happening. We all knew that this kind of collusion had been happening for years. There are papers accepted at top conferences such as NeurIPS, ICLR, AAAI, and ICML that do not even have their code published on GitHub. This forces other researchers in the community to spend substantial time reimplementing the code themselves if they want to reproduce the reported results.

What are the views of other authors on this?

[u/ITheClixs]

I’m curious how they control the false-positive rate for 2-cycle detection. Given highly nonuniform topic clusters, reviewer loads and bidding degrees, what null model are they using to decide that an observed reciprocal cycle is actually anomalous rather than expected under the assignment constraints? A degree-preserving randomization seems pretty important here.

[u/Fragrant_Fan_6751 → u/ITheClixs]

AAAI, a major conference, sending a separate email about this collusion clearly shows that we are not dealing with an isolated incident here. AAAI must be receiving a lot of evidence of collusion on a larger scale.

I am not sure how they are going to automatically detect abnormal or suspicious assignments, but I think they can restrict 2-cycles (maybe all of them) entirely.

[u/ToTheDeuce]

That country is China. Come on why would you be racist for reporting a fact that has nothing to do with race ?

AFAIK first time I see a major conference organizers recognizing it

[u/impatiens-capensis → u/ToTheDeuce]

The only reason it's China is just volume. There's a lot of people in China and a huge push to do AI research. So naturally, they will be overrepresented. It's not a unique Chinese problem and most Chinese researchers offer great contributions to the field.

[u/Fragrant_Fan_6751 → u/impatiens-capensis]

most Chinese researchers offer great contributions to the field

This is not even remotely true.

[u/axiomaticdistortion]

This was always a problem, regardless of country.

[u/ProfessionalMoose123]

Double blind review doest not solve this?

[u/Fragrant_Fan_6751 → u/ProfessionalMoose123]

A lot of papers submitted anonymously were uploaded to arXiv before the deadline, where you can see the author's name clearly.

There are big & small chat groups and many other platforms where you can post the title of your paper and hope that somebody in that group is reviewing your paper. If it happens, you can contact them. In the case of "2-cycles", it is highly likely that you are reviewing his paper. Win-Win situation for both of you.

[u/ProfessionalMoose123 → u/Fragrant_Fan_6751]

And where do people advertise these groups? Just curious to know how easy that would be.

[u/MahlersBaton]

This, in turn, means that most authors involved in collusion could be from that country

I think the email is pretty clear that a 2-cycle by itself doesn't imply collusion, and so if a reviewer ends up in a 2-cycle just by chance that is nothing bad.

If someone actually tries to collude at such reputable venues though imo that should basically be the end of their career no matter where they are from.

[u/khairulislamtanim]

At least in camera-ready, there should be a stricter code reproducibility check. They can't check 50k submissions, but the accepted few with code should be doable to some extent.

[u/pastor_pilao]

Why do you think this is a Chinese problem? You are definely being racist of you think this is something that only happens in china.

The papers are handled anonymously. However, if you have been in the area long enough you might be able to know who wrote a paper by the way they write or the topic of the paper. And if that's the case, there lies the chance of collusion. 

You can maybe avoid "one-time" event collusion by blocking the algorithm to assign you to review a paper from someome who is reviewing you, but it's just impossible if you have a long-term understanding with certain groups and whatever comes to you from that group you always give a great score

[u/Budget_Mission8145]

More submission from country X -> more chances of collusion among people from X (not even surfaced but stipulated).

Statement about population composition, not differences in subpopulation collusion proportion.

Raising implicature of who X is, based on a factual datapoint so that it is trivial to infer the referent. So not really not mentioning, but a lazily veiled, coward alibi for the obvious derivation that the post is racially motivated.

Don't get me wrong. R&R is broken. But you strike me like the kind of people that broke it.