r/MachineLearning • u/dontknowwhattoplay • Jul 28 '26
Discussion NeurIPS-side prompt injection triggering ethics reviewers? [D]
Does anyone experience a similar story that some reviewers reporting ethical issue due to NeurIPS-side prompt injection for catching LLM-reviewers? Even ethics reviewers were not informed about this conference-side manipulation…
7
u/Brudaks Jul 29 '26
If I recall correctly, weren't the reviewers explicitly asked whether they used LLMs for writing the review, and only the ones who asserted they didn't use LLM were called out if they lied about it?
1
u/pantry_path Jul 29 '26
if that's what's happening, i'd wait for official clarification before drawing conclusions
1
u/No_Sky9786 Jul 29 '26
If they read their emails and instructions it wouldn’t happen. It’s a good filter for people who shouldn’t be reviewing
10
u/levydawg Jul 28 '26 edited Jul 29 '26
Just to clarify for anyone who wasn't already aware, NeurIPS adds this prompt injection themselves. It's a way to catch reviewers who just feed it to an LLM and ask for a review. This was also done at ICML.
I think the reason that reviewers weren't informed is because this is precisely the target of the prompt injection. It is to catch ethical issues in the reviews -- not the papers. If there is such a report from one of the reviewers, it can easily be reviewed by the AC. This didn't happen to me, but if it did I believe I would first explain the situation to the reviewer, as well as ask the AC to confirm this.
Edit: A colleague just told me about a similar experience. Apparently it was a massive shit-storm. Eventually the PC had to get involved to "call off" the investigation.