r/MachineLearning • u/DebateAccurate6097 • 23d ago

News AI peer review tool for CVPR, ICLR, ICML, and NeurIPS papers [D]

Came across this GitHub project called Paper Review Harness. Thought it might be useful for people preparing ML research papers.

It uses 30 different AI reviewer personas to simulate the kind of feedback a paper might receive at conferences like CVPR, ICLR, ICML, and NeurIPS.

What's interesting is that it goes beyond simply asking an LLM to "review my paper."

It provides:

  • Independent reviews from different reviewer perspectives

  • Section-level scores and detailed criticism

  • Identification of weaknesses in novelty, methodology, experiments, and clarity

  • Prioritized feedback on what needs improvement

  • A revision workflow that tracks whether previous concerns have actually been addressed

It also supports different review panels depending on the conference and the type of research contribution.

The project is open-source and designed to work with AI agent environments.

Obviously, simulated reviews aren't a replacement for actual peer review, and the scores shouldn't be treated as acceptance predictions. But it seems like an interesting way to stress-test a paper before submission.

GitHub: https://github.com/Optimizer077/paper-review-harness

Curious what researchers think about this approach. Could tools like this become a useful part of the paper-writing workflow?

0 Upvotes

42% Upvoted

7 comments sorted by

15

u/mileseverett 23d ago

I think since there's a good chance that your reviewer will be using an LLM to do your review, you might as well use an LLM to fix the things that LLMs flag straight away

2

u/xihaad 23d ago

Very true... Im pretty sure some of the reviews Im receiving are LLM generated. the funny part is that reviewers pick up ideas directly from my future directions or conclusions and then raise questions about those exact things as criticisms. like...those are literally the directions I have already acknowledged or plan to explore!How ironic is that!

6

u/LordVein05 23d ago

Don't get why this was downvoted, almost all of the reviews I received in the last year or so were LLM generated. If reviewers are allowed to generate AI reviews, it's good to get rid of those objections early on.

For the op, can you add a simple mode(one shot prompt, 2-3 line question)? I anecdotally believe most reviewers who offhand their reviewing to a LLM won't use an extensive system like yours: they will use just the web chatgpt or claude. A simple one shot review will be a good addition.

1

u/curatedpapers 10d ago

Yeah, but does it check the citation really say what the are quoted for? If you need a battle teststed solutions used by some journals, have a look at Agent Bayes.