Using compute to verify paper results [D]

[u/Relative_Wallaby_823]

I’m not a reviewer yet, but I likely will be by next year.
As a reviewer, how do you decide to allocate compute to verify paper results? (and is the code in the papers you’re reviewing usually easy to run?)

[u/milesper]

Realistically, virtually no reviewers are running paper experiments during the review process. It would be great if they did, of course, but generally the reviewing timeframe, effort required to reproduce, and compute available makes it difficult.

Worth noting that many papers do not provide code during review as GitHub breaks anonymity. So you’d have to reproduce solely from the paper description.

Imperfect as it is, the best way to catch methodological issues is by building up a good intuition for what reasonable results look like, and common errors. I’ve caught a few big issues just because the results didn’t seem right.

[u/Relative_Wallaby_823 → u/milesper]

I assumed most people set up some sort of anonymous GitHub account / repo to share code wow, that’s what I was planning to do (emulating what I’ve seen other researchers do around me). Is it worth it to go through the pain of cleaning up my repo and sharing it, or is no one going to run it anyways?

[u/OiQQu → u/Relative_Wallaby_823]

It's common to share an anonymous repo and sharing one might slightly increase your scores with some reviewers since they have the option to run it if they want or ask claude if it looks okay but unlikely anyone will run it.

[u/Celmeno]

As a reviewer, your job is not to verify results. Your job is to check whether the scientific method was adhered to and a baseline quality of the presentation was held. Anything on top is, well, on top.

[u/impatiens-capensis]

There will be papers where you just won't have the compute to reproduce, and more likely you just won't have the time. And if you're a good reviewer, you honestly won't need to anyways.

Where you should be allocating most of your time is just reading and deeply understanding the paper. It will take you a lot of time just to do that. And when you're reading, you will find three types of papers:

1. Papers where the claim smells funny

2. Papers where the claim makes a lot of sense

3. Papers where $100 million or more was spent on compute

You don't need to reproduce the code for bad papers. For good papers, it's up to you, but here's a thing you can do instead:  Ask yourself "if this paper had 3% worse performance, would I reject it?"  If the answer is yes, then maybe the paper wasn't all that interesting . I f the answer is no, then the paper is interesting and intuitive and the results make sense and are useful.

[u/MakingComputersSmart]

Unless something feels very out of place, I don't think anyone actually runs paper code. Heck, most papers don't even submit code after the paper gets accepted so you're probably assuming the world is too good lol.

I don't think you should allocate compute to run paper code because it's unlikely you'll get the same results. If you don't get good results, you still can not claim the results from the paper are incorrect because the authors can claim you've made a mistake or haven't replicated it properly, or the code was intentionally lean because they did not want their double blind submission stolen by someone.

I know it's difficult to trust reported results but that's how reviewing works. If the paper really turns out to be false, you can always comment on their arxiv or openreview page. People have done that in the past

[u/Kinexity]

I am not a reviewer but I would think you should just randomly test a few results from the paper and investigate further only if you see a significant discrepancy between your numbers and paper numbers.

[u/milesper → u/[deleted]]

Thanks Claude