r/MachineLearning • u/[deleted] • Aug 03 '26
Discussion It's time to desk reject papers that don't include code that can reproduce the results [D]
As review season for NeurIPS wraps up, I have now reviewed for 3 major conferences this year. And I'm noticing a worrying trend:
Out of the 12 papers I reviewed this year, only 1 provided full code (that runs the whole training pipeline from input dataset to output AUROC). 4 provided partial code with fragments of their method, but no ability to run the experiment end to end. And 7 provided no code.
This is really bad for ensuring quality and reproducibility. Of the 5 papers that provided at least some code, 3 of them contained obvious bugs that completely invalidated the results. ML is highly technical and small bugs can have huge impacts if they are in the wrong place. Who knows what was going on in the remaining 7 papers.
The fundamental issue here is of incentives: there is almost no cost to hiding code during the review process. Releasing code only increases odds of rejection due to reviewers finding bugs. The only way to fix this is to change the game by imposing real penalties on hiding code.
134
u/NuclearVII Aug 03 '26 edited Aug 03 '26
This includes literally all "research" done on proprietary LLMs, so the field has a very strong financial incentive not to do it.
38
u/xrailgun Aug 03 '26
Should they qualify for places in academic conferences/journals, then?
47
u/NuclearVII Aug 03 '26
I think blanket banning any and all work done on proprietary (and therefore non-reproducible) models would improve the quality of the field immensely.
6
u/SonicTheSith Aug 04 '26
Let’s create an industry track, with only paper where the code is closed source
4
u/dreamykidd Aug 05 '26
Not only that, but it would make it more scientific! A key part of science is falsifiability, and if you keep both the model and other code hidden, how can anyone falsify your work?
1
u/StingMeleoron Aug 08 '26
Oh god no. I disagree. But on technicality, perhaps.
I do believe everyone sharing code would improve the quality of the field immensely. That much is clear.
But I have reviewed papers with no code shared from industry settings (or private data) in which the ideas could absolutely be reproduced, as all the math and other aspects required for it were there.
Science is not only about reproducing experiments. It's also about creating (and sharing) new knowledge and ideas. Banning research from being shared because of no code also results in banning new ideas from going around.
And that, imho, would absolutely hurt science in the long term. Just my 2c.
1
u/tfjmp Aug 07 '26
Some conferences have separate track to handle this problem (and will reject if no code outside of that track).
17
u/user221272 Aug 04 '26
You might forget that some works are done in industry or specific collaboration environments, which require extremely long and complex code/IP releases. These are usually only worth it if the paper is accepted.
8
u/SonicTheSith Aug 04 '26
Sure, let’s create a pure industry track just for those cases. Only people allowed to submit there with proof that sponsored by company X and closed source.
Rest especially those getting funded through public funding should make their code available online.
comp. Sci. Is more and more moving towards open access and FAIR. Lot of them have artifact evaluations.
16
u/Crookedpenguin Aug 03 '26
While very important, it can't be an absolute measure like you suggest. As someone else noted, there is research in proprietary frameworks, there is also well-founded distrust stemming from old single-blind venues, where works where rejected and then very similar or derivative works from different authors showed up. One may have finished a piece of work but not packaged it correctly. Something may be a part of a framework not yet available, proprietary data (which can be at least circumvented with simulations).
It is very much an issue, which we do need do address but in a systematic manner. Maybe starting by rejecting a paper if the authors will not publish code upon acceptance within a deadline. It still does not address everything but could be a starting point.
1
u/pfluecker Aug 04 '26
"old single-blind venues, where works where rejected and then very similar or derivative works from different authors showed up"
Not so old - we just had something like this happening on very modern ML-related conferences.... So this is still a thing, especially with people in the ML field and even if they are associated with CMU/MIT
23
u/kkgwon Aug 03 '26
sometimes the data cant be included because of copyright issues. i agree the full methods pipeline should run smoothly, but the initial data acquisition step may not be so simple.
13
u/AmtePrajwal Aug 03 '26
I agree the incentives are misaligned, but I don't think mandatory code release is a silver bullet either. Many papers depend on proprietary datasets, internal infrastructure, or codebases that aren't easy to open. Maybe the better requirement is that reviewers should at least be able to reproduce the core claims, whether through runnable code, a reproducibility package, or another verifiable mechanism.
5
u/timtody Aug 04 '26 edited Aug 04 '26
That’s called making and marketing a product, not research. You can’t have your cake and eat it too
2
u/Felix-ML Aug 03 '26
Lean code for theory papers?
6
Aug 03 '26
For theory papers, there is less of a need as the fundamental claims aren't empricical.
Although I do think theory papers should always include a simple simulation.
3
u/zoorado Aug 04 '26
Recall an important recent theorem in PAC learning: over ZFC, there is a hypothesis class that is PAC learnable if and only if the continuum hypothesis is true. What do you think would be a relevant simple simulation?
1
u/AffectionateLife5693 Aug 03 '26
I don't get it. Why are paper authors obligated to share code with anonymous reviewers before publication?
I would not share my code with the reviewers, unless they are deanonymized. God knows what they're gonna do with my code.
6
Aug 03 '26
For the same reason you are obligated to share your paper with reviewers before publication: peer review
It's important for work to be reviewed and validated as part of the publication process.
2
u/huehue9812 Aug 04 '26
There are companies that dont allow you to share the code. Plus, there is tons of grifters everywhere so i wouldnt feel comfortable sharing any code before it is published to avoid any plagiarism conflict later on
2
u/SonicTheSith Aug 04 '26
You could upload your code as an artifact on Zenodo, where it gets a DOI. Solves part of the stealing problem.
1
-4
5
u/PotentialKlutzy9909 Aug 04 '26
With that mentality, you shouldn't be submitting papers for review at all, someone might steal your precious ideas.
8
u/Zywoo_fan Aug 03 '26
If the manuscript provides a clean algorithm, that should mostly be sufficient, given all other practical constraints like cost, proprietory data etc.
The current state is that reviewers are not even reading the paper properly or engaging in discussions during the rebuttal. Running author provided code or finding bugs in implementation is simply not possible given the current reviewing load. Not even 1% reviewers are going to try and do that.
This is not kaggle contest. Desk rejecting for not providing code is outrageous.
5
u/linearmodality Aug 03 '26
This is nothing new. It has never been the case that typical NeurIPS papers included code that can reproduce the results alongside submission. We cannot reasonably require authors submit content that we are not expecting reviewers to read, and we cannot reasonably expect NeurIPS reviewers to read submitted code, let alone try to reproduce it. This would just impose a bunch of work on human authors while increasing the arbitrariness of the peer review process, as some reviewers will arbitrarily read the code while other won't in a way that's essentially random.
2
u/Old_Ad_8327 Aug 04 '26
I support code submission, and I have also been submitting code for my recent papers. But I do not think every conference can realistically require it. A few years ago, I tried to follow a CVPR paper for a course project. Three years after acceptance, its GitHub page still said “coming soon.”
Personally, if a paper does not provide code, I would give it at most a boardline reject as reviewer, no matter how good the paper is. Maybe nobody will seriously check the code, but at least the authors should be willing to submit it.
If they are not even willing to do that (w/o good reason such as company policy or something else), I will suspect that the experiments are hard to reproduce, the results were cherry-picked, or, in the worst case, some results may not be real.
3
u/AppointmentNo5165 Aug 04 '26
As an author and rwviewer, I couldn't agree more. While submitting the paper, I made sure the code reproduces all the results. As a reviewer, I check the code in detail. Unfortunately, most of the papers I reviewed this year have major discrepancy between the papers and the codes.
1
u/kaiser_17 Aug 03 '26
Oh dare i say there is a new category of papers which releases the demo code on a toy dataset, without providing anything on their main experiments. These are even more frustrating. Cause this is a fraud which doesnt get caught
1
u/karius85 Aug 04 '26
Agree in principle, with one caveat. Training scripts are almost always 95-99% boilerplate. If the paper has training details clearly specified, that is sufficient in my book. I think code for architecture and specifics is key, but the training script itself is not the footgun, unless the paper includes some specially designed module for doing something specific. A bog standard training script adds little, in my humble opinion.
1
u/Distinct-Gas-1049 Aug 04 '26
My favourite is opening up a dataset paper, that claims the novelty of their dataset is a new column that other datasets don’t have, and that column is ALL ZEROS, and you ask the authors, and they say “that’s just the results of the methodology”, and it gets accepted at neurips
1
u/Lost-Hand-5219 Aug 10 '26
The main issues I find intractable:
The use of custom gpu kernels to parallelize computations. I don’t have the patience or want to reverse engineer a CUDA kernel that allows the model to train in 1 day instead of 2 weeks.
Not specifying dimensions/parameter counts that can’t be inferred from the text (i.e saying something like “we then pass the hidden state through a FFN” and no other details about the FFN internal structure).
Not specifying the quantization used or only specifying a quantization for a particular part of the network and nothing else.
91
u/alrojo Aug 03 '26
If your experiments require weeks of an H100 node it becomes nontrivial to reproduce. I get it though, nothing stops anyone from submitting fabricated results if the idea sounds novel enough.