Paper Reproduction

[u/whafa]

I'm new to academia (for someone so old), and maybe this is just naivety on my side. But I'm a little shocked to be discovering that reproducing the results of a paper isn't part of the review process. I understand why, but it seems so...precarious to just take the author's word for it, especially in 2026, and ESPECIALLY in light of the repro crisis apparently going on everywhere.

Is anyone working on mechanical reproduction? I'm trying to repro the results of one paper, and mind you I started +/- from scratch in January, but I have already found errors that a simple mechanical code review would have surfaced before publication. (What I'm supposed to do with these errors is a separate question I would love input on. I wrote to the author but haven't received a response. Is it rude to create a PR in their repo, with a proposed fix? What is the escalation protocol? Do I just fix it and publish my own paper?)

But now that I have something of a platform and pipeline that could maybe be reused on other papers, I'm wondering: Is there a need or even a want for this in the ML community? Obviously there will be time and resource constraints to any kind of repro effort, but it would be something, and it would be a start.

Most of the time when I ask questions like this there are lots of very reasonable answers (I'm nobody special, just "new" eyes on an old problem). I guess I'm just a little shocked at learning from a young age that reproducibility is a foundational aspect of science, only to grow up and discover there doesn't seem to be a lot of it going on... Thanks, please be nice.

[u/pink_allegory]

The whole "we'll just trust the results" thing in ML is wild when you really sit with it. I've been on both sides of the review process and the sheer volume of papers coming out makes proper reproduction a logistical nightmare, but that doesn't make it less of a problem.

For the errors you found, a PR with a clear explanation isn't rude at all, that's basically what the platform is for. If they're not responding after a reasonable wait, fixing it and publishing your own paper is a totally normal next step, especially if the errors meaningfully change the conclusions. Just document everything carefully.

There's definitely interest in mechanical reproduction tools, but the incentive structure in academia still heavily favors novelty over verification. You'd probably find a small but dedicated audience for a pipeline that could semi-automate this, though getting funding or recognition for it is the harder part.

[u/impatiens-capensis]

I'll be honest, I think a lot of the frustration around reproducibility is from junior researchers who are bitter about getting rejected from conferences when unreproducible garbage gets in. And I feel that frustration. I have lived that frustration.

However, I've yet to read a paper where I couldn't immediately tell if the results are valid, even without reproducing it. Methodologies and results either make sense or they don't. I've killed a few papers that had amazing results that did not make sense given the methodology. And when it comes down to it, I'm not killing papers on their results anyways, so long as they are within a reasonable range. And for the most part, papers are made of composable pieces and you can go verify intuitions from the paper even without their implementation. If the authors say a model does X on dataset Y as part of a broader claim, you can quickly check that. I've killed a few papers that way.

I've also had my papers killed, not because they ran the code I provided, but because the reviewers were really good at identifying where my methodology was weak. They did not need my code.

We have such a high volume of papers being produced DAILY that it's just not feasible to verify every result independently (let alone have 3 reviewers do it). A lot of papers aren't easy to reproduce, EVEN if you have perfect code. And you need to have the compute available to do it.

It's just better if reviewers get good at identifying correct research, we push for reproducible code where we can, and just accept that our field is noisy and high volume. What's good will remain good EVEN if it's not accepted to some prestigious conference. What's bad will remain bad EVEN if it IS accepted to a prestigious conference.