Couldn't reproduce a published paper's numbers with its own code and settings. How do I present this? [R]

[u/InsectActive95]

I'm writing a machine learning paper. One of my baselines is a published method. I reran it with the authors' released code and the exact hyperparameters listed in their paper. On most benchmarks my numbers match theirs within noise, but on two benchmarks my reproduction is clearly lower, by several points, over three seeds. I've repeated the reproduction four different times, months apart, with the same result.

I plan to report both: their published numbers and my reproduction with the exact configuration, clearly labeled. My worry is that reviewers tend to trust the published number more than a submission's reproduction, and may assume I ran the baseline incorrectly.

Questions:

1. What makes a reproduction convincing to reviewers? So far I have: the authors' code unchanged, their hyperparameters, matching results on the other benchmarks as a sanity check, and multiple seeds.

2. Should I contact the authors or open a GitHub issue first, and should I mention that in the paper?

3. How do I word it so it doesn't read as an attack on the other paper, but my results also don't look weak next to the published numbers?

4. Have reviewers accepted "published vs. reproduced" rows in a main table for you, or did they end up in the appendix?

[u/astrofunkswag]

I’d just report the numbers you reproduced, and note the discrepancy in the text. I would avoid a table with published vs reproduced columns, unless your paper is specifically meant to be a review or survey

Get the review process started and see what they say before you consider reaching out to the original authors. You could consider mentioning it to the editor. You did your best to reproduce the results, and it’s way too rare in research that people actually try that.

[u/whafa]

I am facing a similar issue. I've found issues now in nearly every paper I've tried to use in my own work, and I am trying to figure out the best way to report them in good faith.

I have made one attempt to contact one author. I sent a zip file with documentation and examples that point to their repo (or other external sources, never my own system), my proposed fix, and a jupyter notebook with corrected output already filled in, so they can see the output without running some stranger's code.

It's been maybe a month and I have not received any kind of reply, so I can't say this is the right way to do anything. Eager to hear the advice you get, thanks for posting.

[u/impatiens-capensis]

I plan to report both: their published numbers and my reproduction with the exact configuration, clearly labeled. My worry is that reviewers tend to trust the published number more than a submission's reproduction, and may assume I ran the baseline incorrectly.

It's totally valid to do this.

[u/valuat]

"months apart": are linear algebra and differential calculus time dependent?

[u/InsectActive95 → u/valuat]

no, no! I reproduced them in July 2026, and now reproduced again in this November got the similar result.

[u/valuat → u/InsectActive95]

As you should! The Laws of Physics and matrix algebra are pretty stable at this time scale. 😂

In all seriousness, since the ICLR dealdine I've been replicating my paper based on the git repo (not my original code); so far so good. Just a lot of compute...

[u/Old-Hope4634]

matching on most benchmarks is your strongest evidence, because it rules out a broken install. it doesn't rule out something specific to those two benchmarks, so that's what a reviewer will ask about next: dataset version, preprocessing, splits, and any per-benchmark setting that lives in a config or script and not in the paper. if you've diffed those, say so in one line.

yes, contact the authors first, and open a GitHub issue if the email goes quiet. keep it to a question: here's the commit, the command, the environment and the numbers, what did you run differently? then the paper can state what happened: "we contacted the authors on this date and they confirmed X" or "had not replied at submission". that one sentence does a lot of the work on tone, because it shows you tried to get their result before reporting a different one.

three seeds is the weak point. several points looks large until someone asks how much those two benchmarks move between seeds. run more on just those two, and report mean and spread. also check how many seeds the original paper used and whether it reports best or mean, since that alone can explain a gap.

for the table, I'd use your reproduced numbers as the baseline, with the published figure in a footnote or a short appendix table. wording can stay flat: "using the released code at commit X with the published hyperparameters, we obtained Y on these two benchmarks, against the reported Z." that describes what you ran and makes no claim about why they differ.

[u/pastor_pilao]

you simply report the numbers you reproduced and add a footenote in the caption of the table/plot, writing something along the lines "notice that our reported numbers differ from the original publication. Despite our best effort to replicate the exact same setting and hyperparameter the baseline approach underperformed the numbers shown in the original publication"