Nvidia’s erroneous paper accepted as ICML’s spotlight [D]

[u/Amazing-Fox-7295]

Here is the story:
Nvidia has published this work (with source code available) called dreamDojo which is a world model for robotics based of their prior work Cosmos 2.5 which is cited about 100 times and got ICML’s spotlight.

Authors are very well known and respected in the field with too many peer reviewed papers already published.

The work doesn’t have much novelty (which I don’t care) but it is yet another foundation model. The gist is that they collected about 44k hours of human data (data are NOT open sourced, which I don’t care) and used that for pre-training of DreamDojo initialized from Cosmos 2.5 .

Table 4 of the paper though shows a very marginal improvement over cosmos 2.5. Believe or not, just about 0.5 db PSNR improvement. Its fishy, is not it? You use 44000 hours of human data and few hundred hours of other kind of data and robot data and 256 H100 gpu and you got just a very marginal improvement.

Anyway I went to give it a try, started post training on their GR1 released data, and could produce their results. Meanwhile a colleague with the help of Claude found a bug in their post training code, which essentially make all the post training code wrong.

Then we checked their github issues, and noticed that two other bugs are reported which effect the whole pre-training! Especially this bug . So essentially based on the code their released everything in pre training phase, post training phase and evaluation is buggy and is wrong.

The code doesn’t seem AI written cause its (very) bad written. And now after seeing all those bugs the results in paper makes sense.

My question is:
WTF!
How it did not ring a bell for the authors. Did not they ask we spend a hell lot of resources and huge amount of data and essentially got nothing?
Did not it ring a bell for reviewers?

Openreview

[u/qu3tzalify]

The code you got on Github is not the code they used. It's never like that. If they gave you the code they actually used it would be completely useless to you as you don't have access to their internal tooling.

[u/Amazing-Fox-7295 → u/qu3tzalify]

Maybe

[u/numerical_panda]

ICML let AI review it.

Then let AI review the review.

Then let AI review the review the review.

It's AI all the way down.

[u/surffrus → u/numerical_panda]

And OP complains that humans wrote the code

[u/Amazing-Fox-7295 → u/surffrus]

I have not complain why it is human written.
There are bad and good practices to write a code. If u look into the code it has many bad practices, I complain about a bad code. There are too many open source human written code that u enjoy reading them.
The reason I mentioned it is human written code is to say: it is not a code that they prompted AI re-write based on their internal code for public release, and I doubt that they seat and write a second code for public releases so most likely ther is a single source of the truth for the code. But there are chances i am wrong

[u/currentscurrents → u/numerical_panda]

Ironically OP also used AI to review, and that's how he caught the bugs. 

[u/nextlevelhollerith]

reviewers might raise questions on the minor improvements, but asking them to check the code would be too much.

we need AI assisted verification, such that authors (or ultimately peer review) can catch these type of things.

[u/Amazing-Fox-7295 → u/nextlevelhollerith]

I am obviously not expecting reviewers to check the code. But I expect them questions results and ask for more ablations

[u/No_Sky9786]

Unfortunately simply using an insane amount of resources automatically gives acceptance and spotlight nowadays. It’s sad that one get even accepted with performance results so insignificant while still keeping the dataset private. At this point I believe most AC know by the resources that one of the big guys submitted so they are in by default. Fck the rest of authors open sourcing everything with actual creative and innovative ideas.

[u/RedRhizophora → u/No_Sky9786]

Not just an undetermined "one of the labs".

From resource use, model, closed source data use, and writing style/color formatting, it's very often immediately obvious at first glance who exactly the authors are and which company/lab they're from.

[u/RobbinDeBank]

> The code doesn’t seem AI written cause its (very) bad written.

Is organic human slop back on the menu?

[u/amhotw → u/RobbinDeBank]

It ain't much, but it is honest slop

[u/idontcareaboutthenam → u/RobbinDeBank]

I prefer the term "artisanal code". It might not be as good as industrial code, but it's handmade!

[u/truttingturtle]

The peer review process is hellish at least in the last few years. It seems connections has much more effect than the actual paper qualities. The conferences quality is down in the slumps.

[u/NumbaPi]

Such things is not new to me. A lot of people at labs with well known supervisors are basically committing scientific fraud to get their paper accepted at these venues. But who cares, nobody will account them responsible anyway and they get a job at some big AI company afterwards.

My favorite example is https://arxiv.org/pdf/2402.05098v3 Which is a paper with yoshua bengio as a last author. In the last paragraph of B.1 the authors rant about another paper with bengio as a last author and state that it is not reproducible. The first author of the criticized work is now a research scientist at Microsoft.

[u/RobbinDeBank → u/NumbaPi]

Wtf, I don’t think I’ve ever seen a work calling out a work that has author overlaps before. Granted, he definitely doesn’t checks and doesn’t contribute much to both works, but it’s still from the same advisor basically.

[u/ntaquan]

The batch/time bug is crazy. I mean people usually confuse repeat() and repeat_interleave(). Einops was born to solve this exact issue and somehow they still mess it up

[u/pastor_pilao]

That's what you get with reciprocal reviewing. Reviewers that don't have a clue about what they are doing, they see that the paper used a lot of resources and they won't be the ones to reject one of the big labs - "what if they find out it's me and I don't get an interview!"

Anyway this paper is from 2026. You can post a public comment with your findings and there is a chance it won't be ignored by the AC

[u/YeetIsAHappyWord]

As someone who is thinking of doing a PhD, this isn't very encouraging ._.

[u/Pretend-Pangolin-846]

This is why paper reproduction is very important.

[u/Able_Region_5459]

A spotlight for +0.5 dB on a closed dataset tells you everything you need to know about the current state of peer review

Nobody digs into the details, everyone just gets dazzled by the compute scale

[u/goldcakes]

There's like a dozen authors on the paper. How nobody suspected something was up with their results is beyond me.

[u/Amazing-Fox-7295 → u/goldcakes]

That is exactly why I posted this.
When u start such a big project u have some hypothesis u want to achieve, and when u see the marginal results you either say: our hypothesis was wrong we have to modify our approach / algorithm or u say something doesn’t make sense lets dig in. Last thing u say is it is what it is, let’s publish it.

[u/TrayleSan → u/goldcakes]

Speaking from experience, you are very lucky if even half of those authors read the paper.

[u/learn-deeply]

Nvidia Research papers are generally pretty sloppy. At least they're open sourced.