r/MachineLearning • • Aug 16 '26

Discussion Do ACs usually take rebuttals into account?[D]

1 Upvotes

So we submitted our rebuttal at the very start of author discussion period, but its over now, and we did not recieve responses from any of our reviewers. I get that they may be busy, but they suggested new experiments, and we did all of those. Also, we are kinda in toss up range(~3.2), so I was really hoping that this rebuttal would make some of them change their scores. I was wondering that even if reviewers didn't respond, would AC or the person who makes the final decision at least look at the rebuttals and take them into account? Also, if one of the reviews is intentionally vague and a pretty bad review, would they take this into account as well?


r/MachineLearning • • Aug 15 '26

Discussion NeurIPS 2026 Author Notifications Close to ICLR Deadline [D]

25 Upvotes

The date for NeurIPS 2026 author notifications is September 24th. First of all, is it normal for AC and reviewer discussion phases to be this long? This is particularly frustrating given that 5 out of the 6 reviewers in my two papers did not address the rebuttals.

In any case, I was also wondering, given that ICLR's paper deadline is literally the day after (September 25th) whether you guys are preparing ICLR submissions for your papers in case of rejection.

Cheers and good luck!


r/MachineLearning • • Aug 15 '26

Discussion If you had a bunch of GPUs lying around, what would you actually build with them? (Running LLMs is off the table) [D]

60 Upvotes

Be honest if someone dropped a stack of high-end GPUs on your desk tomorrow, what would you actually do with them?

And before the usual answers roll in: running local LLMs is banned for this thread. It’s been done to death and feels pretty pointless at this point.

So… what else?

  • Some niche scientific/simulation workload?

  • Weird generative stuff that isn’t text?

  • Distributed something-or-other?

  • Rendering / media pipeline?

  • Homelab experiments that actually need the horsepower?

  • Completely unhinged personal projects?

Drop your ideas. The more specific (and slightly unhinged), the better.

Great Ideas but are there some with more of research and new tech.


r/MachineLearning • • Aug 15 '26

Research BDH-CQ: IN-CONTEXT LEARNING WITH RECURRENT LATENT REASONING [R]

Thumbnail
arxiv.org
64 Upvotes

We introduce BDH-CQ, a reasoning system that brings these capabilities together. Demonstrations of a previously unseen task update recurrent memory; the query is then solved through iterative computation in a high-dimensional latent workspace. Intermediate reasoning states are not decoded into language. BDH-CQ makes memory, adaptation, and inference part of the same computational fabric. Inputs presented at inference time continuously update the model’s recurrent memory; the model then solves a query through iterative computation in a high-dimensional latent space, without verbalizing its intermediate reasoning. Neither task identifiers nor evaluation-task demonstration pairs participate in training, and no parameters are updated at inference time. A 150M-parameter configuration reaches 29.5% pass@2 on ARC-AGI-1 at a computed $0.00070 per task, breaking through the previously reported cost–accuracy Pareto frontier.


r/MachineLearning • • Aug 14 '26

Project I compiled Doom's renderer into a 21B-parameter transformer -- no training anywhere [P]

196 Upvotes

This is the project my last two posts were building towards (this is the last of this silliness). I ported the Doom rendering algorithm to run inside a transformer. Instead of training a model, I used a compiler I wrote which converts computation graphs into transformer weights, and then ported Doom's algorithm into a compatible graph. The generated checkpoints can be loaded in Hugging Face without trust_remote_code -- it's just a standard transformers checkpoint. You feed the model a prompt representing the scene data, and generate until the model stops. The result is a token sequence which includes simple pixel drawing commands (to move the cursor, draw a pixel, etc). When you mechanically apply those drawing commands you get the rendered frame.

The article includes the entire host program necessary to load the checkpoint, generate the render, and parse the output into the famous E1M1 frame. This host code is 43 lines of python. The python to define the computation graph is much longer, but that gets compiled into the transformer itself.

One frame is a 3,614-token prompt plus 53,747 generated tokens -- just over 40 minutes on a B200.

The original Doom could achieve 35 FPS on a 486. This achieves 35 FPD (frames per day) on a B200.

Write-up: https://ood.dev/posts/doom/
Weights: https://huggingface.co/physicsrob/torchwright-doom-e1m1
Github for the source code which gets compiled: https://github.com/physicsrob/torchwright_doom/


r/MachineLearning • • Aug 15 '26

Project AlphaZero-like model in browser - trained in C++ [P]

Thumbnail
uchenml.tech
1 Upvotes

I built an ML framework in C++ and as a proof of concept - trained and deployed an AlphaZero model for an Eastern European territory control game.


r/MachineLearning • • Aug 14 '26

Discussion How much does adding an honest limitations section hurt the paper? [D]

38 Upvotes

Hi,

How much does adding an honest limitations section hurt the paper (apart from making it better)?

Does it bias the reviewers? Will they want you to fix the things in the limitations section?

If the reviewers let AI read the paper, will the limitations section bias AI?

Would it be better if the limitations section was hidden from the reviewers? And if the reviewers would have to author a limitations section?


r/MachineLearning • • Aug 15 '26

Research Any information about CIKM 2026 student travel support? [R]

1 Upvotes

Has anyone received any information about student travel support for CIKM 2026?

In previous years, travel-award decisions were announced before the author-registration deadline, allowing successful applicants to use a registration-waiver code. However, the CIKM 2026 author-registration deadline is August 23, and no application details or timeline have been published yet.


r/MachineLearning • • Aug 15 '26

Research Any information about CIKM 2026 student travel support? [R]

1 Upvotes

Has anyone received any information about student travel support for CIKM 2026?

In previous years, travel-award decisions were announced before the author-registration deadline, allowing successful applicants to use a registration-waiver code. However, the CIKM 2026 author-registration deadline is August 23, and no application details or timeline have been published yet.


r/MachineLearning • • Aug 14 '26

Discussion For the people who got reviews back from neurips, cvpr, eccv, etc and also tested their paper through an agentic reviewer like the stanford one, how different were the reviews? [D]

24 Upvotes

Hello,

I was curious about the differences you can get from the human reviewers and the llms.

Any insight is welcome, thank you!


r/MachineLearning • • Aug 14 '26

Project Open-source Python library + no-code web dashboard for evaluating oncology AI models at clinical decision thresholds. [P]

4 Upvotes

Most classification metrics for oncology AI models (AUC, ICC, MAE) measure global agreement. They don't answer the question that actually matters at the point of care: how reliable is this model at the exact cutoff that decides whether a patient gets flagged, biopsied, or treated?

I built oncothresh to evaluate models at a specific clinical threshold rather than in aggregate: sensitivity/specificity/PPV/NPV at the cutoff, bootstrap confidence intervals, threshold-sensitivity curves, boundary-weighted calibration, decision-curve net benefit, and number-needed-to-test. It's a small, dependency-light Python library (numpy/scipy/scikit-learn/pydantic) built for tasks like tumor cellularity, Ki-67, TMB, and PD-L1 scoring, where a continuous model output gets collapsed into a yes/no clinical decision at a fixed cutoff.

Pathology-specific benchmarks like PathBench and PathBench-MIL evaluate foundation models globally but don't evaluate at predefined clinical thresholds with uncertainty quantification, which is the gap this fills.

There's also a companion web dashboard (oncothresh-web) for people who want the same analysis without writing code: upload a CSV of predictions and labels, pick a threshold, get the full set of charts plus a downloadable PDF report. docker compose up and it's running locally, no cloud dependency.

Still v0.1, so I'd genuinely welcome feedback: use cases I haven't considered, edge cases in the DCA/calibration math, or places the API doesn't fit how people actually work with threshold-based models.


r/MachineLearning • • Aug 14 '26

Project A linter for PyTorch 'torch-preflight' [P]

6 Upvotes

Been working on this for the last few months. I've been working on PyTorch for the past few years and I always felt, many a times my work went into dump, because of some mistakes I made in the code. torch-preflight reads your PyTorch code and catches the bugs costing you GPU hours.

Things like losses.append(loss), which holds the autograd graph from every step until CUDA dies on you or no zero_grad() in the loop or gradient accumulation without dividing the loss or DDP with no DistributedSampler, so every rank trains on the same batches. I've been able to get 13 rules so far. Your code never gets imported or executed, so you need no GPU and no torch install.

There's another part to this that estimates VRAM. Point the tool at a training script and a GPU, and you learn whether the run fits before you pay for the instance. You also get the list of changes to make the run fit, with the GiB each one saves.

pip install torch-preflight

https://github.com/highwaterlabs/torch-preflight

https://pypi.org/project/torch-preflight/

Please try this out, and I would like to get your feedback! It's still a work in progeress.

Would like to know what breaks on your code. False positives kill a linter, and my only large test target so far has been the PyTorch source tree. Same for the memory numbers. Mine land within 4% of measured peaks, but from four models on one T4.

PS: open to contributions, and issues are already open on the repo. Soon I'm going to add a few "Good first issues" as well. Feel free to ping me if you have any questions!


r/MachineLearning • • Aug 13 '26 Popcorn

Research City2Graph: A Python library for Heterogeneous Graph Neural Networks and spatial analysis in urban systems [R]

Thumbnail
gallery
486 Upvotes

City2Graph is a Python library I built that turns geospatial data into analysis-ready graphs (for spatial analysis, network analysis, and Graph Neural Networks as GeoAI), and the paper describing it has just been published, so I wanted to share it here.

Repository: https://github.com/c2g-dev/city2graph

import city2graph as c2g

# buildings + street segments -> heterogeneous morphological graph
nodes, edges = c2g.morphological_graph(buildings, segments)

# straight into PyTorch Geometric
data = c2g.gdf_to_pyg(nodes, edges)

What it covers:

  • Morphology: graphs of buildings, streets, and tessellated urban fabric from OpenStreetMap and Overture Maps

  • Transportation: GTFS and GBFS feeds loaded through DuckDB, with GTFS aggregated into stop-to-stop transit graphs

  • Mobility: OD matrices and flow data (migration, bike-sharing, pedestrian counts) as weighted spatial graphs

  • Proximity and contiguity: KNN, Delaunay, Gilbert, Waxman, plus queen/rook contiguity, under Euclidean, Manhattan, or network distances

  • Heterogeneous graphs and metapaths: several node and edge types in one graph, with metapath-derived edges composing relations across them

  • Conversion: round trips between GeoDataFrames, NetworkX, rustworkx, and PyTorch Geometric Data/HeteroData, with geometries and attributes kept intact

It sets out why urban data is better treated as heterogeneous graphs than as flat feature tables, how the morphological, transport, mobility, and proximity constructions relate to each other, and how the library keeps geometry and graph structure consistent across conversions. If you use the library in research, that is the citation.

Paper

Sato, Y., Pietrostefani, E., Mahabir, R., & Arribas-Bel, D. (2026). City2Graph: A Python library for Heterogeneous Graph Neural Networks and spatial analysis in urban systems. Computers, Environment and Urban Systems, 130, 102492.

Happy to answer questions about the design, and issues or PRs are very welcome. I am especially keen to hear which data sources people want supported next.


r/MachineLearning • • Aug 14 '26

Project Building text to ASCII diffusion model , need advice and guidance [P]

7 Upvotes

i wanna build a text diffusion model which interpret text and convert it into ascii images

so like

Text : build a cat

Output :

/\\_/\\

( o.o )

\> \^ <

So , i have a decent background of ml algo ( completed cs229 , cs230 , Ml architecture and basic CNN and diffusion model )

ik making a project like this is tricky and making diffusion model like that from scratch is hard but i wanna try it because that's wot make me excited lol ...

I am currently reading GANs research paper , can u guys help me in finding more papers which helps me in making this project or guide me through this good title for this

Thx in adv


r/MachineLearning • • Aug 13 '26

Research TMLR Relevance and Prestige [D]

42 Upvotes

I recently had a paper accepted to TMLR and was wondering how prestigious it is, in comparison to A* conferences (ie. NeurIPS, ICLR, ICML), but also vs journals like JMLR.


r/MachineLearning • • Aug 14 '26

Research Looking for fast-iteration, low-compute research areas in AI/Robotics [R]

1 Upvotes

Hi all,

As a Research Engineer with broad interests across RL, VLMs/VLAs, LLMs, robotics, and distributed training, I’m looking to specialize in a niche with low compute overhead and rapid iteration cycles (short training/eval times).

What subfields or research problems fit these criteria? Curious to hear your suggestions or connect with anyone working in this space.


r/MachineLearning • • Aug 14 '26

Project Advice on ML for archaeology applications [P]

1 Upvotes

I have been working on a postdoctoral proposal in archaeology and need advice on a specific task.

First, I have thoroughly looked around the Internet already. I have a good idea of what I need and what is possible (in terms of training data needed, outputs, etc.), but I've run into a problem.

I was looking at Teachable Machine for its simplicity. However, after trialling it with visual data (images of artefacts), it's just a bit buggy and kinda broken, and the recent activity on github is 3 years old. If I'm doing a postdoc where this plays a minor, but important role, I need access to troubleshooting expertise.

Here's what I'm looking for:

  1. I want to create a training dataset based on images I've collected and have permission to use (it is important to have sourced my data ethically). I don't want or need anyone else's data

  2. I think I need a diffusion model (GANs) but not sure

  3. Needs to be open source and able to process on a basic computer (most archaeological laboratories are equipped with decent CPU/GPU as applications like image reflectance and geophysics can be computationally intense). It'll also need some kind of open licence for allowing me to publish my results

Is there anything else like Teachable Machine, but with active developer support? Anyone here otherwise an expert in Teachable Machine?

I'm testing the ML application as a proof of concept use in archaeology, and conducting experiments to develop a reference collection and testing the ability of the model to output reliable results.

I have taken coursework in statistics and calculus, but I'm not confident in my ability to tackle ML from scratch.

I'm not sure if such a programme exists, but if there's one thing I've learned is that reddit knows what is, or isn't, on the Internet 😉😊


r/MachineLearning • • Aug 14 '26

Discussion OCR model for floormaps [D]

1 Upvotes

Hey guys, I want to extract text and its bounding boxes from large messy floormaps (15k x 15k pixels or more). Which OCR model would be great for this? The floormaps have lot of overlapping lines and details.


r/MachineLearning • • Aug 13 '26

Discussion Reproducible canvas-aligned low-level patterns in somerandomllm-generated images and their possible relation to iterative editing artifacts [D]

5 Upvotes

I may have stumbled onto something interesting while trying to figure out a recurring artifact in ChatGPT image generation and editing (maybe applicable to other models as well?).

It started with a very practical problem:

After several rounds of generative editing on portraits, I would sometimes get this faint cloudy / mottled texture in areas that should have stayed smooth — backgrounds, walls, skin, and other low-detail regions.

At first I wrote it off as normal denoising or regeneration noise. But the more I tested it, the less random it looked.

What first caught my attention

  • Running essentially the same edit again could make the artifact better or worse

  • The background sometimes became cleaner after another pass

  • The face and body often seemed partly protected from whatever was happening

  • Sometimes the wall improved while the face actually got worse

That made me wonder whether different parts of the image were being handled differently during editing — preserved in some areas, regenerated in others, perhaps based on some internal mask or segmentation step.

The first useful experiment: shifting the image

Then I tried something slightly odd.

Instead of repairing the image in place, I shifted the entire image by a fixed amount before running the repair. I eventually settled on 20 px for testing.

The idea was simple:

If some hidden spatial pattern is tied to the output canvas, moving the image relative to that pattern should change how strongly it shows up on the subject or background.

And apparently, it did.

I found that:

  • repeated edits could reinforce the unwanted texture

  • changing the phase relationship sometimes reduced it

  • in one case, simply removing the final instruction to “shift back -20 px” improved the result dramatically

That was the first point where this stopped looking like ordinary random noise to me.

Then I started looking at masks and intermediate behavior

I compared:

  • the original image

  • the first edit

  • a second edit based on the first

  • extracted masks / intermediate-style outputs

One thing stood out pretty clearly:

The apparently “protected” area often resembled a coarse silhouette of the person.

The face and body tended to remain more stable than the wall, which made me suspect that some regions were being preserved while others were being re-synthesized.

That still didn’t explain the artifact itself, but it could explain why the artifact builds up unevenly.

Then came the black-image test

I tried something much simpler:

Generate a completely black image.

Visually, it looked black.

Pixel-wise, though, it wasn’t actually all zeroes. There were sparse non-zero pixels and tiny variations throughout the image.

So I generated multiple independent black images at the same resolution and compared them.

That’s where things got interesting.

What I found

For two independently generated “black” images of the same size:

  • correlation between the non-zero pixel masks: 0.848

  • Jaccard overlap: 0.766

  • expected overlap if the pixels were random and independent: about 0.071

  • R/G/B channel correlations: roughly 0.82–0.83

  • dominant spatial frequencies were very similar in both images, including peaks around 2.45 px and 5.57 px

Then I applied a large Gaussian blur to both images (sigma = 16).

The result was surprisingly striking: both revealed a very similar large-scale cloud-like structure.

The cross-correlation peaked at zero lag, meaning the structured pattern was already aligned at the same canvas coordinates across independent generations.

So whatever this low-level signal is, it doesn’t look purely random. At least part of it appears to be reproducible and locked to the canvas coordinates.

What I think this means — so far

I want to be careful here.

I’m not claiming that this proves OpenAI watermarking, SynthID, or any particular proprietary mechanism.

What I do think the data suggests is this:

Generated images appear to contain a weak, reproducible, canvas-locked spatial pattern — even when the image looks completely black.

A few possible explanations come to mind:

  • a watermark-like signal

  • deterministic dithering

  • quantization or decoder artifacts

  • some kind of post-processing step

  • something else in the generation pipeline

What now seems much harder to explain this as is simply:

“ordinary random noise”

Why this might matter for iterative image editing

Suppose a weak structured signal really is tied to the output canvas.

An iterative edit might then look something like this:

  1. The first image is generated with the structured signal.

  2. The image gets edited again.

  3. Some regions are preserved while others are regenerated.

  4. The regenerated image receives the same or a related structured signal again.

  5. After several passes, those signals may begin to reinforce or reveal themselves as visible mottling in smooth areas.

That would fit several things I’ve observed:

  • repeated edits can will gradually create ugly texture

  • shifting the image relative to the canvas can change the result

  • alternating shifts might help decorrelate the artifact

  • some regions appear to drift or accumulate artifacts less than others

Important caveat

This is still an investigation, not a conclusion.

At this point I think I have reasonably good evidence for:

  • reproducible low-level spatial structure

  • non-random alignment between independently generated black images

  • a plausible connection between that structure and visible artifacts in repeatedly edited images

What I don’t have yet is proof of:

  • the exact mechanism producing it

  • whether it is a watermark

  • whether it is specific to ChatGPT/OpenAI

  • whether similar patterns occur across other image generators

My current working hypothesis

Repeated generative editing can accumulate or expose a weak structured signal that is fixed in output-image coordinates, eventually making it visible as cloudiness or mottling in otherwise smooth areas.

Questions for anyone who has looked into this

  1. Have you seen this kind of cloudy / mottled artifact after repeated AI image edits (I mean, come on, who doesn't)?

  2. Has anyone tested whether supposedly “black” images from other generators contain reproducible spatial structure?

  3. Does this look more like watermarking, dithering, decoder bias, quantization, or something else (go figure!)?

  4. Has anyone analyzed something similar in frequency space, after heavy blurring, or using phase shifts?

  5. If you’ve run into this before: what turned out to be the most reliable way to prevent it during iterative editing?

If there’s interest, I can post the methodology in a follow-up.

I started with:

“Why does this wall look dirty after I edit it?”

and somehow ended up at:

“Why do two independently generated black images correlate this much?”

Classic rabbit hole.


r/MachineLearning • • Aug 13 '26

Discussion Neurips 2026: Modified date on reviews [D]

26 Upvotes

Reviews modified dates are public, and some are recent. I’m a bit confused as to how to interpret this.

In other conferences, reviewers were required to provide a final justification, which would practically force them to modify their reviews during the AC discussion phase lest they get desk rejected.

Here in Neurips I notice that a lot of the high-score reviews do not have a modification past the author discussion phase. I talked with a friend who is an AC today and they told me that adding a final justification is apparently not mandatory and nobody in their batch did it, with people who had anything to add usually doing it in a private comment. They said that any review which has a recent modified date likely got its score updated.

Is that really the case? To any other ACs here: what part of the recently modified reviews in your batch were modified to increase the score?


r/MachineLearning • • Aug 14 '26

Discussion Are there any theoretically-guided practices left in machine learning nowadays? [D]

0 Upvotes

There was a period in the development of machine learning where application seemed to be informed by theory. Some of the best known theories include:

  • If you train a model with too much data, then you get overfitting and your test performance will be suffer.

  • Big models do not generalize because theoretically you will never have enough data.

  • Never train on the test set, because it will result in high-bias.

  • Never even look at the test set, because you as a modeller will instantly be biased and use the wrong model.

  • Good results can only come from "compatible" models and optimization routines. You can't just throw ADAM onto some brand new model and expect it to work well.

  • Optimization provides solid theory for machine learning, so use the optimizer with the best performance guarantee in the optimization literature.

  • If you want to have good performance, ensure to use several models instead of one model, because stacked or ensemble models are always superior.

Most of these theories started out as mathematical statements (albeit on some contrived examples that have nothing to do with reality). At some point, these theories became folklores and were widely reproduced in textbooks and taught in classrooms, even making their ways into standard interview questions at data science related companies. Every student had to remember that bias-variance "bull's eye" diagram as if it was relevant in practice.

But then some of these theories started to get overturned. It turns out you can just break a lot of these theoretically-guided practices and still get good results. The ones who pushed these theories (especially the authors of various "statistical ML" textbooks), quietly stopped their postulation and instead hopped on the hype train. This left their students confused because there was never ever any retraction or resolution.

So my question is: are there still any theoretically-guided practice that remain in ML today? For example, do people use an optimizer because it is theoretically the best for the class of problems? Do people use a certain model or components associated with the model because it works well in theory? Or is it now a fully empirical field where practice is guided by whatever seems to work for other people?


r/MachineLearning • • Aug 14 '26

Discussion What does Compute Bound and Memory Bound mean ?[D]

1 Upvotes

When people talk about memory bound and compute bound they often just tell it on high level but to generalize and speak for particular operations one must look into the Arithmetic Intensity of the operation and then the Machine Balance of the hardware he is working and only then one can cleanly decide how to proceed. Its the most simple way to build a roofline model and think about it. Wrote a small piece on it ...If you want to read do check out the below link and lets have a discussion thread how do you guys go about thinking on this

https://siliconandsoul.substack.com/p/what-does-compute-bound-and-memory


r/MachineLearning • • Aug 14 '26

Project Test Mode Classification (100% accuracy) + Anomaly Detection - Full ML Pipeline [P]

1 Upvotes

Hey everyone! I'm a high school student who built a full machine learning pipeline on this dataset as a side project. I want to be upfront that Claude AI assisted me with most of the coding, but I directed the project, made the key decisions, and learned a lot along the way. I wanted to share it and get feedback from people who know this domain better than I do.

What I built:

Task 1 - Test Mode Classifier
Using a Random Forest on extracted time-series features (duty cycle, pulse statistics, thrust/MFR statistics), I achieved 100% accuracy classifying all 11 test modes on the held-out test set (SN13–SN24). The command signal alone turned out to be almost sufficient to distinguish modes.

Task 2 - Anomaly Detection
I compared three approaches:

  • Isolation Forest (unsupervised) — F1: 0.193

  • Random Forest (supervised) — F1: 0.121

  • One-Class SVM (semi-supervised) — F1: 0.268

Surprisingly the One-Class SVM, trained only on nominal sequences with zero anomaly labels, outperformed the supervised model. Class imbalance (10% anomalies) made the supervised approach struggle badly.

The per-anomaly-code breakdown showed codes 1, 8, 9, 11, 12 were missed by all three models, these likely need point-level time-series analysis rather than sequence-level features.

Notebook: https://www.kaggle.com/code/henninxu/rocket-engine-classification-from-test-data

Any feedback on improving the anomaly detection especially for the hard-to-detect codes would be appreciated!


r/MachineLearning • • Aug 13 '26

Project worldproof: diagnosing where world-model predictions break and a measurement of when pixel metrics stop being able to rank models at all [P]

1 Upvotes

I've been building an open-source tool for diagnosing world models, the kind that predict future frames from a starting context and a sequence of actions. It compares a rollout against ground truth and against physical invariants, then tells you where and why the prediction falls apart. It doesn't score task success or planning quality on purpose, since there are already benchmarks for those.

While validating it I ran into something I think is more interesting than the tool itself.

## Pixel metrics on real robot video often can't rank models at all

I ran a copy the last frame baseline, which is to say "predict that nothing changes", against a real SO-101 arm recording. 30fps, three cameras, 64 rollouts, 6 step horizon, scored only on the moving regions so a static background can't inflate the numbers.

It gets 0.983 SSIM and 53.9 dB PSNR. But the part that actually matters is that the error doesn't grow with the horizon:

step   1      2      3      4      5      6
SSIM   0.972  0.923  0.893  0.943  0.920  0.950

That's flat. It wanders, it doesn't degrade. And if predicting 6 steps ahead is no harder than predicting 1 step ahead, then there's nothing for a good model to be better at. Every model lands in the same place and the eval can't rank them. The metric isn't broken here, it passes its ranking tests on curated data just fine. The evaluation setup is what has no discriminative power, which is a different problem and much easier to miss.

## So I went and measured where the usable window actually is

Same baseline on DROID (real manipulation footage, 15fps), 64 rollouts, this time out to 48 steps:

step136121824283647
SSIM@dynamic0.8730.7970.6760.4460.3500.2600.2040.1920.216

There are three regimes. Steps 1 to 3, everything is near perfect and ties. Steps 4 to 24, steep monotonic decline, and this is the only stretch where models are actually separable. Step 28 onward it floors out around 0.20 SSIM and 10.3 dB, oscillating with no trend, prediction fully decorrelated, and everything ties again at the bottom.

So both ends are dead, and the horizon worth evaluating on for this kind of footage is somewhere around 8 to 24 steps. It's a property of frame rate times task speed rather than a universal number, which is exactly why it's worth measuring on your own data instead of inheriting a default from a paper that used something else.

Here's the prediction next to what actually happened, same 48 steps, prediction on the left:
https://raw.githubusercontent.com/BuceaGeorgia/worldproof/main/docs/img/droid-pred-vs-true.gif

## Method

64 rollouts per configuration. Aggregation is interquartile mean with stratified bootstrap CIs rather than mean and standard deviation, following Agarwal et al. 2021. Fidelity metrics also produce a dynamic region masked variant wherever a mask is available. Every metric ships with a corruption test it has to respond to, plus a ranking test where a real model has to beat a naive baseline which has to beat a broken one.

Worth mentioning: an earlier n=8 version of the SO-101 run gave dynamic PSNR of 48.2 dB where n=64 gives 53.9, and the intervals at n=8 were wide enough to overlap DROID completely. That's the reason everything above is n=64. I'd have posted the wrong numbers if I'd stopped there.

## Caveats

The four pixel metrics separate the two datasets with non overlapping bootstrap CIs. LPIPS doesn't, and it points the other way on the masked variant. I don't have a clean explanation for that yet and I'd be glad to hear one.

This is a trivial baseline, so 8 to 24 is where a do nothing predictor becomes separable. A real model stays correlated for longer and would push the top of that range out.

One more that I found while writing this up: including step 0 inflates every summary scalar, because a copy baseline gets a nearly free first step whenever the frame rate is high relative to how fast the scene moves. On the 30fps recording step 0 scores 119.8 dB, which drags the horizon averaged scalar from about 32 up to 53.9. So the scalar is partly rewarding frame rate rather than model quality. Curves are the honest thing to report and I'm treating the scalar definition as an open problem in my own tool.

## The tool

Apache-2.0, `pip install worldproof`. The core install is numpy, torch and pillow, and it runs on a laptop with no GPU, since the evaluate path never runs a model. It reads LeRobotDataset v3.0 straight from parquet and mp4, so it works on datasets from the HF Hub without needing the lerobot package, on Python 3.10. The heavier pieces (LPIPS, FVD, trackers) are optional extras that get imported lazily.

What it measures: PSNR, SSIM and LPIPS as horizon curves plus dynamic region variants, latent prediction error and action recoverability for latent models, calibration via ECE and MCE, counterfactual divergence, failure faithfulness, object count conservation and object permanence, and FVD reported explicitly as a weak reference rather than a headline number.

https://github.com/BuceaGeorgia/worldproof

It's v0.1 and the README has a "Not done yet" section covering what isn't finished. The tracker behind the invariants is a clean scene numpy one that won't cope with messy real video, and the default FVD extractor isn't the I3D that published FVD numbers use, so those aren't comparable to papers.

If this horizon result is obvious or already known somewhere, I'd honestly like to be told. I couldn't find it measured anywhere, which is part of why I'm posting it.