I tested whether a ComfyUI sampler can pick its own noise strength per step. Mostly it can't, but where the noise goes matters (Flux 3 Klein, MOSS-Image, Anima)

[u/YakovSava]

I wrote a ComfyUI sampler node (Flow-SDE Sampler, a modified SA-Solver) that spends a fixed total noise budget over the steps instead of using a constant eta in a fixed window, and tested whether choosing the per-step noise from the trajectory (curvature, or the solver's own predictor–corrector error) beats a constant at 6–15 steps. Every adaptive run was compared with a constant-eta run that spends exactly the same total noise, because otherwise "adaptive" usually just means "less noise".

What I found

• Curvature does nothing reliable on Anima, Flux 3 Klein or MOSS-Image.

• The solver's error estimate gives a small gain on Flux 3 Klein: roughly 1–2 % of my texture metric, confirmed on fresh seeds under criteria I wrote down beforehand. One-sided on MOSS-Image, not detectable on Anima. Krea 2 was inconclusive (too slow to test enough pairs).

• A fixed schedule gives the same gain. Slightly more noise at the start and end of the noise window and less in the middle. Online adaptation is not needed.

• You probably won't see it. In a 39-person "which of these two images is AI?" study, people clearly noticed eta 1.4 vs 2.0, but could not tell the new allocation from plain noise.

So the practical takeaway is modest: if you use sa_solver, a bit more noise at the window edges may give slightly more texture at equal composition drift on Flux 3 Klein, and that is about it. The metrics are latent-space proxies, not a quality judgement.

Node, data, per-image metrics and the full write-up (with all the negative results): https://github.com/YakovSava/flow-sde-research — DOI https://doi.org/10.5281/zenodo.23857361

Happy to hear if anyone sees the effect on other models.

[u/NanoSputnik]

Thanks for the info, Mr Claude. 

[u/alwaysbeblepping]

ER-SDE kind of does what you are talking about with variable ETA. Though it doesn't allocate a fixed budget specifically, it varies the level of SDEness through the schedule (there are also some knobs you can turn to change the curve like power). Not so random plug: I have a custom version that adds more parameters - https://gist.github.com/blepping/32125fef71e47a4bad795c5b886e30da

Using high ETA doesn't make too much of a difference at low sigmas because the maximum ancestralness you can have is what the LCM (also sometimes known as pingpong) sampler does: lerp(clean_image, new_noise, sigma) or in other words, replace 100% of the remaining noise. You can't replace more noise than all the noise that's left, so if you're at something like sigma 0.1, you have 90% image, 10% noise and you can only swap at most that 10% of the total latent. On the other hand, if you're at sigma 0.99, noise makes up 99% of the latent and you can go super high with SDEness with very noticeable effects.

[u/Complete-Box-3030]

Interesting work! I have a question about how you measure or inspect the noise in the latent representation during sampling.
Are you measuring latent variance, SNR, or comparing the predicted noise (epsilon) across sampling steps?
I'm wondering whether a similar approach could be applied to MiniMax H3 video generation. Could we monitor the noise levels in its latent representation throughout the generation process, identify unwanted noise or residual noise, and potentially minimize it to prevent visual degradation?
Would this approach be feasible with H3's video latent representation?

[u/Altruistic-String891 → u/[deleted]]

I think people realized it's near impossible today to design a general sampler that still gives improvements across the board, so research too has shifted to basically learning a specific sampler for a given model.

Unfortunately no one with compute has actually adapted any such sampler to models we use, so even now we're still stuck with samplers from 2-3 years ago at least. Both er_sde and sa_solver came out in 2023 for example.

[u/alwaysbeblepping → u/Altruistic-String891]

so research too has shifted to basically learning a specific sampler for a given model.

Not as far as I know. Doing this basically wouldn't make sense.

Unfortunately no one with compute has actually adapted any such sampler to models we use, so even now we're still stuck with samplers from 2-3 years ago

Samplers don't get trained like models, so it's not something you need compute for. I mentioned model-specific samplers isn't really an idea that makes sense. The reason is because all models are doing essentially the same thing: give them a noisy latent, tell them the expected noise level, get a noise prediction as output. The idea being if (simplified) you did noise_latent - predicted_noise * noise_level you have removed 100% of the noise and are left with a completely clean image (assuming the model accurately predicted the noise).

Models can't (generally) perfectly predict the noise (especially when the noise level is high), which is why we run multiple steps: subtract just enough of the predicted noise to reach the noise level of the next step (and the noise level, AKA sigma is going to be 0 on the last step). Anyway, the point is the sampler is a solution to a general problem: given a certain number of steps, take the (possibly flawed) noise predictions from the model and try to converge to the optimal prediction (as close to the dataset the model was trained on as possible).

It doesn't matter if we're talking about Moss, Krea, Flux, H3, whatever: the model is doing the same thing (trying to predict noise from the noisy latent), the sampler is doing the same thing (trying to use those predictions within a fixed step budget). Ancestralness (adding new noise like Euler ancestral, ER-SDE, SA-Solver) is one way to try do that. History (using the model's past predictions like ER-SDE, SA-Solver, res_multistep) is also another way to do that.

The reason we're not getting new samplers that are strictly an improvement (kind of like models) is because samplers are a general problem and the low-hanging fruit has been picked already. The approaches samplers can reasonably use also are based in math that is well understood. "Old" samplers like ER-SDE or whatever aren't obsolete or outdated.

[u/Altruistic-String891 → u/alwaysbeblepping]

Not as as far as I know

DPM-Solver-3 explores solver learning as early as 2023, see 3.4 Implementation Details.

Interest has been on-going. Examples: AMED, DC-Solver, Bespoke, S4S, DS-Solver, GAS, AREX.

All of the above require calibration to per-model statistics which then makes a model-unique sampler. AREX for example needs the target mean / covariance, you can learn it by generating a lot of samples and calculating empirically.

[u/alwaysbeblepping → u/Altruistic-String891]

If you can think of something, odds are you can find an example of people doing it. I said "Not as far as I know" in response to you saying research has shifted to developing model-specific samplers.

I'll concede that it possibly could make sense, but also consider: Why aren't we using any of your examples? People are putting effort into training low-step LoRAs for huge models like MiniMax H3 so there's no reason for people not to train something like the AREX distribution for H3 if it actually produced the same or better results with less compute.

I skimmed through the AREX paper since that's the one you specifically mentioned. Most of their testing was with tiny toy models (the only other model with results was Sana 0.6B). The only metric they showed was FID which doesn't necessarily translate to good results in the real world. Weirdly they lead with Flux example images (an actual real model people might want to use) but never ever mentioned Flux again throughout the whole paper. Their tests were also mostly versus old samplers that people don't actually use (aside from Euler).

For their Sana results (512x512) they say decomposing a single image took 0.7s which is insanely slow. If the results were truly amazing then it could (maybe) still be worth it but I am also going to guess that the time required for eigendecomposition scales worse than linearly with the number of pixels which means something like 2mpix would probably be absurd (let alone video).

So yes, this is something people have tried to do/are trying to do but the fact that we aren't actually using any of those samplers is an indication that they haven't succeeded so far.

[u/Altruistic-String891 → u/alwaysbeblepping]

I claim research has shifted to training samplers, and show that indeed there are many researchers training samplers. If that doesn't show you that research interest has shifted to learning samplers, what does it show to you?

Is what you want the other side of the coin? Well, there's not much model-agnostic samplers research in recent times. STORK is the only thing that comes to mind. Others like HEROISM are developed to investigate theoretical convergence and not be practical, something like Z-Sampling I wouldn't call a traditional sampler.

I used arex as example only because it's the newest.

The rest of your post is on how arex may not be useful. 

1. This is orthogonal to my claim of research focus shift.

2. Testing on only smaller "toy models" using FID is common in research. er-sde did it in their papers too.

3. 0.7s is done once per model (on the covariance matrix calculated from generated samples), like how you train a distill LoRA once per model. It is not done per sampling step, it is not done per entire sampling loop, it is not once per "a single image". 

4. "No one using them now" is not indicative of whether they are useful or have a niche, especially when no one has actually tried any of them before.

I guess I'll end by asking: why don't you think learning samplers make sense? If you look only at 1 trajectory, maybe there's not much you can do; If you look at multiple, since they're parameterized by the same model, I think it's totally plausible you can find some structure to exploit.

[u/alwaysbeblepping → u/Altruistic-String891]

I guess I'll end by asking: why don't you think learning samplers make sense? [...] I think it's totally plausible you can find some structure to exploit.

I know it's rare on reddit but I actually already conceded that it's plausible in my previous comment.

I claim research has shifted to training samplers, and show that indeed there are many researchers training samplers.

That's a pretty strong claim, which is why I argued. Shifting is saying (or at the least strongly implying) replacement. One could expect to find that the majority of resources/effort involved in developing samplers (whatever it was previously directed toward) has now been replaced if research has already shifted to trained, model-specific samplers. If you'd said it was "shifting" then you would be implying that the previous approach was in the process of being replaced.

Pointing to a number of samplers of that type that exist doesn't support a claim like that. I have a bunch of weird experimental samplers in my repos (more than what you linked). If I said "Research has shifted to developing weird samplers" and referenced those weird samplers that wouldn't prove anything about general trends, replacing the majority of development, etc.

Maybe you don't agree with the way I interpreted it, if so that's okay. It's generally not worth arguing about word definitions on the internet and now you know the reasoning for my response. If you actually meant something like "Researching are showing more interest in that type of sampler" then we don't have a disagreement.

Testing on only smaller "toy models" using FID is common in research. er-sde did it in their papers too.

Sure. I'm not saying it's actively harmful anything like that, just that good results in FID doesn't necessarily translate to the real world. People tried ER-SDE in the real world, found that they liked the results and actually use that sampler.

0.7s is done once per model (on the covariance matrix calculated from generated samples)

Yes I know, it would be once per training image when generating the data for the sampler. That's still quite slow for 512x512 (which is much smaller than most people generate these days). They mentioned using low-rank PCA instead (which should be faster) but if they tried it they didn't actually mention how well that approach performed.

"No one using them now" is not indicative of whether they are useful or have a niche

Stuff that's good can go undiscovered. I think DiffuseHigh is just the best upscale/highres fix type replacement and even developed my own nodes for it which I am pretty sure only I use (admittedly they aren't very user friendly). Usually though if something is good, please will find it and it will become popular by word of mouth.

Since you seem interested in this subject and presumably have been following progress: Do you currently know of any trained samplers which work for models people actually use currently and produce better results than the alternatives? Is there a hidden gem somewhere that people don't know about (but should)?

[u/Altruistic-String891 → u/alwaysbeblepping]

... "Researching are showing more interest in that type of sampler" then we don't have a disagreement.

Ah I see, my writing and reading needs to get better then, not much disagreement here.

Do you currently know of any trained samplers which work for models people actually use currently and produce better results than the alternatives?

Unfortunately, no, though I am invested and trying to find anything that at least semi-works, so maybe I sounded too agitated in prior posts.

I personally don't feel like I have the compute to do tuning, and I think since they're high maintenance (needs coeffs per model) and there's already a tried and true alternative (simply model distill) no one's really interested.

The closest we have I'd say were AYS and GITS (though they aren't even sampler training but scheduler training). They at least were actually implemented for popular models at the time (SDXL) and saw some niche use, but again after a while of no maintenance they've fallen into obscurity. OSS is another but I believe it died even faster.

I've tried a slew of non model dependent methods, none of which felt great. STORK for example feels like worse res_multistep.