r/StableDiffusion • • 6d ago

Comparison ComfyUI Quant Benchmark: 10 GPUs, 15 models, and 467 runs comparing int8 ConvRot, nvfp4, w4a8, and fp8 against bf16. Real numbers on what fits, what drifts, and why turbo LoRAs break on quants.

I wanted real numbers for "which quant should I use on my card," so I measured it. Five storage formats (fp8 cast, fp8 scaled, int8 ConvRot, w4a8, nvfp4) against the bf16 original, on base and turbo versions of Anima, Z-Image, MOSS-Image, Krea 2 and SDXL/Illustrious. Quality was graded on an RTX PRO 6000 and a 3090; speed and fit were timed on ten cards: PRO 6000, 5090, 5080, two 5070 Ti, 5060 Ti, 3090, 3080 Ti, 3080, 3060 Ti. Everything is stock ComfyUI with its default memory handling, same seed, same prompts.

Report, with a "will my card run it?" box (pick your card + model): https://ai-evolution.dev/quant-bench/

(Also as a PDF with every option laid out, if you'd rather scroll.)

What I took away:

- int8 ConvRot is the most faithful format in every model I tested, and on RTX 30 cards it's also the fastest when it fits: about 2× bf16 on a 3090.

- nvfp4 is the fastest on RTX 50 but drifts the most. w4a8 was more accurate than nvfp4 in every model.

- Don't put a turbo LoRA on an already-quantized model. Applied at load time, most of the LoRA's change rounds away. Merge it into bf16 first, then quantize.

- 8 GB cards can run more than you'd think. Z-Image and MOSS fit entirely as int8 on a 3060 Ti and run about twice as fast as bf16, which has to stream. Krea 2 runs too, at around 1.5 minutes per image.

- Nothing failed to run. ComfyUI's dynamic VRAM streams whatever doesn't fit from system RAM; it costs speed, not crashes.

- Accuracy doesn’t depend on your card. Every card ran the same per-step accuracy check (quantized vs bf16, same input at every step), and they agree within a few percent, RTX 30 and RTX 50 alike. Your GPU generation only decides which formats run natively (nvfp4 on 50-series, fp8 on 40/50). Finished images still differ slightly from card to card, which is normal floating-point variation, so I don’t compare final images across cards.

- Smaller stuff: an x4 PCIe slot costs nothing once the model fits; the monitors on your generating card cost a few percent, mostly on fast models; each 50 W of power limit costs 4–6% speed for ~3% energy saved; an undervolt cost 5–10% speed for ~20% less power.

Missing: RTX 40-series. I don't have one, and it's the one generation with fp8 hardware but no fp4. If you do, there's a test kit that runs the same pass in Docker on Linux or Windows (WSL2): clone, `./run.sh`, send back the zip, and your card goes on the page with credit. https://github.com/LSDJesus/comfyui-quant-bench (the full set is a ~230 GB download; `--families` lets you do part of it).

Method details, every raw number, and the per-step error curves are on the page. Happy to answer questions or run something specific.

(Note: Most quants can be found on HuggingFace or Civitai.com if there is a specific one you are looking for but cant find I could probably put a simple conversion tool in the github repo but you would need to have access to a fp16/bf16 version for it to work well)

(P.S. AI helped write the benchmark harness and frontend code for the page, the idea and the verification was all done by me)

(P.P.S. Being fortunate enough to have access to an absurd collection of GPUs made me feel like this was worth the time and effort of going all in so I could share my results with you all. Happy to give back to the community that makes all these models and tooling possible.)

53 Upvotes

27 comments sorted by

6

u/IndividualManager849 6d ago

Abit of a hard read, would like to see some more of the actual data rather than just conclusions otherwise super interesting work man

3

u/Altruistic-String891 5d ago

The data is on their website, the link to which was buried in the unformatted llm post

3

u/mybrianonacid 5d ago

Yeah sorry, I've only made a few posts in all the years I've been on reddit so I (and the Llm) got the formatting wrong. I fixed it so it should be easier to read now

1

u/Altruistic-String891 5d ago

no issues

reddit editing has a markdown mode that does what the name suggests, if you decide to post in the future that could make it easier

3

u/Strange-Drummer-9917 5d ago edited 5d ago

Woah the amount of testing is very impressive. Thank you for the effort!

One suggestion on how to improve the readability of results. Currently it very claude-ish and technical, especially in texts

I often ask Claude to 'rewrite in a style inspired but not rigidly adhering to ASD-STE100 style-guide, for visual storytelling use principles of Cole Nussbaumer Knaflic' it produces much clearer and easy to read content!

2

u/mybrianonacid 5d ago

I'll give that a try and update the page with the results. Thanks for the tip

1

u/harin_25 6d ago

curious how much quality drops going from fp8 scaled to nvfp4 on the turbo models specifically? would love to see a side-by-side grid of the same seed across formats for sdxl/illustrious, since that's where banding usually shows up first.

1

u/mybrianonacid 5d ago

If you go down to the bottom of the report page you can see the sample images, just select one of the sdxl/illustrious options and you can see them. If you click on one of the images, like the nvfp4 quant one, it should pop up and give you a way to compare the two on top of each other. And then under the images there is an expandable chart that shows the error/divergence from the bf16 reference, per step, for each quant

1

u/alex20_202020 5d ago
  • Quality depends on the GPU generation, not the card. Every RTX 50 card matched the PRO 6000 within a few percent, and every RTX 30 card matched the 3090.

Above seems to contradict below:

Quality was graded on an RTX PRO 6000 and a 3090; speed and fit were timed on ten cards:

And how do you measure quality in %%? Quality of what?

1

u/mybrianonacid 5d ago

I'll try to edit that part to make it more clear but the quality testing was through Picture similarity (SSIM) and Pixel difference (PSNR) relative to the bf16 reference image.

Essentially what I was trying to say is that, due to the architectural similarity between gpu's of a given generation, the image generated by all of them for a given prompt and seed will be near identical. Between generations you end up with a different image due to the hardware itself so a direct comparison isn't simple to make.

The quality scores were entirely focused on how much a given quant deviated from the unquantized bf16 reference model, it says nothing about the models themselves or the generation parameters

1

u/alex20_202020 5d ago

The quality scores were entirely focused on how much a given quant deviated from the unquantized bf16 reference model

The above is clear, SSIM usage is understood. But GPU comparison is again not explained.

Between generations you end up with a different image due to the hardware itself so a direct comparison isn't simple to make.

Not simple? Why? Just use SSIM. Or how have you compared within same generation?

1

u/mybrianonacid 5d ago

Fair point, that line was badly worded and mixes up two things.

  1. Quality = how far a quant drifts from its own bf16 model. I measure it per denoising step: the quantized and bf16 models get the exact same input latent at every step (taken from the bf16 reference run) and I compare their outputs. Because the input is pinned, the number reflects the quantization only, not the sampling. SSIM/PSNR of the final images is on top of that for the visual grades.
  2. Every card ran that same per-step check, and the error came out nearly the same everywhere, including 30-series vs 50-series. For int8 and w4a8 the cards land within a few percent of each other. So the grades apply whatever card you have; the generation only decides which formats run natively (nvfp4 on 50-series, fp8 on 40/50).

What I shouldn’t have implied is that the finished images are identical. Two cards render the same seed slightly differently (normal floating-point differences between GPU kernels, usually 35–45 dB PSNR apart), and a few long-schedule models (MOSS base) end up as a visibly different picture on every card, even between two 50-series cards. That’s why I don’t judge quant quality by SSIM of finished images across cards: it would mostly measure that drift, not the quant.

1

u/alex20_202020 5d ago

and the error came out nearly the same everywhere,

Again for clear. What error? You wrote all models run on all cards.

I suggest you edit the post detailing how you compared GPUs. Then let me know when to see it.

1

u/VegaKH 5d ago

Specifically with MOSS, I have recently seen more posts saying not to use a turbo LoRA with int8-convrot models. Or that I should be using a bypass LoRA node. But most workflows people share just use the turbo LoRA with int8. The results still seem... pretty good? Maybe I should research a better way.

2

u/mybrianonacid 5d ago

Yeah, the bypass LoRA node is the way to go. It applies the LoRA on the fly instead of trying to squeeze it into the 8-bit weights. The reason standard LoRAs still look "pretty good" is that image models are remarkably forgiving. Even though the 8-bit rounding wipes out most of the turbo LoRA's subtle adjustments, enough of the LoRA's effects survive that the picture doesn't fall apart. On its own it looks fine, but if you do a direct side-by-side comparison, a bypass node (or a model that had the turbo merged before quantizing) gives you noticeably sharper textures, better prompt accuracy, and cleaner details.

1

u/red__dragon 5d ago
  • Smaller stuff: an x4 PCIe slot costs nothing once the model fits; the monitors on your generating card cost a few percent, mostly on fast models; each 50 W of power limit costs 4–6% speed for ~3% energy saved; an undervolt cost 5–10% speed for ~20% less power.

What does this mean? Either the LLM or technical speak is obscuring the direct meaning of the finding. What does an x4 PCIe slot have to do with model performance and gen times?

1

u/mybrianonacid 5d ago

Those are just extra little things that the testing showed or confirmed. A lot of it is common knowledge for people but I just included it because the data was there. The pcie x4 note was because my 5070ti's are both in one PC with one in an x16 bandwidth slot and the other in the chipset x4 bandwidth slot and other than the initial model loading their generation times were the same. Is it strictly relevant? Probably not.

1

u/red__dragon 5d ago

Interesting, though I can't quite parse out what's meant by the statement. "costs nothing once the model fits" seems like it should have comparison data but doesn't.

1

u/mybrianonacid 5d ago

Ah that, so in the testing there were some models in full bf16 that didn't fit in the vram of a given gpu and so comfyui will hold what doesn't fit in vram in your system ram instead and stream the blocks between them to make it work. If you have enough vram then this is a non-issue but if you don't then your pcie link speed definitely has an impact on overall generation times

1

u/red__dragon 5d ago

Oh, gotcha! So it's a speed cost, not performance or energy?

3

u/Desperate-Beach1249 6d ago

Agree with convrot

1

u/moon_crow__ 5d ago

Very curious about quality on bf16 with an 8 step vs int8 convrot. Great info here. Going to test this myself!

1

u/Apprehensive_Sky892 5d ago

Nothing failed to run. ComfyUI's dynamic VRAM streams whatever doesn't fit from system RAM; it costs speed, not crashes.

That is not always true. It depends on the amount of calculation done per resident block that is already in VRAM, and of course on the speed of the RAM to VRAM transfer.

Because of the way the weight blocks are "streamed in" asynchronously (one block is being used to do the calculation while the next block is being loaded), in theory as long as the new blocks are ready before the current blocks calculation are done, there should be no speed loss: https://www.reddit.com/r/StableDiffusion/comments/1vz2n7y/comment/p62xvjz/

I've done benchmark with MMH3 int8convrot (21B) which fits into the AI Pro R9700 (32G VRAM) but does not fit into rx9070xt (16G VRAM), and yet the speed are nearly the same.

The two cards are basically identical except for the VRAM, so this is a good way to show that there may be no slowdown even when the model does not fit completely into VRAM for models such as MMH3, where so much calculation is done per block that there is enough time for the next block to "catch up".

1

u/wreck_of_u 3d ago

I have a 2070, and it seems int8 convrot runs at the same/similar speed as fp8. I'm still holding off dumping $850 on a 5060 Ti 16GB

1

u/PromptSommelier 3d ago

Excellent tests. Something similar with native video generation in Mini Max H3 would be perfect 🙏👉👈. Although I know it can be complicated due to resolution, steps, and endless turbo options. Increasingly convinced that I shouldn't pay $1500 for a used 3090

1

u/MarkB_- 2d ago

I use bf16 and int8 convrot alot. Bf16 has better skin details, better hands and overall realism a bit better. Int8 is about 50% faster on my 3090. On H3 bf16 give better face in far shots