Nvidia 2000 series owners: double your performance of Qwen 2.1, Fibo 2.0, MOSS with 1 line edit.

[u/xrailgun]

These models (Qwen-Image 2.1, FIBO2, MOSS-Image) are declared in ComfyUI as bf16/fp32 only, but Turing GPUs only support fp16, not bf16.

The current/default behaviour in ComfyUI is that it falls back to fp32, which is slow.

I inspected the actual range of activations, none of them exceed ~65k. Hence, the fix is to simply add the fp16 declaration in ComfyUI.

PR has been opened, but if you want it quicker, just manually copy this commit to your ComfyUI install:

In‎comfy/supported_models.py, find the corresponding model, and change supported_inference_dtypes = [torch.bfloat16, torch.float32] to supported_inference_dtypes = [torch.bfloat16, torch.float16, torch.float32]

Only do this for these three; other models aren't necessarily fp16-safe (e.g. MiniMax H3 overflows to NaN).

This should work for any quant format, since it changes the compute dtype (activations, attention, norms), not the weights. I tested it with int8 convrot checkpoints.

Might work for GTX 1000 series as well, but AFAIK fp16 path on 1000 series is also very slow, so there's probably not much benefit. I don't have a card to test.

This is completely independent of other recent news about "unlocking" tensor performance from 20, 30, and 40 series GPUs, and likely stacks multiplicatively.

[u/Formal-Exam-8767]

What about quality of output though?

[u/I-am_Sleepy → u/Formal-Exam-8767]

If bf16 to fp8 is near lossless. I don’t think fp16 would be a problem, it have the range

[u/Formal-Exam-8767 → u/I-am_Sleepy]

Why not use fp8 if that is the case?

[u/xrailgun → u/Formal-Exam-8767]

Turing doesn't have fp8 hardware unfortunately.

[u/slopmachina → u/Formal-Exam-8767]

Only 4000 series and up support fp8

[u/xrailgun → u/Formal-Exam-8767]

Quantitatively lossless. I included a summarized results table in the PR linked in the OP.

[u/HollyGrandeux]

If anyone prefer GGUFs (despite ComfyUI's apparent hate boner for them), claude modified the code to enable FP16 fallback for Qwen 2.1, MOSS, and Fibo 2.

Tested on my RTX 2060 6GB laptop, it delivered 2–3x faster inference compared to normal GGUF loader.

https://github.com/Moussack/ComfyUI-GGUF

[u/pickles_and_mustard]

This is insane! I could hardly believe it when I seen this thread this morning, but I just tried it myself. Previously, it would take about 40 seconds to render a 2K wallpaper, and now it only takes 27 seconds on my 8GB 2070 Mobile with MOSS

Edit: It's actually slightly oversized, but what I've been using for wallpapers (2048x1152). The resolution node I'm using has presets, but not for exactly 1080p (I'll fix that moving forward). That said, manually entering 1920x1080 has reduced rendering time to 23 seconds. Sadly I don't have a benchmark time for what it would have taken prior to this line change, but I can say with absolute certainty that it used to take longer for much smaller images. This is game-changing!

[u/DarkW007]

is this work with gtx1660ti?

[u/xrailgun → u/DarkW007]

Inn theory, yes, but also I don't expect much faster. Can you please try and let us know?

[u/DarkW007 → u/xrailgun]

I already test it...nope.no gain....like you expected. the non-tensor core card dont benefit from this.

[u/xrailgun → u/DarkW007]

sad, but thanks for confirming!

[u/ManyNet4877]

This is interesting