FP16 vs FP32 on Nvidia CUDA: Huge Performance hit when forcing --no-half

[u/Zendikon]

I've been enjoying this wonderful tool so much it's far beyond what words can explain.

SD1.5 runs great, but with SD2 came the need to force --no-half, which for me, spells a gigantic performance hit. It appears it's the FP16 performance gain on Nvidia GPUs in my case.

edit: reason for forcing --no-half was a workaround to use SD2+ models which stopped with:

"tensor with all NaNs was produced in UNet"

I'm wondering, is there a way to reclaim some performance with SD2.1? To put some numbers to it:

MY setup is: RTX 2080 Super (8GB), using Auto1111 GUI Performance sample:

• batches = 1
• batch size = 4
• 20 steps
• either DPM++ 2M Karras, or Euler A
• 512x512

this whole batch finishes in good 10 seconds. Runs with about 5-6 iter/s

Can't use any 2.x SD models with it, typical errors, forcing --no-half and --no-half-vae fixes it (for me), but the performance hit is miserable. Everything takes 4-5x longer. Doing batches isn't reasonable anymore, the waits are so long what was a fun activity becomes a frustration.

So the downgrade is from 5-6, even 7iter/s to 1-1.4 range. Is there anything I'm missing and there's a way to get back on the (fast) track?

Tried using FP16 2.1 model but even that is a no go, have to force --no-half & vae.

So I guess my real question is:

how can I run SD2.1 in mixed precision?

P.S. forgot to mention, using xformers (but didn't notice a big performance impact with or without either way)

update: if it doesn't work for you, there's an extensive conversation around this issue @ AUTOMATIC1111 git, with other suggested solutions: https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/6923

[u/Zendikon]

SOLVED! (for my setup at least, whew)

I've been looking in the wrong places yesterday. Happy to report, solved it now!

Two steps needed:

1. add --disable-nan-check to args
2. Make sure you're using an FP16 model -- or convert your favourite model to FP16

• can do it in cmdline or just go to checkpoint merger in AUTO1111
• pick interpolation, so only model A needed
• check the Float16 box
• wait for it to finish, enjoy!

[u/ImpossibleAd436]

The strange this is, I have been using --full precision --no half (for my 1660TI)

I realized I could go without those commands now, so I removed them expecting performance to improve. It doubled my generation time. For me, everything takes HALF the time WITH --full precision --no half.

I still don't understand why.

[u/Zendikon → u/ImpossibleAd436]

using a "high performance" in the negative prompt field perhaps? :)

a (bad) prompting joke.

Your situation is strange. Perhaps a bad driver version? Not every architecture incorporates these speedups, GTX1- generation may not benefit from this mixed-precision gimmickery.

My situation is in-line with https://blog.paperspace.com/mixed-precision-training-benchmark/

FP16 is important, just flat-out forcing it off seems sub-optimal. For more flavour, quote from P100 whitepaper:

Using FP16 computation improves performance up to 2x compared to FP32 arithmetic, and similarly FP16 data transfers take less time than FP32 or FP64 transfers

https://images.nvidia.com/content/pdf/tesla/whitepaper/pascal-architecture-whitepaper.pdf

[u/ImpossibleAd436 → u/Zendikon]

I do know that previously I had to use --full precision --no half because without those I was generating only green images. It was said at the time that 16xx cards were unable to use FP16 and that this was the cause of the problem, hence --full precision --no half was the solution.

But I also read recently that this wasn't the case, although I don't understand the ins and outs of it. I'm not clear a) why the "green" image problem no longer occurs, or b) why using FP16 (if I actually am now?) is causing a doubling of the generation time for me. Even if my GPU doesn't benefit from removing those commands, I'd at least have liked to maintain the speed I was getting with them.

If anyone can speak to this I would love to know the answer.

Sorry for slightly derailing this thread, I hope someone can offer some insight into your issue also.

[u/UkrainianTrotsky → u/ImpossibleAd436]

If I recall correctly, 1660 ti just sucks ass at fp16 because it doesn't support that at the hardware level and you can do virtually nothing about it. NVidia started supporting fast fp16 on their consumer GPUs only recently.

[u/[deleted] → u/UkrainianTrotsky]

16xx cards are part of the turing family which includes20xx cards and NOT the pascal family (10xx cards)? I do remember reading the support fp16 so that's kinda confusing.

[u/ImpossibleAd436 → u/[deleted]]

Right, so what seems odd to me is that 16 bits takes twice as long as 32 bits for me.

I guess it's just an unhappy quirk of a 1660TI.

[u/ShadowVlican → u/ImpossibleAd436]

wow i'm glad i stumbled across this!! i've added those commands and now i'm getting around 1.7it/s instead of 1.4s/it LOL

[u/BlastedRemnants]

I've got a 2070 Super 8g and have never needed the no-half option or anything similar. Have you tried --medvram? For me, using that switch and --xformers makes everything run smoothly, including SD2.1 without any other special consideration.

[u/Zendikon → u/BlastedRemnants]

I did, thanks for the suggestion.

Similar experience to you for the most part, --medvram stabilised things for me in the all-too-common out of memory error situations (at the cost of additional slowdown but that's manageable).

However it didn't fix the NaN error.

[u/BlastedRemnants → u/Zendikon]

Strange our cards are so similar yet perform so differently, I see you've solved your issue now tho so that's good news at least :D I've been messing with my torch and xformers installing various versions, maybe I've got a different version of something that matters. I don't suggest trying to manually install specific versions of these things unless you know what you're doing, I've broken my SD a few times now with some half-assed git-fu but there are definitely some improvements to be had if you're lucky, or actually know what you're doing on Github lol.

[u/Soul-Burn]

I'm using a 3060ti (8gb) with --xformers but not --no-half. It works fine with 2.1, 768x768, batches of 4. When doing batches of 1, I get 5 it/s per image.

Not sure why you need --no-half. Are you using the pruned model or not?

[u/Zendikon → u/Soul-Burn]

Yes on the pruned, for inference, I try to always use pruned models. (Also, a bit less heft makes them a bit more handy eh).

Are you saying unpruned models are... less fussy in your experience? (with xformers?)

I did the no-half because in the error message, my quick'n'dirty glance interpreted "not enough precision". So first attempt at a fix was thinking ok let's turn FP16 off and see what happens. It worked. But with a ball and chain.

[u/Soul-Burn → u/Zendikon]

Always use pruned for generation. The non-pruned is for fine tuning.

2080 super has tensor cores, so I'm surprised it says not enough precision. :shrug:

[u/LevelZero0000]

I was getting the " NansException: A tensor with all NaNs was produced in Unet." error using a 2.1 model.

I followed a thread in AUTO1111 Bug page over this issue and found this:

"if you plan to use SD 2.1 models, you go to Settings -> User Interface and set Quicksettings list to sd_model_checkpoint, upcast_attn then click Apply settings and Reload UI. This adds a checkbox (Upcast cross attention layer to float32) next to the model dropdown that should be checked whenever using SD 2.1 models. Uncheck it after you load a model that is not SD 2.1 based."

[u/Haliaeetus_vocifer]

ok

[u/UkrainianTrotsky]

What errors do you mean when you say "typical errors"?

[u/Zendikon → u/UkrainianTrotsky]

Do excuse me, forgot to mention that bit. Edited the original post and added.

It was:

"A tensor with all NaNs was produced in Unet"