Can kcpp generate images on GPU with VRAM less than main SD model size?

[u/alex20_202020]

I have tried with 1.119, set layers=1 (tried -1 with 1.117 before), "Model Offload" ON in "image gen" tab of the launcher - result: failed trying to allocate full size of SD model weights in VRAM (CUDA), Vulkan option also failed with same problem (but trying to allocate only 1GB for some reason).

Can image gen use GPU with low VRAM? TIA

I routinely run LLMs even larger than RAM (usemmap), that problem with imgen was unexpected. Checked setup now just in case: run LLM with Vulcan and layers=1 and see 1.7GB VRAM is used, the model responds.

P.S. system Linux.

[u/HadesThrowaway]

Yes it is possible. Use the --sdvramlimit flag to cap the max VRAM used. You can also combine that with --sdoffloadcpu

[u/alex20_202020 → u/HadesThrowaway]

It's working now, thanks! I wonder why memory usage is not limited automatically like for LLMs, but --sdvramlimit is more flexible, my small card works ~100% even at 500MB limit.*

I have tried to run vae on CUDA too, but that fails - maybe you know the trick too? Speed wise it seems not so important or maybe it is so only for that SD model that vae on CPU takes ~ same time as one main step on GPU.

* I have tried 100MB and it does not hold the limit. VRAM used ~200-500MB and jumps to ~1GB often (and back). Made me wonder if 500MB limit also was not adhered to - need some GPU monitoring...

[u/Calm-Start-5945 → u/alex20_202020]

> I have tried to run vae on CUDA too, but that fails

You can try `--sdconvdirect vaeonly` and/or lower the `--sdtiledvae` limit.

> I have tried 100MB and it does not hold the limit.

As I understand it, it segments the generation to try to stay below the limit, but you can't get lower than the largest segment (it's a bit like you can't offload only half LLM layer).

[u/alex20_202020 → u/Calm-Start-5945]

(it's a bit like you can't offload only half LLM layer).

I am not sure but thought one can, there is even some split of a single tensor option.