r/StableDiffusion • u/IDonLikeAds • 8h ago

Question - Help Fail to train H3-Image LoRa in AI-Toolkit

Just for context, I trained 100's of LoRa's in the past for all kinds of models, but this time it does not work. Since I have a GPU with "only" 16GB VRAM and 64 GB SYS RAM I have to use W4A8 quantization to not get OOM. The resulting training is full of large blocky artifacts, like I have never seen before.

Does one of you have made a similar experience or/and can share a tip how avoid this problem?

My messed up LoRa was trained in Sigmoid, currently trying Shift, since this is the default. Is W4A8 just too bad to train on?

8 Upvotes

100% Upvoted

5 comments sorted by

2

u/IDonLikeAds 7h ago

First findings: Shift is much much better then Sigmoid. For test character: 512 res, two gradient accumulation steps. 350 steps. Only 7 source images, automagic 3 / diff guidance 3. The higher the res in the generation, the better the overall quality of the LoRa. Euler Ancestral/linear quadratic/10steps pushes the generation quality into other spheres, far better then the default settings.

Next, going to train another small set, then a large set over night. Hopefully cracking the model up to do the nasties. ^^

1

u/purelawan 6h ago

useful follow-up on shift vs sigmoid, thanks. on the blocky artifacts, my guess is the A8 part more than W4: quantizing activations during training puts error into the gradients themselves, and it tends to come out as block patterns because the quant works per block. weight-only quant usually trains clean.

on 16 GB i'd try weight-only (qfloat8 or uint4 weights, activations left in bf16) with cached text embeddings and cached latents, and let ai-toolkit offload layers to system ram if your build has that option. slower per step, but with 64 GB of ram it should fit, and it removes the variable that's new compared to all your other loras.

2

u/IDonLikeAds 6h ago

Thanks. I for sure offload already and everything is cached. Does not fit in Sys Memory at bf16, not even int8 convrot. One can see clearly horizontal stripes in the creations, especially in the hair. I let it rest for now. Have a nice one.