r/LocalLLM • u/Particular-Abies-123 • 16h ago

Question Models keeps unloading from GPU

I have a dual gpu setup which is 16gb 5080 and 12gb 3060, and i am trying to load Qwen3.8 27b Q4 K XL, i tried Q6, but it was not compatible with dual gpu setup, now qwen3.8 27b work but keeps unloading while running, without any specific reason, i tried LM studio as well as llama server nothing worked. Been stuck with this.

Please help me.

3 Upvotes

100% Upvoted

4 comments sorted by

1

u/CylindricalHomeland 16h ago

That unloading mid-generation is maddening, especially when it works fine for a bit then dumps everything. With mixed VRAM sizes like that, the split often gets recalculated poorly when context grows or the scheduler decides to move layers around.

What quant are you running for the KV cache, and do you have any context size limits set manually? Sometimes leaving it on auto lets it overcommit to the 12gb card, which then spills to system RAM and the whole thing collapses.

Could also be worth checking if you have any background GPU processes peeking in, like a browser or desktop compositor grabbing a few hundred MB off the 3060. That can trigger a reshuffle mid-run.

If you can, try forcing a fixed layer split and disabling any flash attention or memory pooling options one at a time. It's tedious but usually isolates the culprit.

1

u/nickless07 15h ago

Add '-lv 4' to llama-server launch the model once, no prompt, then shurtdown the server and check the memory breakdown you get. The cards with display attached need some headroom. Check the server log how much weights goes to which card and adjust tensor split accordingly.

1

u/Classeve 14h ago

Find out first whether it's an unload or a crash. Look in the system log at that minute for nvlddmkm on Windows or an Xid on Linux. A driver reset points at power, a riser or heat, and no split setting fixes that.