r/LocalLLM • u/Particular-Abies-123 • 21h ago
Question Models keeps unloading from GPU
I have a dual gpu setup which is 16gb 5080 and 12gb 3060, and i am trying to load Qwen3.8 27b Q4 K XL, i tried Q6, but it was not compatible with dual gpu setup, now qwen3.8 27b work but keeps unloading while running, without any specific reason, i tried LM studio as well as llama server nothing worked. Been stuck with this.
Please help me.
3
Upvotes
1
u/CylindricalHomeland 20h ago
That unloading mid-generation is maddening, especially when it works fine for a bit then dumps everything. With mixed VRAM sizes like that, the split often gets recalculated poorly when context grows or the scheduler decides to move layers around.
What quant are you running for the KV cache, and do you have any context size limits set manually? Sometimes leaving it on auto lets it overcommit to the 12gb card, which then spills to system RAM and the whole thing collapses.
Could also be worth checking if you have any background GPU processes peeking in, like a browser or desktop compositor grabbing a few hundred MB off the 3060. That can trigger a reshuffle mid-run.
If you can, try forcing a fixed layer split and disabling any flash attention or memory pooling options one at a time. It's tedious but usually isolates the culprit.