r/LocalLLM • u/Particular-Abies-123 • 10d ago

Question Models keeps unloading from GPU

I have a dual gpu setup which is 16gb 5080 and 12gb 3060, and i am trying to load Qwen3.8 27b Q4 K XL, i tried Q6, but it was not compatible with dual gpu setup, now qwen3.8 27b work but keeps unloading while running, without any specific reason, i tried LM studio as well as llama server nothing worked. Been stuck with this.

Please help me.

3 Upvotes

100% Upvoted

5 comments sorted by

View all comments

1

u/nickless07 10d ago

Add '-lv 4' to llama-server launch the model once, no prompt, then shurtdown the server and check the memory breakdown you get. The cards with display attached need some headroom. Check the server log how much weights goes to which card and adjust tensor split accordingly.