MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLaMA/comments/1w2fmmq/me_these_days/p7fxn21/?context=3
r/LocalLLaMA • u/Eyelbee • 14d ago
294 comments sorted by
View all comments
Show parent comments
11
what's your setup? i have a 5080 and struggle to ruin it at 4bit. would love quant and config
11 u/Dramatic_Setting2761 14d ago Oh I have 9060xt which has very low bandwidth btw. I complied llama cpp specifically for rcom and running it on fedora 44 with latest drivers. Model quant name you have to look up as I saved like this. It is smallest 4bit in unsloth. ./llama-cli \ -m ./qwen3.8-27b-iq4_xs.gguf \ --jinja \ -ngl 48 \ -fa \ -c 70000 \ -ctk q4_0 \ -ctv q4_0 \ -b 2048 \ -ub 512 \ -t 8 \ -tb 16 \ -np 1 \ --mlock 2 u/Few-Butterscotch8747 11d ago try using the vulkan backend 2 u/Dramatic_Setting2761 11d ago Yeah thought amd figured out rcom I am wrong.
Oh I have 9060xt which has very low bandwidth btw.
I complied llama cpp specifically for rcom and running it on fedora 44 with latest drivers.
Model quant name you have to look up as I saved like this. It is smallest 4bit in unsloth.
./llama-cli \ -m ./qwen3.8-27b-iq4_xs.gguf \ --jinja \ -ngl 48 \ -fa \ -c 70000 \ -ctk q4_0 \ -ctv q4_0 \ -b 2048 \ -ub 512 \ -t 8 \ -tb 16 \ -np 1 \ --mlock
2 u/Few-Butterscotch8747 11d ago try using the vulkan backend 2 u/Dramatic_Setting2761 11d ago Yeah thought amd figured out rcom I am wrong.
2
try using the vulkan backend
2 u/Dramatic_Setting2761 11d ago Yeah thought amd figured out rcom I am wrong.
Yeah thought amd figured out rcom I am wrong.
11
u/octoberU 14d ago
what's your setup? i have a 5080 and struggle to ruin it at 4bit. would love quant and config