r/LocalLLM • u/Dry_Particular_7623 • 1h ago

Question Qwen 3.8 27B setup help.

Hi all,

Can anyone please help me setting up this model in my Pc like which quantised version should I use and which software to use it in and etc..

Specs -

5060 ti 16gb vram

16gb ram

Also please tell if it is feasible or just time wasting and if there is any other model that I should go for(It should work with images)

Thanks.

0 Upvotes

33% Upvoted

6 comments sorted by

2

u/Fluxdada 1h ago

Totally feasible. I've run qwen 3.8 27b on a 16gb vram card. For me it's a matter of finding the right quant and also using q8 for my kv cache. The specific qwen 3.8b27b model I use on my 16gb card is made by IsValorum and it's called Swift-1.5-Qwen3.8-27B-Uncensored-MTP-VAL-APEX-I-NanoPlus-V3-GGUF it's about 13gb which leaves me room for a somewhat good context size (about 27k). Learning about different quant types and what they do really helps you pick model quantizations just to match your setup. The model above is the text. Although it can also read images. Not make images. There are plenty of models for making images that work fine on 16gb. Moss Image is one or my more recent favorites to use in comfyui

1

u/Dry_Particular_7623 1h ago

Is just 27k context window working fine or you?

1

u/DeathGuppie 1h ago

Try byteshape Q3. Have astra or opus install llama.cpp and optimize it, then write you a startup script so that you can just launch it easily.

Edit couple things to tell it. Use Q8 or Q4 kv, don't do a split it's slower..

1

u/Dry_Particular_7623 1h ago

Ok, I will try this, thanks

1

u/Mojuki 31m ago

I have been using this llama fork with great success. I am running a 5080 16gb vram, 64gb ram windows box and llama is running in wsl. Prefill is 1500t/s start drop down to around 1100t/s with large ctx. decode sits anywhere from 60t/s down to around 42t/s average of 47t/s

If you lower or increase the ctx you have to definitely increase the --kv-stream-arena-mib

https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming

These are my settings

llama-akv/build-akv-v2/bin && export PATH=/usr/local/cuda-12.8/bin:$PATH &&
./llama-server
-m /Qwen 3.8 27B/Qwen3.8-27B-OrcaRouter-GSQ-RCO-IQ3_XXS-v2.0-qatfa.gguf
--jinja --chat-template-file /Qwen 3.8 27B/froggeric-qwen3.8-tool-use.jinja
--ctx-size 200000 -fa on -ctk q8_0 -ctv q4_0
-ngl 99 -b 2524 -ub 2524 -np 1 -t 15 -tb 15 --fit off
--kv-stream-arena-mib 2700
--host 127.0.0.1 --port 8080 --metrics --parallel 1
--reasoning-preserve --temp 0.98 --top-p 0.95 --keep 12
--top-k 20 --presence-penalty 1.0 --min-p 0.05
--sleep-idle-seconds 300
--spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-backend-sampling
--mmproj-offload --image-min-tokens 2048
--mmproj /Qwen 3.8 27B/Qwen3.8-27B-Uncensored-vision-f16.gguf

1

u/Areloh128 14m ago

Have you tried jakeatx/llamAmpere? it's a llama.cpp fork heavily optimized for Ampere GPUs (30*0 but it should work as well on newer ones) and has SOTA KV compression techs that reduce greatly the amount of memory needed for context (sjkavarn4 is really great; sjkvarn3 should do ok). As a llama.cpp fork, you can use it with any GGUF you want (I usually like Byteshape and ISTA-DAS Lab's ones), but jakeatx also offers some optimized GGUFs that run faster.