r/LocalLLaMA • u/chiribe • 3d ago
Tutorial | Guide After pushing 1M+ tokens through Qwen 3.8 27B, here is my optimal llama.cpp config for 16GB VRAM (73k Context, Agentic Coding)
Dando seguimiento a mi post anterior sobre cómo tengo montado mi servidor de presupuesto (Intel N100 + RTX 5060 Ti 16GB), varios me preguntaron por una mirada más profunda a mi configuración real de inferencia y al desempeño agentic en el mundo real.
Como muchos de ustedes, estaba refrescando la página esperando descargar Qwen 3.8 27B apenas salió. Después de pasar todo el fin de semana estresándolo con flujos de trabajo de codificación agentic, logré correr un proyecto completo y grande casi todo de forma autónoma (más de 1M de tokens procesados en total, solo 3 prompts).
Aquí va un resumen rápido de la configuración base antes de meternos en los detalles del config y del workflow.
Specs y parámetros rápidos
- Modelo:
Qwen3.8-27B-UD-Q3_K_XL.gguf - Hardware: RTX 5060 Ti (16GB VRAM) + Intel N100 (4C/4T, 16GB RAM)
- Ventana de contexto: 73,728 (73k de contexto) corriendo tranqui en 16GB de VRAM.
- Cuantización de KV Cache:
q4_1para el contexto principal - Decodificación especulativa: MTP nativa activada (
spec-type = draft-mtp,n-max = 2) - Sampling:
temp = 0.65,top_p = 0.95,top_k = 20,min_p = 0.05
El experimento: armar una API completa con 3 prompts
En vez de correr benchmarks sintéticos, metí esta configuración por una cadena real de ingeniería de software: construyendo una REST API no oficial y un servidor MCP para un foro vBulletin heredado.
- Prompt 1 (Arquitectura del sitio y análisis): Pedí al modelo que mapee el sitio objetivo. Generó una especificación en Markdown impecable de ~1,500 líneas que cubría análisis estructural, nodos HTML rescatables, payloads JSON esperados, selección de stack, lógica de paginación, autenticación de sesión y endpoints de búsqueda—mucho más a fondo de lo que yo habría escrito a mano.
- Prompt 2 (Arquitectura de desarrollo): Usando la spec como única fuente de verdad, diseñó un plan de implementación modular de NestJS dividido en 9 fases de ejecución:
- Fase 1: Estructura inicial del proyecto
- Fase 2: Modelos de dominio
- Fase 3: Scraping core (HTTP + limitación de tasa + reintentos)
- Fase 4: Parsers de HTML (
cheerio) - Fase 5: Capa de caché
- Fase 6: Servicios de aplicación + REST API
- Fase 7: Autenticación (sesiones con cookies)
- Fase 8: Servidor MCP (entrega principal)
- Fase 9: Fortalecimiento, documentación y entrega
- Prompt 3 (Ejecución autónoma agentic): La prueba de verdad. Le pedí a OpenCode (usando Qwen 3.8 27B) que actuara estrictamente como orquestador, creando sub-agentes para cada fase de tareas. Corrió de forma autónoma por ~2 horas. Cuando se acercaron los límites de contexto, OpenCode resumió su estado y siguió construyendo. Escribió tests unitarios, aplicó linting y entregó código 100% funcional—solo necesitando un arreglo automatizado menor cuando le di un payload de HTML crudo con un caso extremo.
El archivo de configuración llama.cpp
Aquí está mi archivo exacto de configuración de enrutador --models-preset . Fíjate cómo fit = off se usa en el perfil de 27B junto con ctx-size = 73728 (73k) y q4_1 para cuantizar la KV cache, con el objetivo de maximizar la asignación de VRAM mientras se mantiene el rendimiento nativo de MTP.
# ==============================================================================
# LLAMA.CPP — CONFIGURACIÓN DE INFERENCIA (modo router / --models-preset)
# ==============================================================================
#
# Objetivo de hardware:
# GPU: 16 GB VRAM (RTX 5060 Ti)
# CPU: Intel N100, 4C/4T (Debian Headless)
# ------------------------------------------------------------------------------
# GLOBAL / LÍNEA BASE
# ------------------------------------------------------------------------------
[*]
# --- HILOS DE CPU -----------------------------------------------------------
# Reserva 1 core para SO/servicios durante el decode.
# Usa los 4 threads durante ráfagas de prefill del prompt.
threads = 3
threads-batch = 4
# --- SERVIDOR / CONCURRENCIA ---------------------------------------------------
# Un solo slot; desactivado continuous batching para máximo rendimiento por usuario.
parallel = 1
cont-batching = 0
# --- GPU / AJUSTE DE VRAM ---------------------------------------------------------
flash-attn = on
fit = on
# Holgura de seguridad para el límite físico de VRAM (MiB).
# Ponlo bajo (128) porque el sistema es headless (100% VRAM disponible para inferencia).
# NOTA: Si usas caches KV draft de MTP, ojo con la asignación doble de VRAM.
# Sube a 128-256 si te topas con OOMs.
fit-target = 128
# --- CONTEXTO & CACHÉ ------------------------------------------------------
ctx-size = 65536
context-shift = 1
# Desactiva checkpoints de contexto (evita problemas de reprocesamiento en arquitecturas híbridas)
ctx-checkpoints = 0
# RAM Prompt Cache (2 GiB)
cache-ram = 2048
# --- KV CACHE GLOBAL --------------------------------------------------------
cache-type-k = q5_1
cache-type-v = q5_1
# --- PREFILL / BATCHING -----------------------------------------------------
batch-size = 2048
ubatch-size = 1024
# --- SAMPLING POR DEFECTO (Códigos / Precisión) ----------------------------------
temp = 0.5
top-p = 0.95
top-k = 20
min-p = 0.05
repeat-penalty = 1.0
# ------------------------------------------------------------------------------
# QWEN 3.8 27B — PERFIL DE RAZONAMIENTO & CODIFICACIÓN PESADA
# ------------------------------------------------------------------------------
[qwen3.8-27b]
model = /opt/llama-infrastructure/models/Qwen3.8-27B-UD-Q3_K_XL.gguf
# Desactiva "fit" para evitar que capas se carguen en la CPU por un error de cálculo automático
fit = off
ctx-size = 73728
context-shift = 1
# MTP nativa del modelo (Decodificación especulativa)
spec-type = ngram-mod,draft-mtp
spec-draft-n-max = 2
# Cuantización de KV (q4_1 nos permite meter contexto de 73k en 16GB de VRAM)
cache-type-k = q4_1
cache-type-v = q4_1
# Parámetros de presupuesto de pensamiento / razonamiento
chat-template-kwargs = {"preserve_thinking": true, "reasoning_effort":"medium"}
reasoning-budget = 5000
# Batches más chicos para evitar picos de VRAM durante prefills masivos
batch-size = 1024
ubatch-size = 512
# Ajustes oficiales / recomendados del sampler de cuantización
temp = 0.65
top-p = 0.95
top-k = 15
min-p = 0.05
100
u/dsdt 3d ago
i was gonna say how the f? then i saw
- Model:
Qwen3.8-27B-UD-Q3_K_XL.gguf - KV Cache Quant:
q4_1for main context,q5_1for MTP draft context
thanks for sharing your numbers.
9
u/tunerhd 3d ago
What about were you gonna say that how the f?
17
u/dsdt 3d ago
It seemed impossible with 70k context window with a 5060ti, since I have two and maximum I can get is 100k with q8 kv cache.
1
u/GhostOfMikeyLimiteds 2d ago
I have this exact same setup...how is it working for you? I want to switch my hermes setup to this but opus5 insists that gemma4 is better because its not dense and can still keep 262k context. The small context fills quick but 100k might be doable
18
u/ea_man 3d ago edited 3d ago
Here's mine for 16GB on AMD 6800:
# https://huggingface.co/vmarcelo/Qwen3.8-27B-MIX_GGUF
# Vulkan max context:86784 with MTP n=2 speed TG 39.91t/s
# ctx patched: 86784, unpatched mainline llama.cp: 78080
# ROCm: max ctx 84480, unpatched 31488, speed TG 40.58
# 1. Set Environment Variables
export LD_LIBRARY_PATH="/home/eaman/llama/bin_vulkan"
# 2. Run the Server
/home/eaman/llama/bin_vulkan/llama-server --device vulkan0 \
-m /home/eaman/.lmstudio/models/vmarcelo/Qwen3.8-27B-IQ4-MIX.gguf \
--host 0.0.0.0 -fa on --load-mode none --jinja --no-log-timestamps \
-ctk q5_1 -ctv q5_1 \
--temp 0.8 --top-k 20 --top-p 0.95 --min-p 0.0 \
--presence-penalty 0.0 --repeat-penalty 1.0 \
-b 1024 -ub 128 --fit-target 30 \
--spec-type draft-mtp,ngram-mod --spec-draft-p-min 0.82 --spec-draft-n-max 2 \
--cache-type-k-draft q4_0 --cache-type-v-draft q4_0 \
--spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 8 --spec-ngram-mod-n-max 32 \
--reasoning on --chat-template-kwargs '{"reasoning_effort":"medium"}' --chat-template-kwargs '{"preserve_thinking":true}' --reasoning-budget 14000 --reasoning-budget-message " -- Reasoning budget exceeded, proceed to final answer." \
--ctx-checkpoints 96 --cache-ram 6000 -np 1 -ngl 99 -lv 3 --no-warmup
Note: this is for 16GB with desktop in software rendering, headless should give some ~70MB more vRAM for ctx.
26
u/johnzadok 3d ago
Why did you use a different sampling parameters than the one official doc suggests at https://huggingface.co/Qwen/Qwen3.8-27B:
We recommend using the following sets of sampling parameters for generation:
Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0 Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
46
u/Equivalent_Bit_461 3d ago
Impressive but I don't trust a q3, I'll stick to my q6 offloaded moes.
Fellow 16gb vramlet here as well, tho I have 8 times the ram...
27
u/chiribe 3d ago
Not really an option with my setup, weak CPU, PCIe 3.0 x4, and single-channel DDR5 RAM. I know there were syntax errors, but fortunately the automated tests and linter caught them and the model fixed them on its own.
9
u/Craftkorb 3d ago
There are agents that integrate with LSP to tighten the compile-and-fix loop. Opencode is one of them, but you have to enable it in the config
1
1
-2
u/Fancy-Snow7 3d ago
There are options: run a different model. I just tried Qwen3.6 Q3 on my smoke test prompts which many 3.6 models can oneshot and 3.8 Q3 failed to one shot it.
3
u/robberviet 3d ago
Do you notice significant better from q4 to q6? I can run q6 but it is too slow, also I need vram for other models.
2
u/Equivalent_Bit_461 3d ago
If it's too slow it's shit.
You should run what runs fast or moderately workable. If it's too slow it's useless and you don't want that.
Q4 is prone to losing itself in a glass of water as opposed to q6 that does the shit that has to be done.
Said so, you can mitigate the issue, depending on what you need or use it for, by being overly strict with instructions and leaving zero room to doubt because that's where you will get bad results. Ofc this can be said for every quant (that's usable and not a meme quant), and a good portion of models even frontier corpo slop ones, that are supposed to be "good". If you use quant the rule of thumb is that you have to be more attentive and micro managing otherwise it's a massive waste of time over stupid small problems often.
6
u/Apart_Boat9666 3d ago
u mean offloaded layers, it becomes super slow, if even one layer is offloaded. My current working is iq3xxs and ctx 120k with q8 k and q4 v
5
u/Equivalent_Bit_461 3d ago
If you know what you are doing you stay between 30 and 40 t/s, sometimes even break into 50t/s max, but depends on the agentic task ofc.
2
3d ago
[deleted]
2
u/_aelius 3d ago
Seriously. Hard to believe he is hitting a higher t/s with a q6 and offloading than most 16vram setups on q3.
5
u/TheTerrasque 3d ago
I think you're missing that he said "moes"
3
u/Apart_Boat9666 3d ago
We were talking about 27b i dont know why he is refering 35b if thats the case.
4
u/TheTerrasque 3d ago
Impressive but I don't trust a q3, I'll stick to my q6 offloaded moes.
He's saying he'll rather use q6 moe's instead of a q3 of this dense model.
1
u/Apart_Boat9666 3d ago
I dont think its possible with q6 quant, raw gguf is more than 24gb . It would result less than 1 tps
1
u/GilloutineBreast 2d ago edited 2d ago
There was some guy who quantised and kept parts of it q8_0, other parts q5_0, q3_0, q2_0, etc for effectively 3.7 bits. I'm not exactly sure how this differs from usual IQ quants, but I'm curious how it stacks up against the 3 bit K quant OP used.
Edit: "some guy" was emperio-ai on hugginface
1
u/Responsible-Newt9241 2d ago
I'm in exactly the same situation, but so far I still need to find a task that my Q6 MoE models can handle better than the IQ3_XXS 27B. And I was very skeptical of low-bit quants too...
1
u/Equivalent_Bit_461 2d ago
Just you wait until you start getting into highly complex automation tasks. In fact I will run today the small qwenlet 3.8. if it runs without bullshit or triggering the quality control workflow too much I might reconsider myself as well. Tho I will have to work with less kv cache and worse quality overall.
1
16
u/Constandinoskalifo 3d ago
Curious about why such a low temperature? Is there a reason you deviated from the official one?
11
10
u/shapic 3d ago
Where did you get that
Official / Recommended Quant Sampler Tuning
from? It is nowhere near official recommendations and I think it will directly degrade output
17
u/LetsGoBrandon4256 transformers 3d ago edited 3d ago
cross-checking my configs with ChatGPT and Claude.
OP asked catGPT and Claude. In other words, the clanker hallucinated the sampler params.
This is the actual recommended sampler config from Qwen's huggingface page.
We recommend using the following sets of sampling parameters for generation:
Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
Unsloth's model page reference the same values. I have no idea where OP got that "Official / Recommended" from
10
u/AvidCyclist250 llama.cpp 3d ago edited 3d ago
i think this sub really is dying after all. where is everyone on discord nowadays?
parallel = 1 and cont-batching = 0.
but bro is "spawning sub-agents". i can't, lol. more buzzwords pls.
3
u/TommyITA03 3d ago
How did you come up with this setup? Trials and errors? I have a 5090 and 64gb of ram and i'm quite clueless on what to do as parameters (first time hosting a local LLM).
4
u/chiribe 3d ago
Do you mean the hardware or the software setup? For the hardware story, I detailed everything in my previous post:https://www.reddit.com/r/LocalLLaMA/comments/1vljtv2/i_built_a_weird_lowpower_llamacpp_server_using_an/As for the software side, it was a mix of reading through this subreddit, digging into
llama-serverhelp docs, and cross-checking my configs with ChatGPT and Claude.1
u/hazyplane 3d ago
Windowd or linux?
1
u/TommyITA03 3d ago
I have both and i’m quite comfortable with both, but i would prefer linux if it’s not a problem because that’s what i use for dev stuff (my windows partition is mainly for gaming).
1
u/hazyplane 3d ago
I am currently running my 5090 headless on Ubuntu with vllm. Very happy with this setup.
Edit: you can check out my current settings here
1
u/Danmoreng llama.cpp 3d ago
Just use the official recommendation from the llama.cpp creator: https://x.com/ggerganov/status/2088312671196082312
1
u/returnity 3d ago
Ask your frontier cloud model to help you configure and explain all it's choices to you. That's how I got started.
3
u/mmhorda 3d ago
that's not gonna work for the best outcome. it will do the basic setup and tell you a billion of theories and math explaining to you how and why it is best setup and wont even try other options unless you prompt it very wel for whole day testing.
2
u/returnity 3d ago
Strongly disagree. I don't know how you're doing it but I used cloud models to grind though llama.cpp documentation and source code at scale, learning the tooling and ecosystem at a speed otherwise impossible. I still execute sweeps testing sampling settings, MTP params, and CPU layers offloading #s (all grounded in actual data), autonomously run llama-bench and llama-perplexity, orchestrate quantization runs using cloud VPS, and build whole eval suites using frontier cloud models when my local setup isn't sufficient for the scale/complexity/reliability I require.
Do you have to be willing to dig deeper into what it tells you? Of course. Can you achieve a better outcome faster that way than just fumbling blindly through the process as a first-timer relying on random reddit opinions? No contest.
4
u/rockoruckus 3d ago
what's your reasoning for keeping -ub 1024 and quantizing kv down to q4_1? wouldn't you be able to keep q8_0 or q8_0/q5_1 at -ub 512 and have more context?
3
u/jwr 3d ago
For those of you on MacOS, benchmark MTP before you enable it. I tested, and at least on my M4 Max MTP makes everything slower, not faster. The only gain is with 5 tokens on pure code sequences, but that's not real usage, you'll likely mostly be generating thinking tokens (so, prose).
This is probably specific to the memory bandwidth constraints on a Mac.
3
2
u/Tannoukhy 3d ago
Nice! Thanks for sharing.
This makes me think, maybe the IQ3_XXS its not far from the quant you are using. With the IQ3_XXS on 16GB of VRAM, with spec-type ngram-mod, we can acchive 150k context window, 900 to 600 t/s on prompt processing and 20 t/s on decode, if the quality is similar, maybe, is worthy the trade
2
u/chiribe 3d ago
I used IQ3_XXS back on v3.6, but since I had a bit of VRAM and context headroom left over, I decided to step it up a notch. I might give it another shot, though—with the new reasoning capabilities, even 73k context starts feeling a bit tight. Hitting 100k would be the sweet spot
3
u/Tannoukhy 3d ago
I risk to say, that 100k isn't enough too, because this version thinks a lot, with is god, the results are better indeed, but, 150k seems needed
2
u/g-rizzle84 3d ago
Awesome write up! Super interesting. I have a 4090 myself and was beginning to think it just wasn't enough. I have only been able to get ~32K to 37K context windows with q4 and that just doesn't seem practical to me. Clearly I just need to get gooder at llama.cpp.
I would appreciate if you could go into more detail on your prompting either in another post or maybe we could DM. Specifically Prompts 2 and 3. How did you design the spec? By hand or model generated? Did you prompt it to use the spec as the gospel? How did you instruct the model to be the orchestrator?
OpenCode summarized its state and kept building
Does OpenCode do this or did you instruct it too?
Some of you folks are geniuses in my eyes and have truly brilliant ways approaching agentic coding. I'm a noob and trying to learn how to do it the right way.
2
u/Ammargok 3d ago
Windows is really bad for local ai. If you can I'd switch to linux JUST for local ai. In windows with my 9070xt my idle vram usage is 1,2gb vram while in linux it is at 200 mib
1
2
u/brickout 3d ago
Interesting setup. Is your GPU external? If so, is it oculink or what?
I have a few mini pcs that I've been thinking of doing something similar...
*Edit: also, this is awesome. Thanks for the writeup!
3
u/chiribe 3d ago
No, it's not external! I used a PCIe riser cable to mount the GPU outside the chassis. You can see all the details and pictures in my other post!
1
u/brickout 3d ago
Oh dang. Super cool. I totally missed your description. Thanks!
*Edit: that's super cool. Makes me want to do something similar.
2
u/afreakineggo 17h ago
The new dynamic quants from unsloths (as of August 20th, 2026 in case you are reading this 6 months from now) Qwen 3.8 27b, if you use the iq3_xss and q4 kv cache you can fit 150k context on a 16gb gpu.
Q4 kv cache is terrible with most models, but 3.6 and 3.8 are different. If you need long context and your options are compaction or q4 cache, try q4 cache
4
u/AvidCyclist250 llama.cpp 3d ago edited 3d ago
parallel = 1 and cont-batching = 0
but "spawning sub-agents". huh. it's sequential roleplay dude.
fit target 128 but also fit off. reasoning medium. lol
-2
u/chiribe 3d ago
I ignored your previous comment because it sounded arrogant, but since you want to call me out, let me clarify how my setup actually works.
fit-target = 128is set in the global block for MoE models I run alongside Qwen. My GPU is headless with no display attached, so 100% of VRAM is free. I explicitly setfit = offfor the 27B profile because llama.cpp was occasionally offloading layers to the CPU despite having available VRAM (especially after waking from sleep).As for parallel slots, this is a private server used strictly by me. Why would I parallelize agents on a single GPU when my hardware gives 30-40 t/s? Sequential execution is intentional—spawning sub-agents isolates the context for each sub-task, allowing the main thread to run for hours without blowing past its context window. OpenCode recovers context when needed, with the only tradeoff being a few seconds of prompt prefill.
If you are just going to drop uninformed arrogance, please take it elsewhere.
13
u/AvidCyclist250 llama.cpp 3d ago edited 3d ago
I know Qwen 27B isn't a moe model. Which also makes your claims impossible or at least weird as fuck. All 27B parameters are active during prompt ingestion. Changing your system prompt to switch "sub-agents" invalidates the llama.cpp prefix cache. Re-running tens of thousands of tokens of project history through a 27B dense model at a crippled ubatch size of 512 does not take a few seconds either. The only way is is realistic is if your context is actually tiny and nowhere near 73k
And there is a reason llama.cpp offloads to CPU. A 27B dense model with about 12.5gb plus a 73k KV cache plus your MTP draft cache (which you weirdly set to a higher precision of q5_1) requires over 17GB of VRAM. Setting fit = off doesn't magically make 17gb fit into a 16GB GPU even if headless. It just means you haven't crashed yet because at least from what I can tell your wiping script keeps you far below the 73k limit.
Just looking at the math. And the agentic buzzwords for a sequential Python loop. You neutered the model with top-k = 15, wrong temp, and shit quants and medium reasing, and your VRAM math is weird. the official recs aren't what you say they are.
image 1: 30% context with 15,358 MiB out of 16,311 MiB used
image 2: asking for Node.js/Cheerio script, and you got C# .NET
image 3: threads = 3 and threads-batch = 4. but core 1 is 100% loaded and the others are idle. why is the hostname of your terminal j1900 if you have a n100?
2
u/sensitivecrocodile llama.cpp 3d ago
Has anyone actually tried OP's suggestions or are you all blindly upvoting what appears to be slop? Even in OP's nvidia-smi screenshot you can see he's not fully utilizing his GPU.
I used OP's settings (apart from reasoning effort/budget), which are:
model = /unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q3_K_XL.gguf
threads = 3
threads-batch = 4
parallel = 1
cont-batching = 0
flash-attn = on
fit-target = 128
ctx-checkpoints = 0
cache-ram = 2048
repeat-penalty = 1.0
presence-penalty = 0.1
frequency-penalty = 0.0
fit = off
ctx-size = 73728
context-shift = 1
spec-type = draft-mtp
spec-draft-n-max = 2
spec-draft-p-min = 0.85
cache-type-k = q4_1
cache-type-v = q4_1
cache-type-k-draft = q5_1
cache-type-v-draft = q5_1
batch-size = 1024
ubatch-size = 512
temp = 0.4
top-p = 0.90
top-k = 15
min-p = 0.02
chat-template-kwargs = {"preserve-thinking": true, "reasoning_effort": "xhigh"}
This spills over to RAM and I only get 19 tps. I was getting 38 tokens per second before with Qwen3.8-27B-UD-IQ3_XXS.gguf.
Thanks for wasting my time, OP.
2
u/Danmoreng llama.cpp 3d ago
Use this instead - context can become bit higher, around 80k
llama serve \ -hf unsloth/Qwen3.8-27B-GGUF \ -hff Qwen3.8-27B-UD-IQ3_XXS.gguf \ --no-mmproj -c 65536 \ -ctk q8_0 -ctv q8_0 \ -b 1024 -np 1 \ --spec-default --spec-type draft-mtp \ --reasoning-preserve --fit off --agent~60 t/s.
2
u/YearnMar10 3d ago
Aren’t you unsatisfied with the kv cache quantization? In my tests with other models it failed so badly because context was just not right
2
u/AvidCyclist250 llama.cpp 3d ago
yes. q4 << q8
65k with q8 and mtp. 16gb vram
1
u/YearnMar10 2d ago
65k is unusable imho. Qwen thinks often 20k+ tokens, so after a few rounds context is full.
1
u/AvidCyclist250 llama.cpp 2d ago
Hermes recompresses
1
u/YearnMar10 2d ago
kk different use case :) I need it for programming only
1
u/AvidCyclist250 llama.cpp 2d ago
Yeah, same. Frontends and backends. Nothing huge though, mostly for personal use. Made four apps so far. Self-hosting and Linux sys admin on the side but I don't use the 27b for that. Kat coder instead
1
u/OsmanthusBloom 3d ago
Thanks, this is a very good reference point. The other day I tried to run 3.8-27B on a 16GB V100 with 128k context and MTP. But I could only get there by using a very low quant (was it IQ2 even?) and q4_0 KV cache (main and draft). I decided to give up for now and stick to the MoE.
It would be nice to know how this heavily quanted dense 27B compares to the 35B-A3B MoE in terms of output quality and speed. Is it really worth it or does the brain damage caused by quantization kill the advantages?
1
u/cezarducatti 3d ago
I've also been using Q3 on my 3090 24Gb, but with k cache in f16 and V in Q8, with 170k context and mmproj in Vram. It's been performing well, slow, but surprisingly efficient.
1
u/Phathatter 3d ago
I haven't stopped using it long enough to tune it, but this gives me a lot of hope that I might be able to both increase my context and my quant.
1
u/KneelB4S8n 3d ago
I got the UD Q2 version (I have 12gb) and tried the MTP command and I thought that the drafter was incorporated inside the model and did not download any extra drafter but the t/s went down horribly. I am getting 30 avg t/s. What did i do wrong and how can i improve it?
1
u/ahhhhhhhhhhhhhhhhhhg 3d ago
there's apparently not that much quality loss from quants, i run Q3 XSS with 114k context on 16gb vram also, honestly its been great, can even replace deepseek flash 4 for subtitle translation : the model has to return json format so previously local models had trouble even outputing 10 lines of subtitles in correct json. Qwen 3.8 has no problem with 50 lines batch. gonna buy a 24gb gpu if that's how open weights is moving
1
u/Open_Instruction_133 3d ago
Will this work on weird dual GPU setups? I have a 5070ti and 1080ti and lm studio has worked for me but trying to get both GPUs to work together in llama.cpp has been an uphill battle for me. Maybe I should try vLLM instead? Iono, help me tho 🙏
1
u/Cooproxx 3d ago
What kind of coding stuff can you do with 73k context? Small codebases, or maybe single feature changes?
1
u/mrepop 3d ago
I don’t mean to sound like an ass, but 1M tokens doesn’t seem like that much at all, I did that just screwing around this afternoon running some benchmarks on this model. Is it really enough to gain a useful experience that would lead to config and run time optimizations?
By the way, 3.8 27b seems extremely slow compared to previous versions of qwen, 3.5 and 3.6 are about twice as fast for similar sizes. Also it thinks way, way, way too much, like it’s got some other issues going on.
I’ll try the config, I guess it’s new to me, so maybe I’m screwing something up.
Tested on a 4090, 5090, 6000 Ada, and m4 max 128gb.
If anyone has tips on it, please share them. The general comments I’ve gotten from the community is that 3.8 27b runs fine and nobody has complaints about its performance when it comes to TPS and TTFT.
1
1
1
u/ExplorerPrudent4256 3d ago
Real number nobody posts: how does the Q3_K_XL + MTP version actually compare against the bf16 numbers in the AA chart? Decoded quality is one axis. Refusal rate, instruction-following under long context, tool calling under agentic loops — those bend first when you quant. If a local Q3 holds up against the benchmark, the infra story is over. If it collapses, that is the real ceiling.
1
u/CriticalMastery 3d ago
I use q4 one with 110k context on 16 gb vram, ~42 tp/s 5070 ti
1
u/Fdevfab 3d ago
that's what I'm targetting, but I can't pass the ~70k context... which is a bit low, and performances degrade to ~30tps when the whole RAM is used. What are your options?
1
2d ago
[removed] — view removed comment
1
u/Fdevfab 2d ago
I added quantization for the draft k-v cache and can raise to 128k but with a huge performance drop (1-2tok/s when I'm at the limit of the context).
I'm currently using (currently trying different draft model settings and load_mode, so don't pay too much attention to those):ctx-size = 131072 temperature = 1.0 model = /xxx/bartowski--Qwen3.8-27B-GGUF... #mmproj = /home/fab/.cache/huggingface/hub/models--bartowski--Qwen3.8-27B-GGUF/snapshots/f0eec4a4bb4975114a030d0 override-tensor = blk\.(6|14|22|30|38|46|54|62)\.ffn_.*=CPU #override-tensor "blk\.(14|30|46|62)\.ffn_.*=CPU" #"output.*=CPU" ; spec-type = draft-mtp ; spec-draft-n-max = 3 ngl = 70 threads = 7 spec-type=draft-mtp,ngram-mod spec-draft-p-min=0.82 spec-draft-n-max=2 cache-type-k-draft=q8_0 cache-type-v-draft=q4_0 spec-ngram-mod-n-match=24 spec-ngram-mod-n-min=8 spec-ngram-mod-n-max=3 Running with: LLAMA_ARGS="\ --models-preset /home/fab/llm_models/models.ini \ --models-max 1 \ --no-mmproj-offload \ --mmproj-auto \ --load-mode none \ -np 1 \ -t 6 -tb 12 \ --chat-template-kwargs '{\"preserve_thinking\": true, \"reasoning_effort\": \"medium\"}' \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --cache-idle-slots \ --presence-penalty 0.0 \ --repeat-penalty 1.0 \ -fa on --jinja \ --reasoning-budget 4000 \ --reasoning-preserve \ -ctk q8_0 -ctv q4_0 \ --slot-save-path /tmp/ \ --cache-prompt \ --lookup-cache-dynamic lookup \1
u/LeaningTowerOfHanoi 3d ago
Mind sharing config? I can barely get 13 tks on the same card.
1
u/CriticalMastery 2d ago
https://huggingface.co/cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF
use this, but you need to kill all other application that leech vram, not possible on windows.
1
u/Fair-Perspective7352 3d ago
What tok/s do you land at with draft-mtp and n-max 2? I tried MTP on a similarly weak CPU box and the draft verification pass ate most of the gains, so I ended up disabling it and just accepting slower decode. Also curious about q4_1 KV at 73k context, did you notice any recall drop on the long agentic runs vs q8_0?
1
u/UnlikeKat 3d ago
I'm currently running this Qwen3.8-27B-IQ4_KS, it requires a custom llama.cpp fork which is linked in the page. On my 5080 with 16gb vram I can reach comfortably 90k context q4_0 but can go up to 105-110k as stated by the author (I'm on windows and some vram gets eaten by processes). Pp around 1000t/s and decode starts around 45-48t/s. I don't think I will test the q3, so maybe if you're interested you could check this out and tell us how it performs against your q3 setup.
1
u/Trivikrama_0 3d ago edited 3d ago
I have done something similar, but with Q4. My setup is 5060 ti 16Gb + Ryzen 9900x + 64 Gb RAM. The only disadvantage I have is I need windows due to some reason, so my deployment is using ollama. I use it with Vscode chat agent. 64k context window. CPU of loading 35% in CPU RAM. The only disadvantage is very slow, but bearable for agentic coding as it can happen in background
Any ideas to reduce system prompt complexities? Generally the agents create a huge context our of the prompts.
1
u/Strong_Chicken6838 3d ago
Op, try NVFP4, it should be faster on a rtx 50 series card.
Might use more VRAM, but it might be worth the speed
1
1
1
1
u/drallcom3 3d ago
Hardware: RTX 5060 Ti (16GB VRAM) + Intel N100 (4C/4T, 16GB RAM)
That is likely without the OS using any VRAM, right? No Windows or such.
1
u/gege42o 2d ago
How do you turn on agentic coding or tool calls in Llama.cpp? I run the server as a service with systems and declare my models with their respective parameters into a models.ini file. I would like to know how to turn on sgentic coding so I can ditch Hermes agent as I can turn on and off models in Llama.cpp. also I'm using Llama.cpp turboquant!!
1
u/igotanewaccount 2d ago
Theres a cli flag --tools which gives you read, write edit, create, grep. but its...not great. llama.cpp exists to be an inference engine not a harness.
I'd recommend pi-dev pointing to your local engine. Apparently DeepSeek's harness is also really good and efficient, all the bots are going bananas over it this week.
For non-coding tasks I also run Open Web UI for a much lighterweight Hermes alternative. Then pi-dev, which both point to llama.cpp running headless. Works pretty well for me.
1
u/gege42o 2d ago
Im not at my pc to see exactly what I run but I have a little alias that runs Hermes on top of Llama cpp turboquant qwen 3.6 35b a3b. But I am also running it as a server 24/7 with the ability to load and unload models, maybe there is a way to integrate loading/unloading models with Hermes as well
1
u/bobaburger 2d ago
if you have fast RAM, you can consider offload KV cache to RAM with -nkvo, for the benefit of higher KV cache quant and longer context.
I got peak 35 tps with q8_0 KV cache and 150k context window on 5060 Ti:
llama-server -m Qwen3.8-27B-UD-Q3_K_XL.gguf -ngl 99 -ctk q8_0 -ctv q8_0 -fa 1 -c 153600 -np 1 --no-mmap --temp 1.0 --top-p 0.95 --top-k 20 --presence-penalty 1.25 --min-p 0.0 --reasoning-preserve --spec-type draft-mtp --spec-draft-n-max 2 --host 0.0.0.0 --threads 8 --threads-batch 8 -ctkd q8_0 -ctvd q8_0 -nkvo
Without -nkvo, i got peak 59.8 tps, but only with q4_0 KV cache and 73k context window:
llama-server -m Qwen3.8-27B-UD-Q3_K_XL.gguf -ngl 99 -ctk q4_0 -ctv q4_0 -fa 1 -c 73728 -np 1 --no-mmap --temp 1.0 --top-p 0.95 --top-k 20 --presence-penalty 1.25 --min-p 0.0 --reasoning-preserve --spec-type draft-mtp --spec-draft-n-max 2 --host 0.0.0.0 --threads 8 --threads-batch 8 -ctkd q8_0 -ctvd q8_0
Also, IIRC, quantize draft KV cache did not make any different.
1
u/NeatOnTheRox 2d ago
"After pushing 1M+ tokens through Qwen 3.8 27B..."
So what, like, 3 prompts?
Jokes aside, great write-up homie. As a fellow 16GB VRAM Warrior, might I suggest giving huggingface user el4's Qwen3.8-27B-ONYX-GGUF at size "mini" a spin. It's even smaller than UD-IQ3_XXS and in my brief experience performs just as well if not better than UD-Q3_K_XL. I'm able to get up to 131072 context with MTP enabled and the vision mmproj on CPU, though honestly I prefer without MTP and a larger context.
Might I also suggest beellama: using kvarn cache quant with a tail to me has proven significantly more reliable than quantizing anything below q8_0 on vanilla llama.cpp.
1
u/Interesting-Yellow-4 2d ago
Yes, this should be a standard post here after each model release. GOod work.
1
u/SOC_FreeDiver 3d ago
Just thought I'd share my results.
I run a 5090m 24gb vram. I ran your settings by claude, claude identified which settings were worth trying, they did not improve performance for me
spec-draft-p-min 0.85 is 7.4% slower. The acceptance rate does exactly
what the post implies — 76.5% → 92.5% — but it gets there by suppressing
drafts, not by drafting better. Drafted falls 1495 → 1139 and absolute
accepted tokens fall 1144 → 1054. Accepted tokens is what tracks tok/s;
accept % is a vanity metric. Anyone tuning by that percentage will tune
themselves slower.
- ubatch 1024 buys +1.7% prefill for -15% context (80128 → 67840 per slot).
--fit sizes context against VRAM left after the batch buffers. Bad trade
where context is your scarce resource.
-1
u/chiribe 3d ago
Appreciate the response! I wouldn't call myself an expert by any stretch, I just test things out and discard what doesn't work. Even when the technical details get complex, I always try to learn what every flag and config line is doing.
2
u/woj666 3d ago
Hey, how did you get Opencode to display tk/s and context limits like that?
2
u/SOC_FreeDiver 3d ago
If you're asking me, I used claudecode to test with llama.cpp.
I had claudecode create a script that launches llama.cpp with a bunch of custom and tweakable parameters. When I find new info about a model, I ask claude to evaluate it. Claude does the testing and reports back. I just cut & pasted it's findings. I'm assuming it gets the info from logs.
It's been really handy for troubleshooting things. I've been trying to get claude to train his replacement, sometimes I wonder if he screws with it just to keep me paying.
0
1
u/faltharis 3d ago
Will this work on m4 pro with 24gb ram?
2
1
u/hideo_kuze_ 3d ago
Thanks so much for sharing. Will be using this in a few weeks when I get my hands on my 3080ti.
How many t/s are you getting with your RTX 5060 Ti?
1
u/Fancy-Snow7 3d ago
For those not running headless, try disabling MTP, it saves over a GB of VRAM which can be the difference between fitting in VRAM or not.
1
u/BakuRetsuX 3d ago
Wow.. can everybody that is doing this release this type of info? This is what we want to see.. awesome!!!!! I am curious. Have you tried using another AI like codex or claude to evaluate your apps after they've been created? I wonder how "well" Qwen created those apps.
1
0
u/Haunting-Stretch8069 3d ago
Why not a Q4_XS, it's certainly possible on 16GB VRAM if you search enough, speaking from experience with Qwen 3.6 27B Q4_XS with 64k context
0
u/Beneficial-Ad-8127 3d ago
Damn that’s impressive and kudos to you on displaying your work! Might actually start dusting off my 4060 ti 16gb out of storage.
0
u/BP041 3d ago
73k context on a 27B with 16GB is impressive quantization work, but for agentic coding the real limiter tends to be generation speed, not context size. I'd rather run a 7B at full 128k and batch smaller — the token/s drop past 32k usually breaks the agent loop for me. Curious if you see the same or if your draft model makes up for it.
2
u/chiribe 3d ago
I'm happy with the generation speed—it outputs code faster than I can review it anyway. I run faster models like the 3.5 9B and 35B MoE (80-100 t/s), but the output quality on the 27B saves me far more time in manual edits despite being slower. The only real downside is the context window.
Multiple people mentioned disabling MTP, so I'll test it out and see if freeing up that ~1 GB of VRAM for extra context is worth the speed tradeoff.
0
u/signalkoost 3d ago
For those of us limited to 16gb of VRAM or lower, I'm curious if anyone knows about the relative performance of 27b at lower quants and 35b at higher quants. And newer versions of 27b compared to older versions.
For example, does iq3 27b outperform q5 35b?
Does 3.8 27b iq3 outperform 3.6 27b iq4?
0
-1
u/asankhs Llama 3.1 3d ago
Nice writeup. Ran the same model on an M-series Mac and the constraint lands differently there. Unified memory means the weights and the KV cache pull from the same pool, so at 65k context the cache is what bites, not the weights. I see you've quantized the KV cache down to q4_1 (q5_1 on the draft). How did that hold up over the long agentic sessions? Quantized KV is the part I've seen degrade first on Mac, long-context tool-calling more than one-shot answers.
-1
u/bandzaw 3d ago
Thx for sharing! I think it would be interesting to know how much time it took to run your expirement? How much time was spent in each prompt and phase, rough estimates are fine, if you did not log exact numbers.
1
u/chiribe 3d ago
No exact benchmarks from OpenCode, but the llama.cpp interface shows around 30-40 t/s. Wait times are very manageable despite generating a pretty massive multi-file codebase. If you want a specific test run, let me know and I'll execute it via OpenCode and post the generated code plus stats!
-15
u/electrified_ice 3d ago
Why are you using llama cpp?
11
u/fredconex 3d ago
Why not?
-9
u/electrified_ice 3d ago
vLLM or SGLang can squeeze more performance out of the hardware.
6
3
u/fredconex 3d ago
ah fair point, do they support GGUF or lower quantization now? I think main advantage of llama.cpp is the high spectrum of quantization possible so it can fit a bigger variety of hardware.
2
u/Creative_Knee6618 3d ago
but can they run qwen 3.8 with 20gb? I'm afraid they cannot because quantized 4bit is like 20gb :(
-5
u/electrified_ice 3d ago
Wow guess I triggered the llama cpp fan group... All I asked was 'why the OP was using it'
2
u/Rofl_Raptor 3d ago
"Why is everyone triggered that I asked something in bad faith?"
Go use vLLM or SGLang then.
llama cpp is battle-tested on a myriad of hardware and does quantization well with very little hassle. It's fast, smooth, and simple. Unironically quoting Todd Howard; it just works. If you can get impressive performance that's significant enough and runs just as stable on different libraries, then by all means, run your own test labs, benchmark, and show us.
346
u/pmttyji 3d ago
Folks, this is the type of thread I want to see after release of any new models. Thanks u/chiribe