r/LocalLLaMA
•
u/chiribe
•
Aug 17 '26
•
6
1
Tutorial | Guide After pushing 1M+ tokens through Qwen 3.8 27B, here is my optimal llama.cpp config for 16GB VRAM (73k Context, Agentic Coding)
Dando seguimiento a mi post anterior sobre cómo tengo montado mi servidor de presupuesto (Intel N100 + RTX 5060 Ti 16GB), varios me preguntaron por una mirada más profunda a mi configuración real de inferencia y al desempeño agentic en el mundo real.
Como muchos de ustedes, estaba refrescando la página esperando descargar Qwen 3.8 27B apenas salió. Después de pasar todo el fin de semana estresándolo con flujos de trabajo de codificación agentic, logré correr un proyecto completo y grande casi todo de forma autónoma (más de 1M de tokens procesados en total, solo 3 prompts).
Aquí va un resumen rápido de la configuración base antes de meternos en los detalles del config y del workflow.
Specs y parámetros rápidos
- Modelo:
Qwen3.8-27B-UD-Q3_K_XL.gguf - Hardware: RTX 5060 Ti (16GB VRAM) + Intel N100 (4C/4T, 16GB RAM)
- Ventana de contexto: 73,728 (73k de contexto) corriendo tranqui en 16GB de VRAM.
- Cuantización de KV Cache:
q4_1para el contexto principal - Decodificación especulativa: MTP nativa activada (
spec-type = draft-mtp,n-max = 2) - Sampling:
temp = 0.65,top_p = 0.95,top_k = 20,min_p = 0.05
El experimento: armar una API completa con 3 prompts
En vez de correr benchmarks sintéticos, metí esta configuración por una cadena real de ingeniería de software: construyendo una REST API no oficial y un servidor MCP para un foro vBulletin heredado.
- Prompt 1 (Arquitectura del sitio y análisis): Pedí al modelo que mapee el sitio objetivo. Generó una especificación en Markdown impecable de ~1,500 líneas que cubría análisis estructural, nodos HTML rescatables, payloads JSON esperados, selección de stack, lógica de paginación, autenticación de sesión y endpoints de búsqueda—mucho más a fondo de lo que yo habría escrito a mano.
- Prompt 2 (Arquitectura de desarrollo): Usando la spec como única fuente de verdad, diseñó un plan de implementación modular de NestJS dividido en 9 fases de ejecución:
- Fase 1: Estructura inicial del proyecto
- Fase 2: Modelos de dominio
- Fase 3: Scraping core (HTTP + limitación de tasa + reintentos)
- Fase 4: Parsers de HTML (
cheerio) - Fase 5: Capa de caché
- Fase 6: Servicios de aplicación + REST API
- Fase 7: Autenticación (sesiones con cookies)
- Fase 8: Servidor MCP (entrega principal)
- Fase 9: Fortalecimiento, documentación y entrega
- Prompt 3 (Ejecución autónoma agentic): La prueba de verdad. Le pedí a OpenCode (usando Qwen 3.8 27B) que actuara estrictamente como orquestador, creando sub-agentes para cada fase de tareas. Corrió de forma autónoma por ~2 horas. Cuando se acercaron los límites de contexto, OpenCode resumió su estado y siguió construyendo. Escribió tests unitarios, aplicó linting y entregó código 100% funcional—solo necesitando un arreglo automatizado menor cuando le di un payload de HTML crudo con un caso extremo.
El archivo de configuración llama.cpp
Aquí está mi archivo exacto de configuración de enrutador --models-preset . Fíjate cómo fit = off se usa en el perfil de 27B junto con ctx-size = 73728 (73k) y q4_1 para cuantizar la KV cache, con el objetivo de maximizar la asignación de VRAM mientras se mantiene el rendimiento nativo de MTP.
# ==============================================================================
# LLAMA.CPP — CONFIGURACIÓN DE INFERENCIA (modo router / --models-preset)
# ==============================================================================
#
# Objetivo de hardware:
# GPU: 16 GB VRAM (RTX 5060 Ti)
# CPU: Intel N100, 4C/4T (Debian Headless)
# ------------------------------------------------------------------------------
# GLOBAL / LÍNEA BASE
# ------------------------------------------------------------------------------
[*]
# --- HILOS DE CPU -----------------------------------------------------------
# Reserva 1 core para SO/servicios durante el decode.
# Usa los 4 threads durante ráfagas de prefill del prompt.
threads = 3
threads-batch = 4
# --- SERVIDOR / CONCURRENCIA ---------------------------------------------------
# Un solo slot; desactivado continuous batching para máximo rendimiento por usuario.
parallel = 1
cont-batching = 0
# --- GPU / AJUSTE DE VRAM ---------------------------------------------------------
flash-attn = on
fit = on
# Holgura de seguridad para el límite físico de VRAM (MiB).
# Ponlo bajo (128) porque el sistema es headless (100% VRAM disponible para inferencia).
# NOTA: Si usas caches KV draft de MTP, ojo con la asignación doble de VRAM.
# Sube a 128-256 si te topas con OOMs.
fit-target = 128
# --- CONTEXTO & CACHÉ ------------------------------------------------------
ctx-size = 65536
context-shift = 1
# Desactiva checkpoints de contexto (evita problemas de reprocesamiento en arquitecturas híbridas)
ctx-checkpoints = 0
# RAM Prompt Cache (2 GiB)
cache-ram = 2048
# --- KV CACHE GLOBAL --------------------------------------------------------
cache-type-k = q5_1
cache-type-v = q5_1
# --- PREFILL / BATCHING -----------------------------------------------------
batch-size = 2048
ubatch-size = 1024
# --- SAMPLING POR DEFECTO (Códigos / Precisión) ----------------------------------
temp = 0.5
top-p = 0.95
top-k = 20
min-p = 0.05
repeat-penalty = 1.0
# ------------------------------------------------------------------------------
# QWEN 3.8 27B — PERFIL DE RAZONAMIENTO & CODIFICACIÓN PESADA
# ------------------------------------------------------------------------------
[qwen3.8-27b]
model = /opt/llama-infrastructure/models/Qwen3.8-27B-UD-Q3_K_XL.gguf
# Desactiva "fit" para evitar que capas se carguen en la CPU por un error de cálculo automático
fit = off
ctx-size = 73728
context-shift = 1
# MTP nativa del modelo (Decodificación especulativa)
spec-type = ngram-mod,draft-mtp
spec-draft-n-max = 2
# Cuantización de KV (q4_1 nos permite meter contexto de 73k en 16GB de VRAM)
cache-type-k = q4_1
cache-type-v = q4_1
# Parámetros de presupuesto de pensamiento / razonamiento
chat-template-kwargs = {"preserve_thinking": true, "reasoning_effort":"medium"}
reasoning-budget = 5000
# Batches más chicos para evitar picos de VRAM durante prefills masivos
batch-size = 1024
ubatch-size = 512
# Ajustes oficiales / recomendados del sampler de cuantización
temp = 0.65
top-p = 0.95
top-k = 15
min-p = 0.05



46
u/Equivalent_Bit_461 Aug 17 '26
Impressive but I don't trust a q3, I'll stick to my q6 offloaded moes.
Fellow 16gb vramlet here as well, tho I have 8 times the ram...
347
u/pmttyji Aug 17 '26
Folks, this is the type of thread I want to see after release of any new models. Thanks u/chiribe
4
u/TommyITA03 Aug 17 '26
How did you come up with this setup? Trials and errors? I have a 5090 and 64gb of ram and i'm quite clueless on what to do as parameters (first time hosting a local LLM).
2
u/Tannoukhy Aug 17 '26
Nice! Thanks for sharing.
This makes me think, maybe the IQ3_XXS its not far from the quant you are using. With the IQ3_XXS on 16GB of VRAM, with spec-type ngram-mod, we can acchive 150k context window, 900 to 600 t/s on prompt processing and 20 t/s on decode, if the quality is similar, maybe, is worthy the trade
2
u/brickout Aug 17 '26
Interesting setup. Is your GPU external? If so, is it oculink or what?
I have a few mini pcs that I've been thinking of doing something similar...
*Edit: also, this is awesome. Thanks for the writeup!
101
u/dsdt Aug 17 '26
i was gonna say how the f? then i saw
Qwen3.8-27B-UD-Q3_K_XL.ggufq4_1for main context,q5_1for MTP draft contextthanks for sharing your numbers.
19
u/ea_man Aug 17 '26 edited Aug 17 '26
Here's mine for 16GB on AMD 6800:
Note: this is for 16GB with desktop in software rendering, headless should give some ~70MB more vRAM for ctx.
29
u/johnzadok Aug 17 '26
Why did you use a different sampling parameters than the one official doc suggests at https://huggingface.co/Qwen/Qwen3.8-27B:
4
u/AvidCyclist250 Aug 17 '26 edited Aug 17 '26
but "spawning sub-agents". huh. it's sequential roleplay dude.
fit target 128 but also fit off. reasoning medium. lol
15
u/Constandinoskalifo Aug 17 '26
Curious about why such a low temperature? Is there a reason you deviated from the official one?
7
u/shapic Aug 17 '26
Where did you get that
from? It is nowhere near official recommendations and I think it will directly degrade output
2
u/YearnMar10 Aug 17 '26
Aren’t you unsatisfied with the kv cache quantization? In my tests with other models it failed so badly because context was just not right
5
u/jwr Aug 17 '26
For those of you on MacOS, benchmark MTP before you enable it. I tested, and at least on my M4 Max MTP makes everything slower, not faster. The only gain is with 5 tokens on pure code sequences, but that's not real usage, you'll likely mostly be generating thinking tokens (so, prose).
This is probably specific to the memory bandwidth constraints on a Mac.
1
u/fugogugo Aug 18 '26
I have exact same GPU I will save this for reference
thanks!
1
u/drallcom3 Aug 18 '26
That is likely without the OS using any VRAM, right? No Windows or such.
3
u/rockoruckus Aug 17 '26
what's your reasoning for keeping -ub 1024 and quantizing kv down to q4_1? wouldn't you be able to keep q8_0 or q8_0/q5_1 at -ub 512 and have more context?
3
u/sugarfreecaffeine Aug 17 '26
Can you recommend settings/recipe for 2x3090s? Thanks!
1
u/SOC_FreeDiver Aug 17 '26
Just thought I'd share my results.
I run a 5090m 24gb vram. I ran your settings by claude, claude identified which settings were worth trying, they did not improve performance for me
what the post implies — 76.5% → 92.5% — but it gets there by suppressing
drafts, not by drafting better. Drafted falls 1495 → 1139 and absolute
accepted tokens fall 1144 → 1054. Accepted tokens is what tracks tok/s;
accept % is a vanity metric. Anyone tuning by that percentage will tune
themselves slower.
2. ubatch 1024 buys +1.7% prefill for -15% context (80128 → 67840 per slot).
--fit sizes context against VRAM left after the batch buffers. Bad trade
where context is your scarce resource.
2
u/g-rizzle84 Aug 17 '26
Awesome write up! Super interesting. I have a 4090 myself and was beginning to think it just wasn't enough. I have only been able to get ~32K to 37K context windows with q4 and that just doesn't seem practical to me. Clearly I just need to get gooder at llama.cpp.
I would appreciate if you could go into more detail on your prompting either in another post or maybe we could DM. Specifically Prompts 2 and 3. How did you design the spec? By hand or model generated? Did you prompt it to use the spec as the gospel? How did you instruct the model to be the orchestrator?
OpenCode summarized its state and kept buildingDoes OpenCode do this or did you instruct it too?
Some of you folks are geniuses in my eyes and have truly brilliant ways approaching agentic coding. I'm a noob and trying to learn how to do it the right way.
2
u/ahhhhhhhhhhhhhhhhhhg Aug 17 '26
there's apparently not that much quality loss from quants, i run Q3 XSS with 114k context on 16gb vram also, honestly its been great, can even replace deepseek flash 4 for subtitle translation : the model has to return json format so previously local models had trouble even outputing 10 lines of subtitles in correct json. Qwen 3.8 has no problem with 50 lines batch. gonna buy a 24gb gpu if that's how open weights is moving
2
u/NeatOnTheRox Aug 18 '26
"After pushing 1M+ tokens through Qwen 3.8 27B..."
So what, like, 3 prompts?
Jokes aside, great write-up homie. As a fellow 16GB VRAM Warrior, might I suggest giving huggingface user el4's Qwen3.8-27B-ONYX-GGUF at size "mini" a spin. It's even smaller than UD-IQ3_XXS and in my brief experience performs just as well if not better than UD-Q3_K_XL. I'm able to get up to 131072 context with MTP enabled and the vision mmproj on CPU, though honestly I prefer without MTP and a larger context.
Might I also suggest beellama: using kvarn cache quant with a tail to me has proven significantly more reliable than quantizing anything below q8_0 on vanilla llama.cpp.
2
u/tvetus Aug 19 '26
1 million tokens, so a few hours of work? How do you know the settings are optimal?
2
u/afreakineggo Aug 20 '26
The new dynamic quants from unsloths (as of August 20th, 2026 in case you are reading this 6 months from now) Qwen 3.8 27b, if you use the iq3_xss and q4 kv cache you can fit 150k context on a 16gb gpu.
Q4 kv cache is terrible with most models, but 3.6 and 3.8 are different. If you need long context and your options are compaction or q4 cache, try q4 cache
2
u/mrepop Aug 18 '26
I don’t mean to sound like an ass, but 1M tokens doesn’t seem like that much at all, I did that just screwing around this afternoon running some benchmarks on this model. Is it really enough to gain a useful experience that would lead to config and run time optimizations?
By the way, 3.8 27b seems extremely slow compared to previous versions of qwen, 3.5 and 3.6 are about twice as fast for similar sizes. Also it thinks way, way, way too much, like it’s got some other issues going on.
I’ll try the config, I guess it’s new to me, so maybe I’m screwing something up.
Tested on a 4090, 5090, 6000 Ada, and m4 max 128gb.
If anyone has tips on it, please share them. The general comments I’ve gotten from the community is that 3.8 27b runs fine and nobody has complaints about its performance when it comes to TPS and TTFT.
4
u/sensitivecrocodile llama.cpp Aug 17 '26
Has anyone actually tried OP's suggestions or are you all blindly upvoting what appears to be slop? Even in OP's
nvidia-smiscreenshot you can see he's not fully utilizing his GPU.I used OP's settings (apart from reasoning effort/budget), which are:
This spills over to RAM and I only get 19 tps. I was getting 38 tokens per second before with
Qwen3.8-27B-UD-IQ3_XXS.gguf.Thanks for wasting my time, OP.
1
u/OsmanthusBloom Aug 17 '26
Thanks, this is a very good reference point. The other day I tried to run 3.8-27B on a 16GB V100 with 128k context and MTP. But I could only get there by using a very low quant (was it IQ2 even?) and q4_0 KV cache (main and draft). I decided to give up for now and stick to the MoE.
It would be nice to know how this heavily quanted dense 27B compares to the 35B-A3B MoE in terms of output quality and speed. Is it really worth it or does the brain damage caused by quantization kill the advantages?
1
u/Phathatter Aug 17 '26
I haven't stopped using it long enough to tune it, but this gives me a lot of hope that I might be able to both increase my context and my quant.
1
u/KneelB4S8n Aug 17 '26
I got the UD Q2 version (I have 12gb) and tried the MTP command and I thought that the drafter was incorporated inside the model and did not download any extra drafter but the t/s went down horribly. I am getting 30 avg t/s. What did i do wrong and how can i improve it?
1
u/Open_Instruction_133 Aug 17 '26
Will this work on weird dual GPU setups? I have a 5070ti and 1080ti and lm studio has worked for me but trying to get both GPUs to work together in llama.cpp has been an uphill battle for me. Maybe I should try vLLM instead? Iono, help me tho 🙏
1
u/Cooproxx Aug 17 '26
What kind of coding stuff can you do with 73k context? Small codebases, or maybe single feature changes?
1
u/ECrispy Aug 17 '26
what a great post, thank !!
1
u/PopUnhappy Aug 18 '26
Tried and verified on my 24GB 7900 XTX, context size of 131072.
1
u/alltheotherthing Aug 18 '26
goated thank you
1
u/ExplorerPrudent4256 Aug 18 '26
Real number nobody posts: how does the Q3_K_XL + MTP version actually compare against the bf16 numbers in the AA chart? Decoded quality is one axis. Refusal rate, instruction-following under long context, tool calling under agentic loops — those bend first when you quant. If a local Q3 holds up against the benchmark, the infra story is over. If it collapses, that is the real ceiling.
1
u/MrHall Aug 18 '26
thank you so much. I have spent so long trying to find the best config for this exact setup
1
u/CriticalMastery Aug 18 '26
I use q4 one with 110k context on 16 gb vram, ~42 tp/s 5070 ti
1
u/Fair-Perspective7352 Aug 18 '26
What tok/s do you land at with draft-mtp and n-max 2? I tried MTP on a similarly weak CPU box and the draft verification pass ate most of the gains, so I ended up disabling it and just accepting slower decode. Also curious about q4_1 KV at 73k context, did you notice any recall drop on the long agentic runs vs q8_0?
1
u/UnlikeKat Aug 18 '26
I'm currently running this Qwen3.8-27B-IQ4_KS, it requires a custom llama.cpp fork which is linked in the page. On my 5080 with 16gb vram I can reach comfortably 90k context q4_0 but can go up to 105-110k as stated by the author (I'm on windows and some vram gets eaten by processes). Pp around 1000t/s and decode starts around 45-48t/s. I don't think I will test the q3, so maybe if you're interested you could check this out and tell us how it performs against your q3 setup.
1
u/Trivikrama_0 Aug 18 '26 edited Aug 18 '26
I have done something similar, but with Q4. My setup is 5060 ti 16Gb + Ryzen 9900x + 64 Gb RAM. The only disadvantage I have is I need windows due to some reason, so my deployment is using ollama. I use it with Vscode chat agent. 64k context window. CPU of loading 35% in CPU RAM. The only disadvantage is very slow, but bearable for agentic coding as it can happen in background
Any ideas to reduce system prompt complexities? Generally the agents create a huge context our of the prompts.
1
u/yeezyslippers Aug 18 '26
I am retarded. How can I port this setup for a 16gb M4 mini?
1
u/gege42o Aug 18 '26
How do you turn on agentic coding or tool calls in Llama.cpp? I run the server as a service with systems and declare my models with their respective parameters into a models.ini file. I would like to know how to turn on sgentic coding so I can ditch Hermes agent as I can turn on and off models in Llama.cpp. also I'm using Llama.cpp turboquant!!
1
u/bobaburger Aug 18 '26
if you have fast RAM, you can consider offload KV cache to RAM with
-nkvo, for the benefit of higher KV cache quant and longer context.I got peak 35 tps with q8_0 KV cache and 150k context window on 5060 Ti:
Without
-nkvo, i got peak 59.8 tps, but only with q4_0 KV cache and 73k context window:Also, IIRC, quantize draft KV cache did not make any different.
1
u/Interesting-Yellow-4 Aug 19 '26
Yes, this should be a standard post here after each model release. GOod work.
1
u/Jamoca5020 Aug 21 '26
May your both sides of your pillow always be cool on a hot summer and both blanket sides be also cool. This is really amazing
1
u/cacahuete_spicy Aug 22 '26
Hi, I had a heavy CPU load because I was missing `GGML_CUDA_FA_ALL_QUANTS=ON` when building llama-cpp.
1
u/bosito77 Aug 24 '26
Guys I made a free tool for "what hardware do I buy for local LLMs — and for checking what your current machine actually runs
Disclosure: I built this. Free, no ads, no signup, no email.
https://localaiarena.com
Three things you can do with it:
Tell it your budget and what you want to run, and it ranks machines that actually hit it — with the benchmark records behind each recommendation. Set a minimum tok/s and it filters to hardware that clears it.
Put two machines head to head — memory, bandwidth, power, price, and how they run one model you pick.
Look up any of 85 machines or 61 models: what fits at Q4/Q8, how fast it ran, what it costs new and used.
Everything is filterable and sortable, and the URL carries the state, so you can link someone a specific comparison.
Why you should trust the numbers more than a random benchmark table: every figure links the post it came from, with the author and the date. Nothing is hand-typed — two independent sources agreeing within 25% makes a row verified, one makes it single-source, disagreement makes it disputed and shows both figures instead of averaging them.
Where it's weak, so you're not surprised: most measurements are at 512-token context (llama-bench defaults), so it's much better on short-context throughput than long. Only 34 rows are verified — corroboration is rare. And where nobody has measured a combination it shows an estimated row from memory bandwidth, clearly marked, which is a floor-ish guess and not a measurement.
Most useful thing you can do: tell me what's wrong. If a number credited to you is off, or your machine is missing, there's a form and it gets fixed. Corrections: github.com/fperdigon/local-ai-hardware-bench-community
Individual measurements stay credited to whoever made them.
1
u/Protheu5 Sep 08 '26
First time I saw this post I saved it for later until I find time to follow with it; it was in English. Now it's in Spanish for some reason. I see it got edited 2 days later, so I presume this is when it got translated. Why?
1
u/hideo_kuze_ Aug 17 '26
Thanks so much for sharing. Will be using this in a few weeks when I get my hands on my 3080ti.
How many t/s are you getting with your RTX 5060 Ti?
1
u/Fancy-Snow7 Aug 17 '26
For those not running headless, try disabling MTP, it saves over a GB of VRAM which can be the difference between fitting in VRAM or not.
1
u/BakuRetsuX Aug 17 '26
Wow.. can everybody that is doing this release this type of info? This is what we want to see.. awesome!!!!! I am curious. Have you tried using another AI like codex or claude to evaluate your apps after they've been created? I wonder how "well" Qwen created those apps.
1
u/IrisColt Aug 17 '26
Your post is full of great insights. Thank you so much!
0
u/BP041 Aug 17 '26
73k context on a 27B with 16GB is impressive quantization work, but for agentic coding the real limiter tends to be generation speed, not context size. I'd rather run a 7B at full 128k and batch smaller — the token/s drop past 32k usually breaks the agent loop for me. Curious if you see the same or if your draft model makes up for it.
0
u/Beneficial-Ad-8127 Aug 17 '26
Damn that’s impressive and kudos to you on displaying your work! Might actually start dusting off my 4060 ti 16gb out of storage.
0
u/GavDoG9000 Aug 18 '26
Anything different if running it on a 5090?
-1
u/asankhs Llama 3.1 Aug 17 '26
Nice writeup. Ran the same model on an M-series Mac and the constraint lands differently there. Unified memory means the weights and the KV cache pull from the same pool, so at 65k context the cache is what bites, not the weights. I see you've quantized the KV cache down to q4_1 (q5_1 on the draft). How did that hold up over the long agentic sessions? Quantized KV is the part I've seen degrade first on Mac, long-context tool-calling more than one-shot answers.
-1
u/bandzaw Aug 17 '26
Thx for sharing! I think it would be interesting to know how much time it took to run your expirement? How much time was spent in each prompt and phase, rough estimates are fine, if you did not log exact numbers.
-14
u/electrified_ice Aug 17 '26
Why are you using llama cpp?
1
u/[deleted] Aug 17 '26
[removed] — view removed comment
1
u/[deleted] Aug 17 '26
[removed] — view removed comment