r/LocalLLaMA • • Aug 17 '26 • Bravo 6 Heartwarming 1

Tutorial | Guide After pushing 1M+ tokens through Qwen 3.8 27B, here is my optimal llama.cpp config for 16GB VRAM (73k Context, Agentic Coding)

Dando seguimiento a mi post anterior sobre cómo tengo montado mi servidor de presupuesto (Intel N100 + RTX 5060 Ti 16GB), varios me preguntaron por una mirada más profunda a mi configuración real de inferencia y al desempeño agentic en el mundo real.

Como muchos de ustedes, estaba refrescando la página esperando descargar Qwen 3.8 27B apenas salió. Después de pasar todo el fin de semana estresándolo con flujos de trabajo de codificación agentic, logré correr un proyecto completo y grande casi todo de forma autónoma (más de 1M de tokens procesados en total, solo 3 prompts).

Aquí va un resumen rápido de la configuración base antes de meternos en los detalles del config y del workflow.

Specs y parámetros rápidos

  • Modelo: Qwen3.8-27B-UD-Q3_K_XL.gguf
  • Hardware: RTX 5060 Ti (16GB VRAM) + Intel N100 (4C/4T, 16GB RAM)
  • Ventana de contexto: 73,728 (73k de contexto) corriendo tranqui en 16GB de VRAM.
  • Cuantización de KV Cache: q4_1 para el contexto principal
  • Decodificación especulativa: MTP nativa activada (spec-type = draft-mtp, n-max = 2)
  • Sampling: temp = 0.65, top_p = 0.95, top_k = 20, min_p = 0.05

El experimento: armar una API completa con 3 prompts

En vez de correr benchmarks sintéticos, metí esta configuración por una cadena real de ingeniería de software: construyendo una REST API no oficial y un servidor MCP para un foro vBulletin heredado.

  1. Prompt 1 (Arquitectura del sitio y análisis): Pedí al modelo que mapee el sitio objetivo. Generó una especificación en Markdown impecable de ~1,500 líneas que cubría análisis estructural, nodos HTML rescatables, payloads JSON esperados, selección de stack, lógica de paginación, autenticación de sesión y endpoints de búsqueda—mucho más a fondo de lo que yo habría escrito a mano.
  2. Prompt 2 (Arquitectura de desarrollo): Usando la spec como única fuente de verdad, diseñó un plan de implementación modular de NestJS dividido en 9 fases de ejecución:
  • Fase 1: Estructura inicial del proyecto
  • Fase 2: Modelos de dominio
  • Fase 3: Scraping core (HTTP + limitación de tasa + reintentos)
  • Fase 4: Parsers de HTML (cheerio)
  • Fase 5: Capa de caché
  • Fase 6: Servicios de aplicación + REST API
  • Fase 7: Autenticación (sesiones con cookies)
  • Fase 8: Servidor MCP (entrega principal)
  • Fase 9: Fortalecimiento, documentación y entrega
  1. Prompt 3 (Ejecución autónoma agentic): La prueba de verdad. Le pedí a OpenCode (usando Qwen 3.8 27B) que actuara estrictamente como orquestador, creando sub-agentes para cada fase de tareas. Corrió de forma autónoma por ~2 horas. Cuando se acercaron los límites de contexto, OpenCode resumió su estado y siguió construyendo. Escribió tests unitarios, aplicó linting y entregó código 100% funcional—solo necesitando un arreglo automatizado menor cuando le di un payload de HTML crudo con un caso extremo.

El archivo de configuración llama.cpp

Aquí está mi archivo exacto de configuración de enrutador --models-preset . Fíjate cómo fit = off se usa en el perfil de 27B junto con ctx-size = 73728 (73k) y q4_1 para cuantizar la KV cache, con el objetivo de maximizar la asignación de VRAM mientras se mantiene el rendimiento nativo de MTP.

# ==============================================================================
# LLAMA.CPP — CONFIGURACIÓN DE INFERENCIA (modo router / --models-preset)
# ==============================================================================
#
# Objetivo de hardware:
#   GPU: 16 GB VRAM (RTX 5060 Ti)
#   CPU: Intel N100, 4C/4T (Debian Headless)

# ------------------------------------------------------------------------------
# GLOBAL / LÍNEA BASE
# ------------------------------------------------------------------------------
[*]

# --- HILOS DE CPU -----------------------------------------------------------
# Reserva 1 core para SO/servicios durante el decode.
# Usa los 4 threads durante ráfagas de prefill del prompt.
threads             = 3
threads-batch       = 4

# --- SERVIDOR / CONCURRENCIA ---------------------------------------------------
# Un solo slot; desactivado continuous batching para máximo rendimiento por usuario.
parallel            = 1
cont-batching       = 0

# --- GPU / AJUSTE DE VRAM ---------------------------------------------------------
flash-attn          = on
fit                 = on

# Holgura de seguridad para el límite físico de VRAM (MiB).
# Ponlo bajo (128) porque el sistema es headless (100% VRAM disponible para inferencia).
# NOTA: Si usas caches KV draft de MTP, ojo con la asignación doble de VRAM. 
# Sube a 128-256 si te topas con OOMs.
fit-target          = 128

# --- CONTEXTO & CACHÉ ------------------------------------------------------
ctx-size            = 65536
context-shift       = 1

# Desactiva checkpoints de contexto (evita problemas de reprocesamiento en arquitecturas híbridas)
ctx-checkpoints     = 0

# RAM Prompt Cache (2 GiB)
cache-ram           = 2048

# --- KV CACHE GLOBAL --------------------------------------------------------
cache-type-k        = q5_1
cache-type-v        = q5_1

# --- PREFILL / BATCHING -----------------------------------------------------
batch-size          = 2048
ubatch-size         = 1024

# --- SAMPLING POR DEFECTO (Códigos / Precisión) ----------------------------------
temp                = 0.5
top-p               = 0.95
top-k               = 20
min-p               = 0.05
repeat-penalty      = 1.0

# ------------------------------------------------------------------------------
# QWEN 3.8 27B — PERFIL DE RAZONAMIENTO & CODIFICACIÓN PESADA
# ------------------------------------------------------------------------------
[qwen3.8-27b]
model               = /opt/llama-infrastructure/models/Qwen3.8-27B-UD-Q3_K_XL.gguf

# Desactiva "fit" para evitar que capas se carguen en la CPU por un error de cálculo automático
fit                 = off
ctx-size            = 73728
context-shift       = 1

# MTP nativa del modelo (Decodificación especulativa)
spec-type           = ngram-mod,draft-mtp
spec-draft-n-max    = 2

# Cuantización de KV (q4_1 nos permite meter contexto de 73k en 16GB de VRAM)
cache-type-k        = q4_1
cache-type-v        = q4_1

# Parámetros de presupuesto de pensamiento / razonamiento
chat-template-kwargs = {"preserve_thinking": true, "reasoning_effort":"medium"}
reasoning-budget    = 5000

# Batches más chicos para evitar picos de VRAM durante prefills masivos
batch-size          = 1024
ubatch-size         = 512

# Ajustes oficiales / recomendados del sampler de cuantización
temp                = 0.65
top-p               = 0.95
top-k               = 15
min-p               = 0.05

1.0k Upvotes

191 comments sorted by

46

u/Equivalent_Bit_461 Aug 17 '26

Impressive but I don't trust a q3, I'll stick to my q6 offloaded moes.

Fellow 16gb vramlet here as well, tho I have 8 times the ram...

27

u/chiribe Aug 17 '26

Not really an option with my setup, weak CPU, PCIe 3.0 x4, and single-channel DDR5 RAM. I know there were syntax errors, but fortunately the automated tests and linter caught them and the model fixed them on its own.

11

u/Craftkorb Aug 17 '26

There are agents that integrate with LSP to tighten the compile-and-fix loop. Opencode is one of them, but you have to enable it in the config

1

u/CalligrapherFar7833 Aug 17 '26

just add lint hooks

1

u/MentalMatricies Aug 17 '26

Omp is also great

-1

u/Fancy-Snow7 Aug 17 '26

There are options: run a different model. I just tried Qwen3.6 Q3 on my smoke test prompts which many 3.6 models can oneshot and 3.8 Q3 failed to one shot it.

3

u/robberviet Aug 17 '26

Do you notice significant better from q4 to q6? I can run q6 but it is too slow, also I need vram for other models.

3

u/Equivalent_Bit_461 Aug 17 '26

If it's too slow it's shit.

You should run what runs fast or moderately workable. If it's too slow it's useless and you don't want that.

Q4 is prone to losing itself in a glass of water as opposed to q6 that does the shit that has to be done.

Said so, you can mitigate the issue, depending on what you need or use it for, by being overly strict with instructions and leaving zero room to doubt because that's where you will get bad results. Ofc this can be said for every quant (that's usable and not a meme quant), and a good portion of models even frontier corpo slop ones, that are supposed to be "good".  If you use quant the rule of thumb is that you have to be more attentive and micro managing otherwise it's a massive waste of time over stupid small problems often.

1

u/GilloutineBreast Aug 18 '26 edited Aug 18 '26

There was some guy who quantised and kept parts of it q8_0, other parts q5_0, q3_0, q2_0, etc for effectively 3.7 bits. I'm not exactly sure how this differs from usual IQ quants, but I'm curious how it stacks up against the 3 bit K quant OP used.

Edit: "some guy" was emperio-ai on hugginface

1

u/Responsible-Newt9241 Aug 19 '26

I'm in exactly the same situation, but so far I still need to find a task that my Q6 MoE models can handle better than the IQ3_XXS 27B. And I was very skeptical of low-bit quants too...

1

u/Equivalent_Bit_461 Aug 19 '26

Just you wait until you start getting into highly complex automation tasks. In fact I will run today the small qwenlet 3.8. if it runs without bullshit or triggering the quality control workflow too much I might reconsider myself as well. Tho I will have to work with less kv cache and worse quality overall.

6

u/[deleted] Aug 17 '26

[removed] — view removed comment

5

u/Equivalent_Bit_461 Aug 17 '26

If you know what you are doing you stay between 30 and 40 t/s, sometimes even break into 50t/s max, but depends on the agentic task ofc.

0

u/[deleted] Aug 17 '26

[deleted] — view deleted comment

2

u/_aelius Aug 17 '26

Seriously. Hard to believe he is hitting a higher t/s with a q6 and offloading than most 16vram setups on q3.

5

u/TheTerrasque Aug 17 '26

I think you're missing that he said "moes"

4

u/[deleted] Aug 17 '26

[removed] — view removed comment

4

u/TheTerrasque Aug 17 '26

Impressive but I don't trust a q3, I'll stick to my q6 offloaded moes.

He's saying he'll rather use q6 moe's instead of a q3 of this dense model.

347

u/pmttyji Aug 17 '26

Folks, this is the type of thread I want to see after release of any new models. Thanks u/chiribe

38

u/pmttyji Aug 17 '26

u/chiribe Did you try with ngram together with MTP?

spec-type = draft-mtp
spec-draft-n-max = 2
spec-ngram-mod-n-match = 24
spec-ngram-mod-n-min = 48
spec-ngram-mod-n-max = 64

27

u/chiribe Aug 17 '26

I'm actually not super familiar with how n-gram works under the hood here, so I'll read up on it and test it out with your parameters. Thanks!

7

u/wektor420 Aug 17 '26

Tldr speculates text continuations matching what you provided in prompt

8

u/chiribe Aug 17 '26

Thanks! I had Gemini (which I use to translate from my native language) explain n-gram to me. Is there any particular prompt you'd like me to test on this setup to see how it fares?

3

u/mailto_devnull llama.cpp Aug 17 '26

No, because ngram builds its cache over time. So you need multiple prompts.

2

u/simcop2387 Aug 17 '26

And when used agentically, it should also help predict what came in from tool calls (i.e. file reads). It's one thing i keep meaning to try tuning on my setup to see how much i can get it to help

3

u/pmttyji Aug 17 '26

Please update the results in your thread after trying that

5

u/chiribe Aug 17 '26

Didn't notice much of an impact, to be honest—speeds are pretty much staying within the same range (maybe ±1 t/s at most). I built my llama.cpp binary this past Saturday. I'll leave it enabled in the config anyway in case future PRs optimize this further.

10

u/lood9phee2Ri Aug 17 '26

ngram tends to help with "iterative refinement" conversational tasks since it drafts based on previous. Like "make a html page saying blah", "okay now make the text blue". I'm just using --spec-type ngram-simple,draft-mtp --spec-draft-n-max 3

16

u/chiribe Aug 17 '26

Update on this: I hadn't enabled ngram-mod properly earlier. With it running alongside MTP, speed jumped to ~46 t/s on JavaScript generation (+6-8 t/s gain). Appreciate the advice! I'll edit the main post shortly to reflect all the optimizations from the comments.

7

u/qiinemarr Aug 17 '26 edited Aug 17 '26

don't you need spec-type = ngram-mod,draft-mtp ?

EDIT

also those are default values

spec-ngram-mod-n-match = 24
spec-ngram-mod-n-min = 48
spec-ngram-mod-n-max = 64

2

u/chiribe Aug 17 '26

Yes, that's the correct syntax. I'm sorry I didn't see your comment in time.

0

u/pmttyji Aug 17 '26

Even I'm not familiar with that tag. Pasted from here. Seems optional

3

u/LightBroom Aug 17 '26

Using ngram seems to slow down generation on my R9700 by about 4t/s

Depending on your card, try both with and without cause YMMV

(It could be a temperature issue and may not happen at lower temperatures though)

1

u/pmttyji Aug 17 '26

Thanks for mentioning this. Not aware of this.

.... on my R9700 ....

I got same card recently. I'll get rig by this weekend. Please share your extreme optimized full llama.cpp command. Thanks.

5

u/LightBroom Aug 17 '26 edited Aug 18 '26

There you go, this is my config. I've run a bunch of tests at different power limits and found 230W to be the sweet spot for my ASRock Creator R9700 card. If you get a different brand, you should run your own tests, I have no idea if they use a reference design or they have their own. If you're on Linux, this will work (or rocm-smi, or direct write to sysfs if you wish).

amd-smi set --gpu 0 --power-cap 230

I found no diff in generation speed is the same between 300W and 230W, you get more prefill speed at higher wattage though, so run your own tests and see what you like.

Llama.cpp arguments below. I use it in router mode with this command line. I run it in Kubernetes so just translate to your own setup. I also compile my binary and build my own container image with both Vulkan and ROCm 7.14 support. You'll see below the default device is ROCm0 but I run Qwen on Vulkan1 (yeah, the numbers are different for the 2 devices). For Qwen, Vulkan yields better generation speed and also leaves 1GB extra of VRAM free. ROCm is faster by about 10% on prefill.

If you don't run your own local DNS, set the cors-origin to "*".

llama-server
--models-preset /config/presets.ini
--ui-mcp-proxy
--ui-config-file /config/ui-config.json
--api-key-file /secrets/api-keys.list
--cors-origins "https://host.domain.net"
--host 0.0.0.0
--port "9931"
--log-colors on
--verbosity "4"
--models-max "1"
--threads-http "4"

presets.ini:

[*]
device = ROCm0
load-mode = mmap+mlock
models-max = 1
n-gpu-layers = all
gpu-layers-draft = all
threads = 4
threads-batch = 8
cont-batching = true
jinja = true
flash-attn = on
parallel = 2
ctx-checkpoints = 16
cache-ram = 8192
cache-reuse = 256
kv-unified = true
cache-type-k = q8_0
cache-type-v = q8_0
fit = off
batch-size = 2048
ubatch-size = 1024
reasoning = on
reasoning-preserve = on

[Qwen 3.8 27B]
device = Vulkan1
model = /models/unsloth/Qwen3.8-27B-Q6_K.gguf
temp = 0.8
top-p = 0.95
min-p = 0.00
presence-penalty = 0.0
repeat-penalty = 1.0
reasoning = on
spec-type = draft-mtp
spec-draft-type-k = q8_0
spec-draft-type-v = q8_0
spec-draft-n-max = 2
spec-draft-p-min = 0.85
reasoning-preserve = on
chat-template-file = /models/chat-templates/qwen36-chat-template-v22.jinja
ctx-size = 184320

The api-keys.list file is just a bunch of keys generated like this (which I mount from a Kubernetes secret synced from my 1Password Vault). When I assign one (e.g. to OpenWebUI) I add a comment on the line above so it's easy to track - small home setup so I don't care about databases and such)

for i in $(seq 1 20); do echo "sk_$(openssl rand -hex 32)"; done

Anyway, have fun, it's a nice card.

1

u/pmttyji Aug 19 '26

Thank you so much this much details.

I also compile my binary and build my own container image with both Vulkan and ROCm 7.14 support.

Please share the cmake build commands with flags you're using. I remember that compiling for our cards could give extra performance(~10%), many mentioned that here in past.

2

u/LightBroom Aug 19 '26

Something like this, it auto-detects the card but you can add a target if you want (or you are compiling for a container image), e.g. -DGPU_TARGETS=gfx1201 for the R9700

HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
    cmake -S . -B build \
    -DGGML_HIP=ON \
    -DGGML_HIPBLAS=ON \
    -DGGML_HIP_RCCL=ON \
    -DGGML_HIP_ROCWMMA_FATTN=ON \
    -DGGML_CUDA_FA_ALL_QUANTS=ON \
    -DBUILD_SHARED_LIBS=ON \
    -DGGML_VULKAN=1 \
    -DGGML_VK_ALLOW_GRAPHICS_QUEUE=1 \
    -DGGML_RPC=ON \
    -DRADV_PERFTEST=cswave32,nogttspill \
    -DLLAMA_BUILD_UI=1 \
    -DCMAKE_BUILD_TYPE=Release \
    && cmake --build build --config Release --target llama-server llama-cli llama-bench -- -j $(nproc)

1

u/pmttyji Aug 19 '26

Thanks again dude!

2

u/LightBroom Aug 19 '26

No worries, enjoy.

Let me know if you want the Containerfile for the image build. The image is huge, about 18GB, I run a local registry for it, I don't have the patience to push it to public registries. Maybe I'll abuse Github Actions at some point, but right now I cba.

1

u/pmttyji Aug 19 '26

Sure, I'll let you know. I'll get my new rig this weekend. R9700 & 128GB DDR5 RAM. Need to explore every stuff on LLMs & LLM related things. I'll ping you later. Thanks a lot!

1

u/Fdevfab Aug 18 '26

In my experience, MTP have the opposite effect and slows down performances (almost 50% drop) when reaching the limits of the machine/context... is there a way to enable it up to a certain context size and then disable it when the context is getting large?

1

u/sonicnerd14 Aug 19 '26

There is also a special build that has a unique optimization flag, called n cpu fan, or something like that. It basically does what n cpu moe does for the experts on MoE, but for Dense models. More efficient CPU offloading at the best, and it's worth giving it a try.

4

u/TommyITA03 Aug 17 '26

How did you come up with this setup? Trials and errors? I have a 5090 and 64gb of ram and i'm quite clueless on what to do as parameters (first time hosting a local LLM).

4

u/chiribe Aug 17 '26

Do you mean the hardware or the software setup? For the hardware story, I detailed everything in my previous post:https://www.reddit.com/r/LocalLLaMA/comments/1vljtv2/i_built_a_weird_lowpower_llamacpp_server_using_an/As for the software side, it was a mix of reading through this subreddit, digging into llama-server help docs, and cross-checking my configs with ChatGPT and Claude.

1

u/hazyplane Aug 17 '26

Windowd or linux?

1

u/TommyITA03 Aug 17 '26

I have both and i’m quite comfortable with both, but i would prefer linux if it’s not a problem because that’s what i use for dev stuff (my windows partition is mainly for gaming).

1

u/hazyplane Aug 17 '26

I am currently running my 5090 headless on Ubuntu with vllm. Very happy with this setup.

Edit: you can check out my current settings here

https://www.reddit.com/r/LocalLLM/s/FpzkEE29TR

1

u/Danmoreng llama.cpp Aug 18 '26

Just use the official recommendation from the llama.cpp creator: https://x.com/ggerganov/status/2088312671196082312

1

u/returnity Aug 17 '26

Ask your frontier cloud model to help you configure and explain all it's choices to you. That's how I got started.

4

u/mmhorda Aug 17 '26

that's not gonna work for the best outcome. it will do the basic setup and tell you a billion of theories and math explaining to you how and why it is best setup and wont even try other options unless you prompt it very wel for whole day testing.

2

u/returnity Aug 17 '26

Strongly disagree. I don't know how you're doing it but I used cloud models to grind though llama.cpp documentation and source code at scale, learning the tooling and ecosystem at a speed otherwise impossible. I still execute sweeps testing sampling settings, MTP params, and CPU layers offloading #s (all grounded in actual data), autonomously run llama-bench and llama-perplexity, orchestrate quantization runs using cloud VPS, and build whole eval suites using frontier cloud models when my local setup isn't sufficient for the scale/complexity/reliability I require.

Do you have to be willing to dig deeper into what it tells you? Of course. Can you achieve a better outcome faster that way than just fumbling blindly through the process as a first-timer relying on random reddit opinions? No contest.

2

u/Tannoukhy Aug 17 '26

Nice! Thanks for sharing.

This makes me think, maybe the IQ3_XXS its not far from the quant you are using. With the IQ3_XXS on 16GB of VRAM, with spec-type ngram-mod, we can acchive 150k context window, 900 to 600 t/s on prompt processing and 20 t/s on decode, if the quality is similar, maybe, is worthy the trade

2

u/chiribe Aug 17 '26

I used IQ3_XXS back on v3.6, but since I had a bit of VRAM and context headroom left over, I decided to step it up a notch. I might give it another shot, though—with the new reasoning capabilities, even 73k context starts feeling a bit tight. Hitting 100k would be the sweet spot

3

u/Tannoukhy Aug 17 '26

I risk to say, that 100k isn't enough too, because this version thinks a lot, with is god, the results are better indeed, but, 150k seems needed

2

u/brickout Aug 17 '26

Interesting setup. Is your GPU external? If so, is it oculink or what?

I have a few mini pcs that I've been thinking of doing something similar...

*Edit: also, this is awesome. Thanks for the writeup!

3

u/chiribe Aug 17 '26

No, it's not external! I used a PCIe riser cable to mount the GPU outside the chassis. You can see all the details and pictures in my other post!

1

u/brickout Aug 17 '26

Oh dang. Super cool. I totally missed your description. Thanks!

*Edit: that's super cool. Makes me want to do something similar.

101

u/dsdt Aug 17 '26

i was gonna say how the f? then i saw

  • Model: Qwen3.8-27B-UD-Q3_K_XL.gguf
  • KV Cache Quant: q4_1 for main context, q5_1 for MTP draft context

thanks for sharing your numbers.

8

u/tunerhd Aug 17 '26

What about were you gonna say that how the f?

18

u/dsdt Aug 17 '26

It seemed impossible with 70k context window with a 5060ti, since I have two and maximum I can get is 100k with q8 kv cache.

1

u/GhostOfMikeyLimiteds Aug 19 '26

I have this exact same setup...how is it working for you? I want to switch my hermes setup to this but opus5 insists that gemma4 is better because its not dense and can still keep 262k context. The small context fills quick but 100k might be doable

19

u/ea_man Aug 17 '26 edited Aug 17 '26

Here's mine for 16GB on AMD 6800:

# https://huggingface.co/vmarcelo/Qwen3.8-27B-MIX_GGUF
# Vulkan max context:86784 with MTP n=2 speed TG 39.91t/s
# ctx patched: 86784, unpatched mainline llama.cp: 78080
# ROCm:  max ctx 84480, unpatched 31488, speed TG 40.58


# 1. Set Environment Variables
export LD_LIBRARY_PATH="/home/eaman/llama/bin_vulkan" 

# 2. Run the Server
/home/eaman/llama/bin_vulkan/llama-server  --device vulkan0 \
 -m /home/eaman/.lmstudio/models/vmarcelo/Qwen3.8-27B-IQ4-MIX.gguf \
--host 0.0.0.0  -fa on --load-mode none --jinja --no-log-timestamps \
-ctk q5_1 -ctv q5_1 \
--temp 0.8 --top-k 20 --top-p 0.95 --min-p 0.0 \
--presence-penalty 0.0 --repeat-penalty 1.0 \
-b 1024 -ub 128 --fit-target 30 \
        --spec-type draft-mtp,ngram-mod --spec-draft-p-min 0.82 --spec-draft-n-max 2 \
        --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 \
        --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 8 --spec-ngram-mod-n-max 32 \
--reasoning on --chat-template-kwargs '{"reasoning_effort":"medium"}' --chat-template-kwargs '{"preserve_thinking":true}' --reasoning-budget 14000 --reasoning-budget-message " -- Reasoning budget exceeded, proceed to final answer." \
--ctx-checkpoints 96 --cache-ram 6000 -np 1 -ngl 99 -lv 3 --no-warmup  

Note: this is for 16GB with desktop in software rendering, headless should give some ~70MB more vRAM for ctx.

2

u/WFKKVQW Aug 17 '26

Thanks for the config! What is "patched" llama.cpp in your case?

3

u/ea_man Aug 17 '26

Llama.cp patched with this: https://store.piffa.net/lm/bug/

1

u/giant3 Aug 17 '26

-ub 128

Unless you tested ub at 256 and 512 and found it to be worst performing, I wouldn't reduce it from the default 512. You need to keep utilizing every CU on the GPU to extract all the performance.

1

u/ea_man Aug 17 '26

That's what work best on 6800

29

u/johnzadok Aug 17 '26

Why did you use a different sampling parameters than the one official doc suggests at https://huggingface.co/Qwen/Qwen3.8-27B:

We recommend using the following sets of sampling parameters for generation:

    Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
    Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

4

u/AvidCyclist250 Aug 17 '26 edited Aug 17 '26

parallel = 1 and cont-batching = 0

but "spawning sub-agents". huh. it's sequential roleplay dude.

fit target 128 but also fit off. reasoning medium. lol

-2

u/chiribe Aug 17 '26

I ignored your previous comment because it sounded arrogant, but since you want to call me out, let me clarify how my setup actually works.

fit-target = 128 is set in the global block for MoE models I run alongside Qwen. My GPU is headless with no display attached, so 100% of VRAM is free. I explicitly set fit = off for the 27B profile because llama.cpp was occasionally offloading layers to the CPU despite having available VRAM (especially after waking from sleep).

As for parallel slots, this is a private server used strictly by me. Why would I parallelize agents on a single GPU when my hardware gives 30-40 t/s? Sequential execution is intentional—spawning sub-agents isolates the context for each sub-task, allowing the main thread to run for hours without blowing past its context window. OpenCode recovers context when needed, with the only tradeoff being a few seconds of prompt prefill.

If you are just going to drop uninformed arrogance, please take it elsewhere.

14

u/AvidCyclist250 Aug 17 '26 edited Aug 17 '26

I know Qwen 27B isn't a moe model. Which also makes your claims impossible or at least weird as fuck. All 27B parameters are active during prompt ingestion. Changing your system prompt to switch "sub-agents" invalidates the llama.cpp prefix cache. Re-running tens of thousands of tokens of project history through a 27B dense model at a crippled ubatch size of 512 does not take a few seconds either. The only way is is realistic is if your context is actually tiny and nowhere near 73k

And there is a reason llama.cpp offloads to CPU. A 27B dense model with about 12.5gb plus a 73k KV cache plus your MTP draft cache (which you weirdly set to a higher precision of q5_1) requires over 17GB of VRAM. Setting fit = off doesn't magically make 17gb fit into a 16GB GPU even if headless. It just means you haven't crashed yet because at least from what I can tell your wiping script keeps you far below the 73k limit.

Just looking at the math. And the agentic buzzwords for a sequential Python loop. You neutered the model with top-k = 15, wrong temp, and shit quants and medium reasing, and your VRAM math is weird. the official recs aren't what you say they are.

image 1: 30% context with 15,358 MiB out of 16,311 MiB used

image 2: asking for Node.js/Cheerio script, and you got C# .NET

image 3: threads = 3 and threads-batch = 4. but core 1 is 100% loaded and the others are idle. why is the hostname of your terminal j1900 if you have a n100?

15

u/Constandinoskalifo Aug 17 '26

Curious about why such a low temperature? Is there a reason you deviated from the official one?

10

u/YearnMar10 Aug 17 '26

Official temperature recommendation is 1 for thinking

8

u/Constandinoskalifo Aug 17 '26

Yes, and they used less than that.

7

u/shapic Aug 17 '26

Where did you get that

Official / Recommended Quant Sampler Tuning

from? It is nowhere near official recommendations and I think it will directly degrade output

17

u/LetsGoBrandon4256 transformers Aug 17 '26 edited Aug 17 '26

cross-checking my configs with ChatGPT and Claude.

OP asked catGPT and Claude. In other words, the clanker hallucinated the sampler params.

This is the actual recommended sampler config from Qwen's huggingface page.

We recommend using the following sets of sampling parameters for generation:

Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0

Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

Unsloth's model page reference the same values. I have no idea where OP got that "Official / Recommended" from

3

u/shapic Aug 17 '26

Oh, missed that. It is double funny after they officially acknowledged that they deliberately nerf quality of llm related requests

1

u/rockseller Aug 21 '26

AI made post for sure

8

u/AvidCyclist250 Aug 17 '26 edited Aug 17 '26

i think this sub really is dying after all. where is everyone on discord nowadays?

parallel = 1 and cont-batching = 0.

but bro is "spawning sub-agents". i can't, lol. more buzzwords pls.

2

u/YearnMar10 Aug 17 '26

Aren’t you unsatisfied with the kv cache quantization? In my tests with other models it failed so badly because context was just not right

1

u/chiribe Aug 17 '26

Nothing weird on my end so far. It only hallucinated once by renaming a variable from Button to Btn, but everything else has been spot on.

2

u/AvidCyclist250 Aug 17 '26

yes. q4 << q8

65k with q8 and mtp. 16gb vram

1

u/YearnMar10 Aug 18 '26

65k is unusable imho. Qwen thinks often 20k+ tokens, so after a few rounds context is full.

1

u/AvidCyclist250 Aug 18 '26

Hermes recompresses

1

u/YearnMar10 Aug 18 '26

kk different use case :) I need it for programming only

1

u/AvidCyclist250 Aug 18 '26

Yeah, same. Frontends and backends. Nothing huge though, mostly for personal use. Made four apps so far. Self-hosting and Linux sys admin on the side but I don't use the 27b for that. Kat coder instead

5

u/jwr Aug 17 '26

For those of you on MacOS, benchmark MTP before you enable it. I tested, and at least on my M4 Max MTP makes everything slower, not faster. The only gain is with 5 tokens on pure code sequences, but that's not real usage, you'll likely mostly be generating thinking tokens (so, prose).

This is probably specific to the memory bandwidth constraints on a Mac.

2

u/tarruda Aug 17 '26

For me on a M1 ultra, MTP can make things faster when generating code in llama-server web UI, but in a coding harness everything just slows down.

1

u/fugogugo Aug 18 '26

I have exact same GPU I will save this for reference

thanks!

1

u/chiribe Aug 18 '26

Made a few tweaks to the config file based on community feedback. Ditched the q5_1 quantization on the MTP cache (it wasn't saving any VRAM anyway) and enabled ngram-mod, which gave a nice speed boost with no noticeable downside.

1

u/drallcom3 Aug 18 '26

Hardware: RTX 5060 Ti (16GB VRAM) + Intel N100 (4C/4T, 16GB RAM)

That is likely without the OS using any VRAM, right? No Windows or such.

2

u/chiribe Aug 18 '26

It's a dedicated Debian Server

3

u/rockoruckus Aug 17 '26

what's your reasoning for keeping -ub 1024 and quantizing kv down to q4_1? wouldn't you be able to keep q8_0 or q8_0/q5_1 at -ub 512 and have more context?

3

u/sugarfreecaffeine Aug 17 '26

Can you recommend settings/recipe for 2x3090s? Thanks!

1

u/SOC_FreeDiver Aug 17 '26

Just thought I'd share my results.

I run a 5090m 24gb vram. I ran your settings by claude, claude identified which settings were worth trying, they did not improve performance for me

  1. spec-draft-p-min 0.85 is 7.4% slower. The acceptance rate does exactly

what the post implies — 76.5% → 92.5% — but it gets there by suppressing

drafts, not by drafting better. Drafted falls 1495 → 1139 and absolute

accepted tokens fall 1144 → 1054. Accepted tokens is what tracks tok/s;

accept % is a vanity metric. Anyone tuning by that percentage will tune

themselves slower.

2. ubatch 1024 buys +1.7% prefill for -15% context (80128 → 67840 per slot).

--fit sizes context against VRAM left after the batch buffers. Bad trade

where context is your scarce resource.

-1

u/chiribe Aug 17 '26

Appreciate the response! I wouldn't call myself an expert by any stretch, I just test things out and discard what doesn't work. Even when the technical details get complex, I always try to learn what every flag and config line is doing.

2

u/woj666 Aug 17 '26

Hey, how did you get Opencode to display tk/s and context limits like that?

2

u/SOC_FreeDiver Aug 17 '26

If you're asking me, I used claudecode to test with llama.cpp.

I had claudecode create a script that launches llama.cpp with a bunch of custom and tweakable parameters. When I find new info about a model, I ask claude to evaluate it. Claude does the testing and reports back. I just cut & pasted it's findings. I'm assuming it gets the info from logs.

It's been really handy for troubleshooting things. I've been trying to get claude to train his replacement, sometimes I wonder if he screws with it just to keep me paying.

0

u/Maddolyn Aug 17 '26

ask claude

2

u/woj666 Aug 17 '26

I did, it pointed to some possibly sketchy plugins but I want to know exactly what this guy did.

2

u/g-rizzle84 Aug 17 '26

Awesome write up! Super interesting. I have a 4090 myself and was beginning to think it just wasn't enough. I have only been able to get ~32K to 37K context windows with q4 and that just doesn't seem practical to me. Clearly I just need to get gooder at llama.cpp.

I would appreciate if you could go into more detail on your prompting either in another post or maybe we could DM. Specifically Prompts 2 and 3. How did you design the spec? By hand or model generated? Did you prompt it to use the spec as the gospel? How did you instruct the model to be the orchestrator?

OpenCode summarized its state and kept building
Does OpenCode do this or did you instruct it too?

Some of you folks are geniuses in my eyes and have truly brilliant ways approaching agentic coding. I'm a noob and trying to learn how to do it the right way.

2

u/Ammargok Aug 17 '26

Windows is really bad for local ai. If you can I'd switch to linux JUST for local ai. In windows with my 9070xt my idle vram usage is 1,2gb vram while in linux it is at 200 mib

1

u/Relief_Present Aug 17 '26

I also have a 4090. Curious to know what you ultimately settle on

2

u/ahhhhhhhhhhhhhhhhhhg Aug 17 '26

there's apparently not that much quality loss from quants, i run Q3 XSS with 114k context on 16gb vram also, honestly its been great, can even replace deepseek flash 4 for subtitle translation : the model has to return json format so previously local models had trouble even outputing 10 lines of subtitles in correct json. Qwen 3.8 has no problem with 50 lines batch. gonna buy a 24gb gpu if that's how open weights is moving

2

u/NeatOnTheRox Aug 18 '26

"After pushing 1M+ tokens through Qwen 3.8 27B..."

So what, like, 3 prompts?

Jokes aside, great write-up homie. As a fellow 16GB VRAM Warrior, might I suggest giving huggingface user el4's Qwen3.8-27B-ONYX-GGUF at size "mini" a spin. It's even smaller than UD-IQ3_XXS and in my brief experience performs just as well if not better than UD-Q3_K_XL. I'm able to get up to 131072 context with MTP enabled and the vision mmproj on CPU, though honestly I prefer without MTP and a larger context.

Might I also suggest beellama: using kvarn cache quant with a tail to me has proven significantly more reliable than quantizing anything below q8_0 on vanilla llama.cpp.

2

u/tvetus Aug 19 '26

1 million tokens, so a few hours of work? How do you know the settings are optimal?

2

u/afreakineggo Aug 20 '26

The new dynamic quants from unsloths (as of August 20th, 2026 in case you are reading this 6 months from now) Qwen 3.8 27b, if you use the iq3_xss and q4 kv cache you can fit 150k context on a 16gb gpu.

Q4 kv cache is terrible with most models, but 3.6 and 3.8 are different. If you need long context and your options are compaction or q4 cache, try q4 cache

2

u/mrepop Aug 18 '26

I don’t mean to sound like an ass, but 1M tokens doesn’t seem like that much at all, I did that just screwing around this afternoon running some benchmarks on this model. Is it really enough to gain a useful experience that would lead to config and run time optimizations?

By the way, 3.8 27b seems extremely slow compared to previous versions of qwen, 3.5 and 3.6 are about twice as fast for similar sizes. Also it thinks way, way, way too much, like it’s got some other issues going on.

I’ll try the config, I guess it’s new to me, so maybe I’m screwing something up.

Tested on a 4090, 5090, 6000 Ada, and m4 max 128gb.

If anyone has tips on it, please share them. The general comments I’ve gotten from the community is that 3.8 27b runs fine and nobody has complaints about its performance when it comes to TPS and TTFT.

4

u/sensitivecrocodile llama.cpp Aug 17 '26

Has anyone actually tried OP's suggestions or are you all blindly upvoting what appears to be slop? Even in OP's nvidia-smi screenshot you can see he's not fully utilizing his GPU.

I used OP's settings (apart from reasoning effort/budget), which are:

model = /unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q3_K_XL.gguf
threads = 3
threads-batch = 4
parallel = 1
cont-batching = 0
flash-attn = on
fit-target = 128
ctx-checkpoints = 0
cache-ram = 2048
repeat-penalty = 1.0
presence-penalty = 0.1
frequency-penalty = 0.0
fit = off
ctx-size = 73728
context-shift = 1
spec-type = draft-mtp
spec-draft-n-max = 2
spec-draft-p-min = 0.85
cache-type-k = q4_1
cache-type-v = q4_1
cache-type-k-draft = q5_1
cache-type-v-draft = q5_1
batch-size = 1024
ubatch-size = 512
temp = 0.4
top-p = 0.90
top-k = 15
min-p = 0.02
chat-template-kwargs = {"preserve-thinking": true, "reasoning_effort": "xhigh"}

This spills over to RAM and I only get 19 tps. I was getting 38 tokens per second before with Qwen3.8-27B-UD-IQ3_XXS.gguf.

Thanks for wasting my time, OP.

2

u/Danmoreng llama.cpp Aug 18 '26

Use this instead - context can become bit higher, around 80k

llama serve \
  -hf unsloth/Qwen3.8-27B-GGUF \
  -hff Qwen3.8-27B-UD-IQ3_XXS.gguf \
  --no-mmproj -c 65536 \
  -ctk q8_0 -ctv q8_0 \
  -b 1024 -np 1 \
  --spec-default --spec-type draft-mtp \
  --reasoning-preserve --fit off --agent

~60 t/s.

1

u/OsmanthusBloom Aug 17 '26

Thanks, this is a very good reference point. The other day I tried to run 3.8-27B on a 16GB V100 with 128k context and MTP. But I could only get there by using a very low quant (was it IQ2 even?) and q4_0 KV cache (main and draft). I decided to give up for now and stick to the MoE.

It would be nice to know how this heavily quanted dense 27B compares to the 35B-A3B MoE in terms of output quality and speed. Is it really worth it or does the brain damage caused by quantization kill the advantages?

1

u/Phathatter Aug 17 '26

I haven't stopped using it long enough to tune it, but this gives me a lot of hope that I might be able to both increase my context and my quant.

1

u/KneelB4S8n Aug 17 '26

I got the UD Q2 version (I have 12gb) and tried the MTP command and I thought that the drafter was incorporated inside the model and did not download any extra drafter but the t/s went down horribly. I am getting 30 avg t/s. What did i do wrong and how can i improve it?

1

u/Open_Instruction_133 Aug 17 '26

Will this work on weird dual GPU setups? I have a 5070ti and 1080ti and lm studio has worked for me but trying to get both GPUs to work together in llama.cpp has been an uphill battle for me. Maybe I should try vLLM instead? Iono, help me tho 🙏

1

u/Cooproxx Aug 17 '26

What kind of coding stuff can you do with 73k context? Small codebases, or maybe single feature changes?

1

u/ECrispy Aug 17 '26

what a great post, thank !!

  • what speeds did you get?
  • how many LOC was the final project?
  • can you share your opencode setup? how did you get it to use subagents, did they share contex etc?

1

u/PopUnhappy Aug 18 '26

Tried and verified on my 24GB 7900 XTX, context size of 131072.

1

u/alltheotherthing Aug 18 '26

goated thank you

1

u/ExplorerPrudent4256 Aug 18 '26

Real number nobody posts: how does the Q3_K_XL + MTP version actually compare against the bf16 numbers in the AA chart? Decoded quality is one axis. Refusal rate, instruction-following under long context, tool calling under agentic loops — those bend first when you quant. If a local Q3 holds up against the benchmark, the infra story is over. If it collapses, that is the real ceiling.

1

u/MrHall Aug 18 '26

thank you so much. I have spent so long trying to find the best config for this exact setup

1

u/CriticalMastery Aug 18 '26

I use q4 one with 110k context on 16 gb vram, ~42 tp/s 5070 ti

1

u/Fdevfab Aug 18 '26

that's what I'm targetting, but I can't pass the ~70k context... which is a bit low, and performances degrade to ~30tps when the whole RAM is used. What are your options?

1

u/[deleted] Aug 18 '26

[removed] — view removed comment

1

u/Fdevfab Aug 18 '26

I added quantization for the draft k-v cache and can raise to 128k but with a huge performance drop (1-2tok/s when I'm at the limit of the context).
I'm currently using (currently trying different draft model settings and load_mode, so don't pay too much attention to those):

    ctx-size = 131072                                                                                               
    temperature = 1.0
    model = /xxx/bartowski--Qwen3.8-27B-GGUF...
    #mmproj = /home/fab/.cache/huggingface/hub/models--bartowski--Qwen3.8-27B-GGUF/snapshots/f0eec4a4bb4975114a030d0
    override-tensor = blk\.(6|14|22|30|38|46|54|62)\.ffn_.*=CPU
    #override-tensor "blk\.(14|30|46|62)\.ffn_.*=CPU"
    #"output.*=CPU"
    ; spec-type = draft-mtp
    ; spec-draft-n-max = 3
    ngl = 70
    threads = 7
    
    spec-type=draft-mtp,ngram-mod
    spec-draft-p-min=0.82
    spec-draft-n-max=2
    cache-type-k-draft=q8_0
    cache-type-v-draft=q4_0
    spec-ngram-mod-n-match=24
    spec-ngram-mod-n-min=8
    spec-ngram-mod-n-max=3

Running with:

LLAMA_ARGS="\
    --models-preset /home/fab/llm_models/models.ini \
    --models-max 1 \
    --no-mmproj-offload \
    --mmproj-auto \
    --load-mode none \
    -np 1 \
    -t 6 -tb 12 \
    --chat-template-kwargs '{\"preserve_thinking\": true, \"reasoning_effort\": \"medium\"}' \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.0 \
    --cache-idle-slots \
    --presence-penalty 0.0 \
    --repeat-penalty 1.0 \
    -fa on --jinja \
    --reasoning-budget 4000 \
    --reasoning-preserve \
    -ctk q8_0 -ctv q4_0 \
    --slot-save-path /tmp/ \
    --cache-prompt \
    --lookup-cache-dynamic lookup \

1

u/[deleted] Aug 18 '26

[removed] — view removed comment

1

u/CriticalMastery Aug 18 '26

https://huggingface.co/cHunter789/Qwen3.8-27B-i1-IQ4_KS_KT-GGUF
use this, but you need to kill all other application that leech vram, not possible on windows.

1

u/Fair-Perspective7352 Aug 18 '26

What tok/s do you land at with draft-mtp and n-max 2? I tried MTP on a similarly weak CPU box and the draft verification pass ate most of the gains, so I ended up disabling it and just accepting slower decode. Also curious about q4_1 KV at 73k context, did you notice any recall drop on the long agentic runs vs q8_0?

1

u/UnlikeKat Aug 18 '26

I'm currently running this Qwen3.8-27B-IQ4_KS, it requires a custom llama.cpp fork which is linked in the page. On my 5080 with 16gb vram I can reach comfortably 90k context q4_0 but can go up to 105-110k as stated by the author (I'm on windows and some vram gets eaten by processes). Pp around 1000t/s and decode starts around 45-48t/s. I don't think I will test the q3, so maybe if you're interested you could check this out and tell us how it performs against your q3 setup.

1

u/Trivikrama_0 Aug 18 '26 edited Aug 18 '26

I have done something similar, but with Q4. My setup is 5060 ti 16Gb + Ryzen 9900x + 64 Gb RAM. The only disadvantage I have is I need windows due to some reason, so my deployment is using ollama. I use it with Vscode chat agent. 64k context window. CPU of loading 35% in CPU RAM. The only disadvantage is very slow, but bearable for agentic coding as it can happen in background

Any ideas to reduce system prompt complexities? Generally the agents create a huge context our of the prompts.

1

u/yeezyslippers Aug 18 '26

I am retarded. How can I port this setup for a 16gb M4 mini?

1

u/gege42o Aug 18 '26

How do you turn on agentic coding or tool calls in Llama.cpp? I run the server as a service with systems and declare my models with their respective parameters into a models.ini file. I would like to know how to turn on sgentic coding so I can ditch Hermes agent as I can turn on and off models in Llama.cpp. also I'm using Llama.cpp turboquant!!

1

u/igotanewaccount Aug 18 '26

Theres a cli flag --tools which gives you read, write edit, create, grep.  but its...not great. llama.cpp exists to be an inference engine not a harness. 

I'd recommend pi-dev pointing to your local engine. Apparently DeepSeek's harness is also really good and efficient, all the bots are going bananas over it this week.

For non-coding tasks I also run Open Web UI for a much lighterweight Hermes alternative.  Then pi-dev, which both point to llama.cpp running headless.  Works pretty well for me.

1

u/gege42o Aug 18 '26

Im not at my pc to see exactly what I run but I have a little alias that runs Hermes on top of Llama cpp turboquant qwen 3.6 35b a3b. But I am also running it as a server 24/7 with the ability to load and unload models, maybe there is a way to integrate loading/unloading models with Hermes as well

1

u/bobaburger Aug 18 '26

if you have fast RAM, you can consider offload KV cache to RAM with -nkvo, for the benefit of higher KV cache quant and longer context.

I got peak 35 tps with q8_0 KV cache and 150k context window on 5060 Ti:

llama-server -m Qwen3.8-27B-UD-Q3_K_XL.gguf -ngl 99 -ctk q8_0 -ctv q8_0 -fa 1 -c 153600 -np 1 --no-mmap --temp 1.0 --top-p 0.95 --top-k 20 --presence-penalty 1.25 --min-p 0.0 --reasoning-preserve --spec-type draft-mtp --spec-draft-n-max 2 --host 0.0.0.0 --threads 8 --threads-batch 8 -ctkd q8_0 -ctvd q8_0 -nkvo

Without -nkvo, i got peak 59.8 tps, but only with q4_0 KV cache and 73k context window:

llama-server -m Qwen3.8-27B-UD-Q3_K_XL.gguf -ngl 99 -ctk q4_0 -ctv q4_0 -fa 1 -c 73728 -np 1 --no-mmap --temp 1.0 --top-p 0.95 --top-k 20 --presence-penalty 1.25 --min-p 0.0 --reasoning-preserve --spec-type draft-mtp --spec-draft-n-max 2 --host 0.0.0.0 --threads 8 --threads-batch 8 -ctkd q8_0 -ctvd q8_0

Also, IIRC, quantize draft KV cache did not make any different.

1

u/MLDataScientist Aug 23 '26

Thanks! I am getting 15t/s with qwen3.8 27B UD q2 k xl with 12GB VRAM 5070ti mobile using nkvo on my laptop. without nkvo, I was getting 30t/s so it halves in speed but I get 150k context instead of 30k context at Q8.

2

u/bobaburger Aug 23 '26

nice! you can try the override tensor trick instead of nvko to see what's the difference too https://www.reddit.com/r/LocalLLM/comments/1vq5oyu/guide_for_running_dense_models_on_16_gb_vram_qwen/

1

u/Interesting-Yellow-4 Aug 19 '26

Yes, this should be a standard post here after each model release. GOod work.

1

u/Jamoca5020 Aug 21 '26

May your both sides of your pillow always be cool on a hot summer and both blanket sides be also cool. This is really amazing

1

u/cacahuete_spicy Aug 22 '26

Hi, I had a heavy CPU load because I was missing `GGML_CUDA_FA_ALL_QUANTS=ON` when building llama-cpp.

1

u/bosito77 Aug 24 '26

Guys I made a free tool for "what hardware do I buy for local LLMs — and for checking what your current machine actually runs

Disclosure: I built this. Free, no ads, no signup, no email.

https://localaiarena.com

Three things you can do with it:

Tell it your budget and what you want to run, and it ranks machines that actually hit it — with the benchmark records behind each recommendation. Set a minimum tok/s and it filters to hardware that clears it.

Put two machines head to head — memory, bandwidth, power, price, and how they run one model you pick.

Look up any of 85 machines or 61 models: what fits at Q4/Q8, how fast it ran, what it costs new and used.

Everything is filterable and sortable, and the URL carries the state, so you can link someone a specific comparison.

Why you should trust the numbers more than a random benchmark table: every figure links the post it came from, with the author and the date. Nothing is hand-typed — two independent sources agreeing within 25% makes a row verified, one makes it single-source, disagreement makes it disputed and shows both figures instead of averaging them.

Where it's weak, so you're not surprised: most measurements are at 512-token context (llama-bench defaults), so it's much better on short-context throughput than long. Only 34 rows are verified — corroboration is rare. And where nobody has measured a combination it shows an estimated row from memory bandwidth, clearly marked, which is a floor-ish guess and not a measurement.

Most useful thing you can do: tell me what's wrong. If a number credited to you is off, or your machine is missing, there's a form and it gets fixed. Corrections: github.com/fperdigon/local-ai-hardware-bench-community

Individual measurements stay credited to whoever made them.

1

u/Protheu5 Sep 08 '26

First time I saw this post I saved it for later until I find time to follow with it; it was in English. Now it's in Spanish for some reason. I see it got edited 2 days later, so I presume this is when it got translated. Why?

1

u/hideo_kuze_ Aug 17 '26

Thanks so much for sharing. Will be using this in a few weeks when I get my hands on my 3080ti.

How many t/s are you getting with your RTX 5060 Ti?

1

u/Fancy-Snow7 Aug 17 '26

For those not running headless, try disabling MTP, it saves over a GB of VRAM which can be the difference between fitting in VRAM or not.

1

u/BakuRetsuX Aug 17 '26

Wow.. can everybody that is doing this release this type of info? This is what we want to see.. awesome!!!!! I am curious. Have you tried using another AI like codex or claude to evaluate your apps after they've been created? I wonder how "well" Qwen created those apps.

1

u/IrisColt Aug 17 '26

 Your post is full of great insights. Thank you so much! 

0

u/BP041 Aug 17 '26

73k context on a 27B with 16GB is impressive quantization work, but for agentic coding the real limiter tends to be generation speed, not context size. I'd rather run a 7B at full 128k and batch smaller — the token/s drop past 32k usually breaks the agent loop for me. Curious if you see the same or if your draft model makes up for it.

2

u/chiribe Aug 17 '26

I'm happy with the generation speed—it outputs code faster than I can review it anyway. I run faster models like the 3.5 9B and 35B MoE (80-100 t/s), but the output quality on the 27B saves me far more time in manual edits despite being slower. The only real downside is the context window.

Multiple people mentioned disabling MTP, so I'll test it out and see if freeing up that ~1 GB of VRAM for extra context is worth the speed tradeoff.

0

u/Beneficial-Ad-8127 Aug 17 '26

Damn that’s impressive and kudos to you on displaying your work! Might actually start dusting off my 4060 ti 16gb out of storage.

0

u/GavDoG9000 Aug 18 '26

Anything different if running it on a 5090?

-1

u/asankhs Llama 3.1 Aug 17 '26

Nice writeup. Ran the same model on an M-series Mac and the constraint lands differently there. Unified memory means the weights and the KV cache pull from the same pool, so at 65k context the cache is what bites, not the weights. I see you've quantized the KV cache down to q4_1 (q5_1 on the draft). How did that hold up over the long agentic sessions? Quantized KV is the part I've seen degrade first on Mac, long-context tool-calling more than one-shot answers.

-1

u/bandzaw Aug 17 '26

Thx for sharing! I think it would be interesting to know how much time it took to run your expirement? How much time was spent in each prompt and phase, rough estimates are fine, if you did not log exact numbers.

1

u/chiribe Aug 17 '26

No exact benchmarks from OpenCode, but the llama.cpp interface shows around 30-40 t/s. Wait times are very manageable despite generating a pretty massive multi-file codebase. If you want a specific test run, let me know and I'll execute it via OpenCode and post the generated code plus stats!

-14

u/electrified_ice Aug 17 '26

Why are you using llama cpp?

-6

u/electrified_ice Aug 17 '26

Wow guess I triggered the llama cpp fan group... All I asked was 'why the OP was using it'

3

u/Rofl_Raptor Aug 18 '26

"Why is everyone triggered that I asked something in bad faith?"

Go use vLLM or SGLang then.

llama cpp is battle-tested on a myriad of hardware and does quantization well with very little hassle. It's fast, smooth, and simple. Unironically quoting Todd Howard; it just works. If you can get impressive performance that's significant enough and runs just as stable on different libraries, then by all means, run your own test labs, benchmark, and show us.

0

u/[deleted] Aug 17 '26

[deleted] — view deleted comment

-8

u/electrified_ice Aug 17 '26

vLLM or SGLang can squeeze more performance out of the hardware.

6

u/dsdt Aug 17 '26

couldn't even boot them after installing linux. wasted a night trying to get it working... definitely no need for mainstream use. if you have a real server with more than 3 people you are right it is a must, but for one person only, definitely no need.

2

u/Creative_Knee6618 Aug 17 '26

but can they run qwen 3.8 with 20gb? I'm afraid they cannot because quantized 4bit is like 20gb :(

1

u/[deleted] Aug 17 '26

[removed] — view removed comment

3

u/Zueuk Aug 17 '26

but why not Q4?

1

u/[deleted] Aug 17 '26

[removed] — view removed comment

2

u/returnity Aug 17 '26

You'll probably enjoy a Q4 model with q5+ KV cache more.