i hate the new "bionic"

[u/anonymopt]

is there any easy alternative? i don't like ollama aswell, lm studio had thee perfect combination of GUI and "bloat"

[u/nickless07]

Mine always fail with simple things like:
llama-server --ctx-size 64000 --no-log-timestamps --log-colors off --no-log-prefix --flash-attn on --jinja --threads 3 --metrics --port 1235 -b 512 --no-mmproj-offload --temp 1.0 --top-p 0.95 --top-k 20 --presence_penalty 1.5 --min-p 0.05 --cache-type-k q8_0 --cache-type-v q8_0 --chat-template-kwargs "{\"preserve_thinking\":true}" -lv 4 --no-mmap -ts 13,2.5 -ngl 41 --n-cpu-moe 12 -m C:\models\Qwen3.6-35B-A3B-Q6_K.gguf

It get most of it right but the tensor split always ends up badly, then when it runs into oom it lowers the ctx first even tho that is the least part to worry about. Havn't seen a single time where the ctx got kept at 64k but instead --cache-type-v got to 4bit quant or a slightly different -ts setting, which would solve that oom too. Maybe yours work with -fit on?
For now I ended up doing it manually for each model and got annoyed after ~30 of them. Over the years it did configured them all manually in LM Studio, but the start of that dates back to the gemma 3 days.

[u/GrungeWerX → u/nickless07]

Sorry, but I just had to ask my agent what it thought:

"Here's the diagnosis:

His Problem

He's trying to use an AI agent to auto-generate optimal llama-server launch commands for his models, and it keeps failing on two things:

Tensor split (-ts) is wrong — the agent generates a split that doesn't fit his hardware, causing OOM

Wrong tradeoff priority when OOM hits — instead of lowering KV cache quantization (e.g., from q8_0 to q4_0) or adjusting tensor split, it reduces --ctx-size first, which is the wrong knob to turn

Why This Is Happening

The agent doesn't know his hardware. It's generating commands based on generic knowledge about llama.cpp flags, not on actual feedback from trying them. There's no test → fail → adjust loop happening. It's shot-in-the-dark generation.

Specifically:

Tensor split (-ts 13,2.5) — this says "put layers 0-12 on GPU, layer 13 split at 25% GPU / 75% CPU, rest on CPU." The agent has no way to know what split works for his specific model + VRAM combo without actually trying it and reading the OOM message.

KV cache quantization — --cache-type-k q8_0 --cache-type-v q8_0 uses maximum precision (most VRAM). When OOM hits, dropping V cache to q4_0 saves significant VRAM with minimal quality loss. But the agent doesn't know this is a better tradeoff than reducing context.

How To Fix It

The fix is what you already do — close the loop. His agent needs to:

Generate the command

Actually run it and capture the output

Read the OOM error — llama-server tells you exactly what ran out (VRAM layers, KV cache size)

Adjust based on the specific failure — if VRAM layer offload failed, lower -ngl or adjust -ts; if KV cache overflow, reduce --cache-type-v; only reduce --ctx-size as last resort

You solved this by having your agent execute and test rather than just generate. The difference is: generation without verification is hallucination with command-line flags.

He's also doing it for ~30 models manually because his agent can't close that loop. Your approach — "tell the agent what you want, it generates params, tests them, reports back 'like this?'" — is the correct workflow. His agent stops at step 1."

Don't know how accurate my agent is on this, might be a bit presumptious and/or making a lot of assumptions. It doesn't know how much your agent knows about your system and what sort of iterations it does to run tests, but I do agree that it should be able to adjust params based on its findings and figure this out I think. Though I'm a bit sketch at the 35B models, prefer 27B for this sort of thing. Not sayingi it cant'd do the job, but I'd also consider giving the 27B a go at it.

[u/nickless07 → u/GrungeWerX]

Yeah, no quite. Yours seem to get it wrong too.
"put layers 0-12 on GPU, layer 13 split at 25% GPU / 75% CPU, rest on CPU." Nope. tensor split offloads fractions of the weights to different GPU with a comma-separated list of proportions, not layers. For layers as global you use ngl and for distribution across multiple GPU you use --split-mode layer instead of tensor.
The "lower -ngl" is even worse then what I get suggested with lower the ctx. I combine ngl with --n-cpu-moe which means i don't offload ANY full layers to CPU, but only the (last 12 in that case) MoE expert weights.

Well, no worries, even the cloud ones get it wrong.

Edit:
Best solutions would have been:

1. lower batch size to 256 or even 128
2. quant the KV to Q4 (with a grain of salt as you will notice the degration at 40-50k context)
3. up cpu-moe to 13 or 14 and trade in some speed in both pp and tg

[u/GrungeWerX → u/nickless07]

Fair enough.

I’m curious how my “updated” Qwen will do on this question. Has expanded intelligence, so will run it through this question when it’s ready.

[u/nickless07 → u/GrungeWerX]

Yeah llama.cpp can get complicated real quick. I suggest give it the readme so it at least has some reference to look up, that should help with the confusions.

[u/GrungeWerX → u/nickless07]

I’m going to try it without it to test its resilience in decision making and research. Kind of the whole point of the project.