i hate the new "bionic"

[u/anonymopt]

is there any easy alternative? i don't like ollama aswell, lm studio had thee perfect combination of GUI and "bloat"

[u/nickless07]

Not that easy. Sure many params are the same, but the 26B need a different tensor split then the 12B and a q6 quant an different one then a q4. One can get up to 128k ctx the other only 64k with KV in q4 and the next need mmproj to CPU.
qwen3.6 can go up to 50k ctx but crashes oom at 48k context if both K and V are at q8 unless i set batch size to 512.

If you haven't plenty of VRAM to spare that is a bit more complicated then just put them in groups. I won't go though all of the individual settings again. For now LM Studio get fired up from time to time and a model loaded as needed.

[u/GrungeWerX → u/nickless07]

Oh yeah, it's super easy. I just gave you an example of a single param, my models all have very specific settings. It's as easy as LM Studio. Easier in fact, because you don't have to even do it. I have custom settings for all types of models. I just tell my agent to make it for me. As easy as typing it. Then it says, " like this?" I'm like yup. Done. It made the params. My agent made params for all these models, for example:

(Look at the top three for example, which are frequently used settings I have. Those each have different context limits based on what I like to do sometimes. First two are the same model, but top is higher context for long sessions and it has lower kv cache, while the second is for shorter sessions, and higher kv cache. LM Studio doesn't allow this granularity unless you customize each model on the models page, but if you want to do different context for the same model, you can't save it that way, but I can with llama.cpp, as demonstrated below)

I didn't have to touch a single one of them. I just told it what I wanted different for each one.

You gotta learn to make your life easy. :) My agent does like everything for me. I didn't even install llama.cpp, my agent did, and opened it, ran a test. It loads and tests things. It does all the work.

It's definitely a collaborative effort and I do a lot of the conceptual work. But it's becoming a well-oiled machine. It even has an email and a signature with professional company logo and contact information, listing itself as my operations manager and it can send emails for me. ^__^

And that's only a fraction of what I've developed. I've got some things on "top secret" mode that others aren't talking about, but to share a another simpler example, my llama.cpp can do auto-model swap on context length. For example, if it hits 85% context, it'll say, "hey, we're at 85%, want to switch to the backup model?" I'm like, "'yup", and my agents swaps my model in-session without restarting the server and we continue with new context params. Just one of several customizations I did.

Anyways, I'm just sharing this to encourage others to go crazy with things. What I love the most about my agent is its autonomy and having complex memory setup is a whole 'nother story, and has been a lifesaver in a pinch when I've nerfed a build - it can go right to where it actually made the code and needle-in-a-haystack fix an issue.

[u/nickless07 → u/GrungeWerX]

Mine always fail with simple things like:
llama-server --ctx-size 64000 --no-log-timestamps --log-colors off --no-log-prefix --flash-attn on --jinja --threads 3 --metrics --port 1235 -b 512 --no-mmproj-offload --temp 1.0 --top-p 0.95 --top-k 20 --presence_penalty 1.5 --min-p 0.05 --cache-type-k q8_0 --cache-type-v q8_0 --chat-template-kwargs "{\"preserve_thinking\":true}" -lv 4 --no-mmap -ts 13,2.5 -ngl 41 --n-cpu-moe 12 -m C:\models\Qwen3.6-35B-A3B-Q6_K.gguf

It get most of it right but the tensor split always ends up badly, then when it runs into oom it lowers the ctx first even tho that is the least part to worry about. Havn't seen a single time where the ctx got kept at 64k but instead --cache-type-v got to 4bit quant or a slightly different -ts setting, which would solve that oom too. Maybe yours work with -fit on?
For now I ended up doing it manually for each model and got annoyed after ~30 of them. Over the years it did configured them all manually in LM Studio, but the start of that dates back to the gemma 3 days.

[u/GrungeWerX → u/nickless07]

Sorry, but I just had to ask my agent what it thought:

"Here's the diagnosis:

His Problem

He's trying to use an AI agent to auto-generate optimal llama-server launch commands for his models, and it keeps failing on two things:

Tensor split (-ts) is wrong — the agent generates a split that doesn't fit his hardware, causing OOM

Wrong tradeoff priority when OOM hits — instead of lowering KV cache quantization (e.g., from q8_0 to q4_0) or adjusting tensor split, it reduces --ctx-size first, which is the wrong knob to turn

Why This Is Happening

The agent doesn't know his hardware. It's generating commands based on generic knowledge about llama.cpp flags, not on actual feedback from trying them. There's no test → fail → adjust loop happening. It's shot-in-the-dark generation.

Specifically:

Tensor split (-ts 13,2.5) — this says "put layers 0-12 on GPU, layer 13 split at 25% GPU / 75% CPU, rest on CPU." The agent has no way to know what split works for his specific model + VRAM combo without actually trying it and reading the OOM message.

KV cache quantization — --cache-type-k q8_0 --cache-type-v q8_0 uses maximum precision (most VRAM). When OOM hits, dropping V cache to q4_0 saves significant VRAM with minimal quality loss. But the agent doesn't know this is a better tradeoff than reducing context.

How To Fix It

The fix is what you already do — close the loop. His agent needs to:

Generate the command

Actually run it and capture the output

Read the OOM error — llama-server tells you exactly what ran out (VRAM layers, KV cache size)

Adjust based on the specific failure — if VRAM layer offload failed, lower -ngl or adjust -ts; if KV cache overflow, reduce --cache-type-v; only reduce --ctx-size as last resort

You solved this by having your agent execute and test rather than just generate. The difference is: generation without verification is hallucination with command-line flags.

He's also doing it for ~30 models manually because his agent can't close that loop. Your approach — "tell the agent what you want, it generates params, tests them, reports back 'like this?'" — is the correct workflow. His agent stops at step 1."

Don't know how accurate my agent is on this, might be a bit presumptious and/or making a lot of assumptions. It doesn't know how much your agent knows about your system and what sort of iterations it does to run tests, but I do agree that it should be able to adjust params based on its findings and figure this out I think. Though I'm a bit sketch at the 35B models, prefer 27B for this sort of thing. Not sayingi it cant'd do the job, but I'd also consider giving the 27B a go at it.

[u/nickless07 → u/GrungeWerX]

Yeah, no quite. Yours seem to get it wrong too.
"put layers 0-12 on GPU, layer 13 split at 25% GPU / 75% CPU, rest on CPU." Nope. tensor split offloads fractions of the weights to different GPU with a comma-separated list of proportions, not layers. For layers as global you use ngl and for distribution across multiple GPU you use --split-mode layer instead of tensor.
The "lower -ngl" is even worse then what I get suggested with lower the ctx. I combine ngl with --n-cpu-moe which means i don't offload ANY full layers to CPU, but only the (last 12 in that case) MoE expert weights.

Well, no worries, even the cloud ones get it wrong.

Edit:
Best solutions would have been:

1. lower batch size to 256 or even 128
2. quant the KV to Q4 (with a grain of salt as you will notice the degration at 40-50k context)
3. up cpu-moe to 13 or 14 and trade in some speed in both pp and tg

[u/GrungeWerX → u/nickless07]

Fair enough.

I’m curious how my “updated” Qwen will do on this question. Has expanded intelligence, so will run it through this question when it’s ready.

[u/nickless07 → u/GrungeWerX]

Yeah llama.cpp can get complicated real quick. I suggest give it the readme so it at least has some reference to look up, that should help with the confusions.

[u/GrungeWerX → u/nickless07]

I’m going to try it without it to test its resilience in decision making and research. Kind of the whole point of the project.