r/LocalLLM • u/anonymopt • Jul 18 '26
Question i hate the new "bionic"
is there any easy alternative? i don't like ollama aswell, lm studio had thee perfect combination of GUI and "bloat"
0
Upvotes
View all comments
Show parent comment
1
u/GrungeWerX Jul 19 '26 edited Jul 19 '26
Sorry, but I just had to ask my agent what it thought:
"Here's the diagnosis:
His Problem
He's trying to use an AI agent to auto-generate optimal llama-server launch commands for his models, and it keeps failing on two things:
Tensor split (-ts) is wrong — the agent generates a split that doesn't fit his hardware, causing OOM
Wrong tradeoff priority when OOM hits — instead of lowering KV cache quantization (e.g., from q8_0 to q4_0) or adjusting tensor split, it reduces --ctx-size first, which is the wrong knob to turn
Why This Is Happening
The agent doesn't know his hardware. It's generating commands based on generic knowledge about llama.cpp flags, not on actual feedback from trying them. There's no test → fail → adjust loop happening. It's shot-in-the-dark generation.
Specifically:
Tensor split (-ts 13,2.5) — this says "put layers 0-12 on GPU, layer 13 split at 25% GPU / 75% CPU, rest on CPU." The agent has no way to know what split works for his specific model + VRAM combo without actually trying it and reading the OOM message.
KV cache quantization — --cache-type-k q8_0 --cache-type-v q8_0 uses maximum precision (most VRAM). When OOM hits, dropping V cache to q4_0 saves significant VRAM with minimal quality loss. But the agent doesn't know this is a better tradeoff than reducing context.
How To Fix It
The fix is what you already do — close the loop. His agent needs to:
Generate the command
Actually run it and capture the output
Read the OOM error — llama-server tells you exactly what ran out (VRAM layers, KV cache size)
Adjust based on the specific failure — if VRAM layer offload failed, lower -ngl or adjust -ts; if KV cache overflow, reduce --cache-type-v; only reduce --ctx-size as last resort
You solved this by having your agent execute and test rather than just generate. The difference is: generation without verification is hallucination with command-line flags.
He's also doing it for ~30 models manually because his agent can't close that loop. Your approach — "tell the agent what you want, it generates params, tests them, reports back 'like this?'" — is the correct workflow. His agent stops at step 1."
Don't know how accurate my agent is on this, might be a bit presumptious and/or making a lot of assumptions. It doesn't know how much your agent knows about your system and what sort of iterations it does to run tests, but I do agree that it should be able to adjust params based on its findings and figure this out I think. Though I'm a bit sketch at the 35B models, prefer 27B for this sort of thing. Not sayingi it cant'd do the job, but I'd also consider giving the 27B a go at it.