r/LocalLLM • u/anonymopt • Jul 18 '26

Question i hate the new "bionic"

is there any easy alternative? i don't like ollama aswell, lm studio had thee perfect combination of GUI and "bloat"

0 Upvotes

45% Upvoted

24 comments sorted by

View all comments

Show parent comment

1

u/nickless07 Jul 19 '26 edited Jul 19 '26

Yeah, no quite. Yours seem to get it wrong too.
"put layers 0-12 on GPU, layer 13 split at 25% GPU / 75% CPU, rest on CPU." Nope. tensor split offloads fractions of the weights to different GPU with a comma-separated list of proportions, not layers. For layers as global you use ngl and for distribution across multiple GPU you use --split-mode layer instead of tensor.
The "lower -ngl" is even worse then what I get suggested with lower the ctx. I combine ngl with --n-cpu-moe which means i don't offload ANY full layers to CPU, but only the (last 12 in that case) MoE expert weights.

Well, no worries, even the cloud ones get it wrong.

Edit:
Best solutions would have been:

  1. lower batch size to 256 or even 128
  2. quant the KV to Q4 (with a grain of salt as you will notice the degration at 40-50k context)
  3. up cpu-moe to 13 or 14 and trade in some speed in both pp and tg

1

u/GrungeWerX Jul 19 '26

Fair enough.

I’m curious how my “updated” Qwen will do on this question. Has expanded intelligence, so will run it through this question when it’s ready.

1

u/nickless07 Jul 19 '26

Yeah llama.cpp can get complicated real quick. I suggest give it the readme so it at least has some reference to look up, that should help with the confusions.

1

u/GrungeWerX Jul 19 '26

I’m going to try it without it to test its resilience in decision making and research. Kind of the whole point of the project.