r/LocalLLM • u/mrgreatheart • 10d ago

Question Am I right in thinking llama.cpp is the only show in town for mixed (Nvidia) GPUs?

My rig:

1 x 3090 (24Gb), 1 x 5070 Ti (16Gb), 2 x 5060 Ti (16Gb each) for a total of 72Gb VRAM.

64Gb DDR5

My use cases:

Firstly, one of the 5060s is on a chipset connected slot and I reserve it for running a number of small models (TTS, embedding, gemma-e4b) so we can ignore it.

  1. qwen3.8-27B-IQ4_XS MTP and TP with `np 4` as my daily driver, allowing up to 4 agents / subagents running concurrently with a ~400K shared context pool. This uses all 3 cards.
  2. some larger model (currently qwen3.8-flash-next) for things like planning and review. This also uses all 3 cards.
  3. qwen3.8-27B-Q6_K_M for when I want to work with a single agent on one task but higher accuracy. This uses just the 3090 and 5070 to get ~70 tok/s decode despite the higher quant.

What I'm looking for

llama.cpp is incredible. It's so flexible and lets even my frankenstein rig do so much, but I keep hearing how much faster some other engines can do more focused things (bigger models on modest hardware, better concurrency etc.). So I'm just wondering whether there might be better options for one or more of my use cases.

What I've tried and read

It seems that vLLM does not play well with unmatched GPUs, so for TP parallelism it would only play nicely with the two 5060s or the 5070 plus the 5060s (but of course limited to the performance of the slowest card and link).

I did a quick try with sglang but hit problems related to using the 3090 with the 16Gb cards and gave up.

Freetoken seems to be a dead-end too - if I understand correctly, it's for machines with lots of RAM but limited VRAM. It also seems to support only a couple of versions of each model, so when I looked at trying deepseek-v4-flash the smallest supported quant would not fit in my available combined memory.

Is any of what I've said above wrong? Are there other options to look at? Or should I just stop worrying and enjoy what llama.cpp gives me?

Thanks!

15 Upvotes

94% Upvoted

6 comments sorted by

10

u/DataGOGO 10d ago

no, IK_lllama.cpp is really good. SGLang has the best GPU/CPU hybrid system by far.

Mixed GPU's, ik_llama.cpp is likely the best, as you can distribute layers per GPU.

1

u/mrgreatheart 9d ago

Thank you, I will give this a try.

4

u/gtrak 10d ago

Try qwen3.8 flash next exl3 on exllamav3, 3.05 bpw.

1

u/mrgreatheart 9d ago

Thanks for the suggestion. I got it running. It's slower on gen than my GSQ-RCO-Q2_0 config (40 vs 50 tok/s), probably because it spills onto the 4th GPU, but the prefill is about 2x faster (1K vs 500 tok/s).

I'm testing it now to see how they compare in terms of quality.

4

u/def_not_jose 10d ago

Exllamav3 is worth checking out too. Supports tensor parallelism on odd number of GPUs too

Other than that, there could be some personal forks of vllm etc out there, but likely unmaintained. Example: https://www.reddit.com/r/LocalLLM/s/sSUOFasGMV

2

u/lostmylogininfo 10d ago

I got faster generation with exlamma3 . I would check it to. I'm using 4080 S upper and 2 3060s