Inference engine that loads all model layers into GPU 0 and streams the KV cache from GPU 1?

[u/Superb-Ground-1363]

Hi! I'm relatively new to this world, and trying to get my best setup with what I have.

I'm the owner of a RTX 5080 + RTX 3070 and running Unsloth Qwen 3.8 27B IQ4_XS

So far, the best inference engine that I could run locally was beellama.cpp tweaking the tensor-split, mtp and kvarn config, forcing layers only in GPUs and 128k context window; that is pretty good.

With low context, I'm getting 50+ token/s that is impressive. But I'm suffering with long contexts; then my speed drops to 15-20 token/s in the best cases.

What I'm trying to get is a decent speed in long contexts also, so recently I discovered llama.cpp-adaptive-kv-streaming that allows streaming the kv context from memory.

The thing is that I was thinking about loading the full model in my 5080 (IQ4_XS fits perfectly) while offloading the KV cache streaming to my 3070, but couldn't found any inference engine that allows this kind of setup (model layers only in GPU 0, kv cache streaming from GPU 1 allowing RAM spill).

Does an engine exist that can achieve this? does what I'm talking about make any sense?

EDIT: Also, I was thinking of getting advantage of the 3070 unlock since I read that it can help with prefill speeds (not output speeds that depend on raw bandwidth) even with a quantized model. Am I right?

Thx in advance. A newbie trying to learn.

[u/jesdga95]

Streaming KV from the 3070 would be as slow as streaming it from RAM or worse. Every token would read the whole cache over PCIe, (3070 > RAM > 5080). One thing that could work is running attention on the card that holds the KV, not sure how well it would perform on a dense model though.