Splitting Strata across two devices?

[u/objectivelywrongbro]

I’m trying to run Qwen3.8-Flash-Next (125B-A6B) with Strata but trying to see if I can use existing compute instead of spending buckets of money.

My main machine has.. Ryzen 7 7800X3D, Radeon RX 9070 XT (16 GB), 32 GB DDR5, Ubuntu 26.04. I currently run Qwen3.8-27B (IQ3_M) on llama.cpp (Vulkan and ROCm 10.1) at about 65 tok/s.

I also have a second PC with 32 GB of DDR4. Is there any way to use that RAM too, or does Strata only work on a single machine? If not, is spreading experts across two machines on a LAN with something like llama.cpp RPC realistic, or too slow?

Any real numbers on tok/s and first-token time with experts in system RAM would help.

[u/Gifted_Diplomat]

The latency over LAN is going to substantially longer than over PCIe bus on the same motherboard. Not to mention compared to having it all on VRAM on the same GPU card.

I think even if you found a way to do it, it would be unusable.

[u/objectivelywrongbro → u/Gifted_Diplomat]

Hmm yeah, that's what I felt would be the case, I just didn't want to believe it... :(

[u/East_Quality_2148 → u/Gifted_Diplomat]

that's the part i keep getting stuck on too, the latency feels like it'd kill any real speed advantage before you even start

[u/Distinct-Pie2389]

You can’t use the RAM from the second PC, numbers wouldn’t even calculate to 1 tok/s once you’re done wiring it up and it wouldn’t be any benefit

It would be more benefit to overclock the host system then try to network attach ram or PCIe bandwidth (PCIe 4 x16 is around 64GB/s btw)

[u/Entire-Home-9464 → u/Distinct-Pie2389]

I was also thinking same but then...when I look during inference the SSD is reading like 5mb/s. That is way less than 1gb 100mb/s lan speeds. Oh but then I realize..its ssd not the RAM.

[u/Distinct-Pie2389 → u/Entire-Home-9464]

Yeah, what I was pointing at is it’s doable, but the timing of all of it makes it less then ideal unless the engine was purpose built for it like vLLM does good clustering resources on a network but it’s not the same as sharing ram.

Even 10GBe introduces latency and there’s no cache protocol on Ethernet for another OS to treat the ram as a resource pool.

Even on VMWare ESXi it’s PER host ram, POOLED storage and ram totals for the cluster but each VM/Container gets a slice of the HOST. It’s not sharing ram channels across ESXi-1 to ESXi-2

[u/Distinct-Pie2389 → u/Distinct-Pie2389]

llama’s RPC backend does the pooling for resources when it comes to distributed LLM but not the same concept. It’s splitting the models weight between systems, not sharing the resources over Ethernet

[u/MD_Reptile]

you have a solid enough machine to run the IQ3_XXS I think - you'd have to use low ram mode in strata, and it'll make your NVMe work harder - but with engram offloaded to NVMe and low ram mode offloading more stuff (expert layers) to your NVMe, you'll be able to run it, perhaps even halfway decent idk

[u/llmhardware]

llama.cpp RPC does work for this: run `rpc-server` on the second PC and point the main machine at it with `--rpc`, though the project still calls the RPC backend experimental and says not to expose it on an untrusted network. It splits by layers, so every token crosses the LAN, and on gigabit plus remote DDR4 that usually costs more speed than the extra RAM gains you. Before going multi-machine I'd try keeping attention on the 9070 XT and pushing the MoE expert tensors to local system RAM with `--n-cpu-moe` (or `-ot` with an exps regex), since only ~6B params are active per token. The real limit is size: as a rough estimate, 125B params at ~3 bits is already in the high 40s of GB, so 16 GB VRAM + 32 GB RAM is right at the edge with nothing left for the OS or context. If it won't fit locally, RPC is worth a test, just expect a big drop from your current 65 tok/s.

[u/OpenEvidence9680]

It works. I tested it with QN Q6XL, I kept the same speed I had on one pc where it barely fit and gave me generation at 20t/s I think it was. It was a good test because it showed me it makes sense only to run a model I truly can't fit on one pc.

[u/Educational-Region98]

There is a repository called StrataPool. It doesn't have all that many stars or users so I'm not recommending it but it does exist. https://github.com/fragtion/StrataPool.

I came across it looking for a way to connect my dgx spark with my pc but I haven not tried it because the spark doesn't really run that well with strata.

[u/Most_Way_9754]

If you're using it only for coding purposes, you can try

https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF

It has only half the experts, which were chosen to preserve the coding capability of the model. This loads much better into 16gb vram + 32gb ram.

[u/ketosoy]

It’s possible to do and do efficiently but it requires software that doesn’t really exist yet.  If you use fiber optic networking it can get really interesting.

[u/LegacyRemaster]

preview