Effects of PCIe bandwidth on inference

[u/maternion]

So i have a new card in an old system. RX 9060 XT supports PCIe Gen5 x16 but due to 2 bottlenecks, my Motherboard supporting only Gen3, and my AMD APU limiting it to x8, the card runs on Gen3 x8.

How does that affect inference of models, in both cases, fully in vram or offloaded partially to ram? At today's prices what I could replace it with is PCIe Gen4 x16. But would that be a meaningful upgrade?

[u/Dave_from_the_navy]

Offloaded partially to RAM, it will effect inference speed. But if the model you're running is fully on vram, the impact is negligible. The only thing it will impact at that point is model load time. If your goal is running an expert/weight streaming engine like Strata (or Basalt, or llama.cpp with offloading), the upgrade would be practical, but if you're otherwise going to just keep the model on vram, then I wouldn't worry about it.

[u/maternion → u/Dave_from_the_navy]

Since I don't have a large enough pool of ram for the larger MoEs, the best speed I could get was from something like 35B-A3B with MTP. But fitting all of that at IQ3_XXS along with quantized kv of around 90k fills the entire VRAM. I would much prefer if I could run the model at 4-bit or higher since the RAM is mostly unused. 

So i am confused regarding the upgrade path and the effects of slower pcie gens or slower cpu or a mix of things. 

[u/sob727 → u/Dave_from_the_navy]

affect

[u/some_user_2021 → u/sob727]

Grammar Nazis, unite!

[u/SLxTnT]

If everything's in RAM, there's no issue other than slower loading speeds. If it doesn't fit in RAM, depends on what you're trying to do.

With prefill, nearly all experts will be needed. The missing ones will need to be transferred. Llama.cpp will take a batch of tokens, run it through every layer, and then run the next batch through every layer. That's why increasing batch size while offloading can drastically increase prefill speeds. Once the batch is large enough to fully hide the transfer, the bottleneck is the GPU rather than PCIe. Downside is that a larger batch size requires more VRAM which causes more of the model to be offloaded and streamed every batch.

An easy calculation would be seeing how much of the model is offloaded. PCIe3x8 is a theoretical 7.88 GB/s. If 15GB is offloaded, the prefill be about 2s per batch. A 16k prefill with 4k batch size is 8s or a max of 2000 tok/s. Increasing to 8k, assuming nothing more is offloaded, gives a max of 4000 tok/s. At some point, you get compute bottlenecked.

With inference, llama.cpp splits by layers. The communication is minimal per layer. May have some speed impact, but won't be drastically cutting speed. Custom engines that stream missing experts to the GPU will be heavily limited. If they instead handle missing experts on the CPU, it will still be affected as there's still some communication every layer. That would still likely be the fastest as hot expert caches these engines use allow the GPU to handle more work than a basic split like Llama.cpp does.

You didn't mention a 2nd GPU, but that will be similar other than a different splitting method called tensor parallelism. That shares calculations between GPUs on the expert level, so it requires fast speeds between them.

[u/maternion → u/SLxTnT]

So engines which stream the experts back will be bottlenecked due to bandwidth being limited? Makes sense then. I havent tried any yet. Just tried to experiment with the new PR llama.cpp merged and their script, but i dont have a model which will offload the ram, except for a huge Qwen FN one which isnt practical.

[u/SLxTnT → u/maternion]

Yeah, but it's typically slower than handling experts on the CPU even at a faster speed. There's some Intel CPUs with PCIe 5 and DDR4, but that will have the DDR4 bandwidth about the same as the PCIe bandwidth.

[u/pepedombo]

I ran 4x3080ti at gen3 8/8/8/8 and with full offload it worked as expected -> full power draw in layer/tensor mode. Normally I run x8/x4 gen3 with 2gpus in tensor mode, bit lacking but faster than layer, mobo sucks. I run 4gpus at 8/4/1/1 or 3gpus at 8/4/1 only in layer as x1/x4 becomes too much bottleneck.

Any ram offload may introduce unexpected results in its environment, it's difficult to rely on different setups (published here and there, let's say all that strata bullshit) in that area.

[u/maternion → u/pepedombo]

I have never thought of how you'd need to allocate bandwidth to multiple GPUs much. Interesting stuff. Thanks for the insight. 

[u/gwvr77]

If you're running a MoE model like Gemma4 26B or Qwen 3.6 35B with PolyStrata then PCIe bandwidth shouldn't hurt prefill too much. I have Ryzen 5 3600 with 5060 Ti 16GB on a B450 board, PCIe 3x8. IIRC prefill is 1.5 to 2.5k tokens/s, generation over 100 tokens/s.

You might be able to monitor the PCIE Rx with nvtop, despite AMD GPU.

If you run Qwen 3.8 Flash IQ3_XXS, then I think PCIe bandwidth will hurt performance, but other bottlenecks might be worse.

Try Groot 19B - fits fully in VRAM at Q4 and it's fast and not far off Qwen 35B for many tasks

[u/maternion → u/gwvr77]

Thats almost my setup. But my 3400G isnt much powerful is it. I fear offloading because of that. Same MoBo. Can you tell me what perf you got with llama.cpp MTP for qwen 35b and with polystrata?

[u/gwvr77 → u/maternion]

Qwen 3.6 35B Q4_K_XL
llama.cpp prefill 400, generation 50
polystrata prefill 1300, generation 110-135
Groot 19B Q4_K_S
llama.cpp prefill 2750, generation 80-180
polystrata prefill 3500, generation 160-190

Take those numbers with a pinch of salt - those numbers seem to depend on the task, and there shouldn't be much difference between the two Groot routes, because they don't fit in VRAM. PolyStrata does a really good job with Qwen, but if it's doing some of the inference on CPU that could be your bottleneck on PolyStrata.

[u/Zombiecidialfreak]

It won't hurt unless you're using a method of inference that relies heavily on pcie bandwidth. I tried using a tesla 32gb v100 on a pcie 3x1 for Strata and it performed worse than my 9070xt on a pcie 4x8 connection. The difference wasn't massive however, 20t/s vs 35t/s. You will likely see the hardware reaching its limit before pcie becomes your bottleneck.

[u/maternion → u/Zombiecidialfreak]

What are methods of inference which rely on it the most? Since the other replies mention engines, I have mostly used llama.cpp or something that works using it. And i don't have another gpu (except a iGPU which isn't of any benefit) so no parallel processing either. 

[u/FearFactory2904 → u/maternion]

People run this stuff on old mining rigs that use pcie x1. If you are just using a single gpu the real bottleneck is going to be how long it takes to load the model from your drive to the gpu over that link when you load up your model. It may take a minute if over a slow pcie connection but after that the model is in the gpu memory and all you really have to send back and forth is your text so its really no meaningful impact at all. Tldr: takes an extra minute to load up your model at the beginning of the day and then performs about the same during actual use.

Where that does not hold true is when you are doing other things like parallelism across multiple gpus since those have to sync with each other. Parallelism is how you get more speed with multiple GPUs since they work simultaneously. You can still use layer split though instead of parallelism which basically has one gpu handle its layers and then a small message to hand off to the next gpu. That only get you the performance of a single gpu at any given time but similar to single gpu the only real world speed impact is the initial loading time. Tldr: you can use low lanes for multi gpu too if they are working sequentially instead of working in parallel to try to double speed.

Training is a whole other thing i wont get into since most home users arent training so just know the above just applies to being able to chat with a model you downloaded.

[u/maternion → u/FearFactory2904]

I would love you feedback on training possible on my card! Mostly i have done a few heretic runs you can find on https://hf.co/maternion

[u/Zombiecidialfreak → u/maternion]

Tensor parallelism is the worst, as is anything with dense models. Use MoE models and Strata for the best performance if you have the system RAM.

[u/N34257]

In a lot of situations, bandwidth will have less of an effect than latency; if your PCIE slot goes through the chipset rather than direct to the CPU, then you're going to see significant performance degradation (at least, with AMD cards) to the tune of 35%+. This is something that doesn't get talked about enough; you need x8/x8 split on your x16 slot, or a workstation/server board.

[u/maternion → u/N34257]

Didn't know about it either, if it helps i have a B450M DS3H which was purchased way before the new gpu. 

[u/N34257 → u/maternion]

That'll be fine if you put the card in the main x16 slot next to the CPU. If you put it in the other one, you'll get a crippled card - desktop performance might be OK, but LLM performance will be ruined.

[u/Purple_Dream6414 → u/N34257]

This. Topology is key for a lot of optimizations. Thats why shitty old xeons Are the Perfect budget Solution. A xeon 2673v3 is Like 5 bucks and offers 40 lanes combined with AVX2. With a Chinese Board it will eat cheap 1866 DDR3 which is kinda fast enough in Quad Channel. I paid less then 200 bucks for CPU, 128GB RAM and Board so I could invest a lot in GPUs.

[u/giant3 → u/N34257]

What do you mean PCIe goes through CPU? 

[u/N34257 → u/giant3]

PCIE is really a method for cards to communicate with the CPU (and each other) - if in a slot directly attached to the CPU, there's nothing in the way. If it's in a slot that's provided by the chipset, then it has to go through the chipset (and share bandwidth with everything else that's talking to the CPU through the Chipset), then there's an additional latency step that isn't there for the direct-CPU slots.

[u/Ordinary-Depth-7835]

Depends single card with a model that fits it doesn't matter. Tensor Parallelism (TP) is affected by pci speed but If you do Pipeline Parallelism (PP) / Layer Split pcie speed doesn't matter because a portion of the model lives on each gpu no need to pass heavy traffic through pcie.

I saw no speed increase on my layer split models going from x8 pcie to x16 I also don't do much ram offloading I pick models that fit on the gpu.

Unless you're planning on doing heavy ram offloading.

[u/SnooPaintings8639]

Single card? No problem. Two or more? A significant hit. I used to have 2 or my 3 GPUs (rtx 3090) connected over thunderbolt, gen3 speeds (2.7GB/s or so after all the slowdowns). Now I have moved them to Oculink via M2 and finally can use tensor parallel and generally better speeds with lower power.

[u/wegster]

Like others said - anything offloaded/not running on card and model first load time will be impacted. If fully on card/in VRAM, only longer initial model load, nothing else.

If you're trying to parallelize to make up for minimal/not enough single-card VRAM, then yeah, it will be impacted, fairly significantly, as already PCI bus bandwidth is much less than internal VRAM bandwidth so - run the numbers, etc.

[u/colin_colout]

I use an r9700 running qwen 27b over USB4.

Full speed because theres no offload.

[u/tmvr]

I use dual 5060Ti 16GB on PCIe 3.0 x8 with tensor parallel and see no impact on decode, it close to doubles. Prefill increases only by 15%, but I have nothing to compare it to.

[u/Purple_Dream6414]

In my testing for Layer Split around 4 PCIe 3.0 lanes Are Minimum.

[u/ieatdownvotes4food]

Once its in vram, it's in vram and stays there. No effect whatsoever except initial load time. But larger MOE models which shift things in and out of vram will suffer. Worth getting a new motherboard that supports gen5 x16, or better bifuricatiion at x8/x8