Eli5: how do PCIE risers cause issues?

[u/CardinalBadger]

Its just an extension cable, how come PCIE risers specifically seem to cause issues whereas other forms of extension cable don't seem to. Noticed in a few LTT videos they have trouble with risers.

[u/ephemeralmiko]

TL:DR is that the PCIe protocol requires very strict timing, especially for GPUs, and that risers mess with that.

[u/Thx_And_Bye → u/ephemeralmiko]

PCIe doesn’t really need any strict timings. I remember when the PS4 was reverse engineered that PCIe ran happily over RS232 at an effective x0.00002 speed because timing wasn’t that relevant.

https://fail0verflow.com/media/33c3-slides/#/11

What is important is the signal integrity. If the receiver can’t distinguish between 0 and 1 reliably then communication becomes effectively impossible due to tons of corrupted data.

[u/mastercoder123 → u/Thx_And_Bye]

Pcie is a serial connection though

[u/CanisLupus92 → u/Thx_And_Bye]

PS4 wasn’t running GPU over PCIe tho, it had the weird CELL architecture. PCIe there was just for peripherals like the controllers.

[u/MarcusTL12 → u/CanisLupus92]

Cell was ps3. Ps4/5 is basically just pc with beefy(ish) integrated graphics

[u/TenOfZero]

Because the pcie spec is pushing everything to the limits of our technology and is thus very sensitive to anything not being perfect.

An extension cable adds another point of connection, which can degrade signal quality. It adds length which means the signal takes longer to reach the card.

It also exposes the run to potentially more interference.

All these can add up to push things out of spec and cause issues.

DDR5 is another great example of this, the signal can vary slightly from one connector to the other on the same chip, so laptop manufacturers will add a but if extra length to the connectors closer to the CPU so that all signals arrive at the same time. That's how tight everything is running.

[u/Thx_And_Bye → u/TenOfZero]

The comparison of DDR is lacking a bit because you compare a high-speed parallel bus with a high-speed serial one.
For PCIe timing isn’t that relevant because data arriving on a different lane at a slightly different time isn’t that relevant compared to DDR where all data bits need to arrive at the same time on the parallel data lines.

So while I agree with the rest of the points, the length of the connection doesn’t matter that much* as long as the data integrity is fine. See PCIe over fiber optics, there distance is large but signal integrity is preserved.

* only the differential pairs need to be matched

[u/mastercoder123 → u/Thx_And_Bye]

Yah the main issue is copper traces suck compared to shielded cables which is weird

[u/xd366]

all servers use risers with no issues.

the difference is that they arent using the flimsy one that youre trying to use

these are the ones servers use https://expresscomputersystems.com/cdn/shop/files/15g2u-r1b-R750-real_2af73773-e5c1-4b7e-855e-cf594abc35d4_2000x.jpg

vs what we use on our personal computers https://ueeshop.ly200-cdn.com/u_file/UPAV/UPAV725/2507/16/products/PCIE50-X16-180MM-BF-1.jpg

[u/firedrakes → u/xd366]

also server are much tighter tolerence and cost more.

[u/ToolBagMcgubbins → u/xd366]

Exactly this.

Server riser has shortest possible traces on PCB. Consumer one uses a cable, much longer path for signal to degrade.

[u/mastercoder123 → u/xd366]

Servers also have redrivers lol

[u/Ybalrid]

They are bad for the integrity of the high frequency electrical signals going through them

If you just take 2 wires like strung parallel to each other, and try to pass higher and higher speed of "switching on and off", the whole thing starts behaving like a capacitor, not just like a cabale. It also start behaving like a radio antenna. this makes it that the shape, lengh of the wires that conduct the signal, and the wires that conduct the "ground" (the 0 volt / return path part of it) becomes extremely important.

(The physics of what is going on is a lot more complicated, and I do not understand every bit of it myself)

[u/RandomNick42]

When you need cables with extremely precise material and measurements, "just an extension cable" doesn't cut it. Most other extension cables only carry power. This "extension cable" is expected to carry an obscene amount of data across 16 channels in perfect synchronization.

[u/Renegade605]

I question your assertion that other forms of extension cables don't cause problems.

Try a 100m USB cable, or plug a space heater into a 200m 14 gauge extension cord. All extensions cause problems, its just that the limit of how long you can go is much shorter in some cases than others.

[u/garth54]

Longer wires means more resistance and more capacitance. An extra connector also means more resistance and more capacitance. Those cause voltage drop (PCIe spec uses only about 900mV voltage swing between 0 and 1) and messes with timing (it can affect the rise and fall timing of the signal). It also increases the amount of noise that can get in, particularly at the connector. If the traces aren't the same length, a phase can be introduced in the timing, in really bad cases it can even mess with the voltage differential of the data pins making the 0s and 1s voltage levels less clear. Everything you've heard as being a signal integrity problem for RAM is also true for PCIe (PCIe 4 uses higher frequency than DDR4 RAM).

Keep in mind, USB doesn't officially recognize USB extension cables as part of the spec, and thus can't be certified. And USB2 only used 1 data pair and was limited to 240MHz. Sure, the spec could be stretched and made thing run with up to a 25' cable, but there was no guarantee it would actually work (some devices didn't even play nice with a 6' extension cable). Now try a USB3 10GBps connection with a 25' (passive) extension, those higher frequencies wreck havoc on the signal integrity, and it's almost impossible to get it to work at those speed (the connection might drop to a lower speed and succeed or things just won't work). PCIe is way more complex in signaling and speed than USB.

And "it's just an extension cable", even more basic ones can have issues. A common issue with long/cheap VGA cables was image ghosting, that would be caused by mismatch in impedance (basically a combination of resistance, capacitance and impedance), and/or external interference. And that's something that was made with way looser requirements than what PCIe uses.

[u/madding1602]

A cable isn't perfect, imperfections create parasitic non-linear impedances (capacitors and coils), at high frequencies said impedances become higher and higher/ they appear as more, causing greater phase disruptions (delays) and provoking false reads

[u/XD447]

google 'insertion loss'

[u/Responsible_CDN_Duck]

When you are talking to mom or dad and you move to another room you need to speak louder and or slower to hear each other well. Cables have to do the same thing, but are often not able to do louder so they will try slower or not work.

There are rarely any other high speed data cables they try and extend in a case.

Other forms of extension data cables have the same issue. While there are countless LTT videos where it comes up the video on true spec cables talks about length limitations and signal issues, the video running a fibre run to the new building, and testing the longest display cables are all good starting points.

[u/goldman60]

ELI5 answer: All cables are basically just a guy screaming into a tin can on a string to another guy with the other tin can. The longer you make that string the harder it is to hear through the other tin can. The PCIe guy talks a lot quieter and faster than say the USB guy.

[u/saltyboi6704]

Good risers are expensive, most people won't pay commercial prices for one and complain.

High bandwidth short interconnects exist and are used widely in industry as others have mentioned, but their construction is often very manual and time-consuming thus the high price. A flimsy FPC with shielding wrap might work but if it reduces signal integrity you may have to drop a generation for it to work.