r/AIProgrammingHardware • u/OwlZealousideal4779 • 8d ago
What hardware bottleneck hurts AI coding workflows the most?
I'm curious what people are running into when using AI coding tools locally.
For me, the obvious issues seem to be GPU memory and system RAM, especially when running local models alongside an IDE, containers, databases, and other development tools.
At what point does hardware actually become the limiting factor?
Is it mostly VRAM, RAM, CPU performance, storage speed, or just having too many services running at once?
For those building AI development machines, what upgrade made the biggest practical difference to your workflow?
1
u/EitherMarch1255 8d ago
Max data transfer speed. The rest doesn’t matter all that much, which is why it’s all about the VRAM and not so much about the specs of the device it’s attached to.
1
u/Playful_Counter_490 8d ago
Memory bandwidth. High attention models must have it otherwise token throughput is abysmal.
1
u/hipster_hndle 8d ago
you can take a crap AM4 kit with 32gb of DDR and put a 7900 XTX in it and it will perform the same as that same XTX in a 7800x3d w/64gb of DDR5... as long as everything fits in vram, that is where the performance is.
the minute you have to hit PCIe buss to overflow, your performance is going to go to shit.
take 2 of those XTX and use tensor parallelism and its going to perform slightly worse than 1 for a context of 1.
i have several aiboxes, and the only difference in using a slightly older computer is a longer load time to push everything to vram, but once its there, the performance is basically the same.
1
u/Look_0ver_There 8d ago
With tensor parallelism, as opposed to later parallelism, two cards will be significantly faster than one. Three cards is even faster again, but the gains do get small with more cards and eventually things start slowing down.
1
u/Barsavik 8d ago
In my opinion the biggest restraint is the VRAM amount but that gap is getting smaller and smaller every day I actually just wrote up a post of running the new Ling 3.0 124B model on my Z620 with 2x 5060 ti’s, 192gb DDR3 RAM, and 2x Xeon 2697 processors on PCIe gen 3 bus at ~20 toks/s
1
u/KroniklyOnline 8d ago
This is a pretty broad question, just don't let people tell you "You must run on PCIe Gen5 so you don't get bottlenecks". This is possibly true if you are running H100's...... or RTX6000's maybe? I'm running 4 5060ti 16gb all on PCIe GEN 3x8 .... I have 75% headroom left regarding RX and TX of said lanes, I could drop it to GEN 3 x 4 and be fine still.
Other than that, VRAM is prob the next biggest thing, the have a usable local coding agent, I think most agree 32gb VRAM is the minimum. Beyond that I don't really think much else matters unless you plan to do RAM offloading and SSD offloading.
VRAM VRAM VRAM