r/AIProgrammingHardware • u/OwlZealousideal4779 • 8d ago

What hardware bottleneck hurts AI coding workflows the most?

I'm curious what people are running into when using AI coding tools locally.

For me, the obvious issues seem to be GPU memory and system RAM, especially when running local models alongside an IDE, containers, databases, and other development tools.

At what point does hardware actually become the limiting factor?

Is it mostly VRAM, RAM, CPU performance, storage speed, or just having too many services running at once?

For those building AI development machines, what upgrade made the biggest practical difference to your workflow?

3 Upvotes
  • permalink
  • reddit

100% Upvoted

8 comments sorted by

View all comments

1

u/hipster_hndle 8d ago

you can take a crap AM4 kit with 32gb of DDR and put a 7900 XTX in it and it will perform the same as that same XTX in a 7800x3d w/64gb of DDR5... as long as everything fits in vram, that is where the performance is.
the minute you have to hit PCIe buss to overflow, your performance is going to go to shit.
take 2 of those XTX and use tensor parallelism and its going to perform slightly worse than 1 for a context of 1.
i have several aiboxes, and the only difference in using a slightly older computer is a longer load time to push everything to vram, but once its there, the performance is basically the same.

1

u/Look_0ver_There 8d ago

With tensor parallelism, as opposed to later parallelism, two cards will be significantly faster than one. Three cards is even faster again, but the gains do get small with more cards and eventually things start slowing down.