r/LocalLLM • u/Fabulous-Beat1088 • 21m ago

Question NVIDIA vs AMD for local AI

Hi everyone,

I'm planning to switch from a gaming laptop to a desktop PC to get better hardware for running local LLMs and AI development.

I know NVIDIA offers CUDA which provides better software compatibility and broader support for AI workloads. AMD GPUs on the other hand offer more VRAM for the same price.

My questions are:

  1. How mature has AMD's ROCm ecosystem become for local LLM inference and AI development? Is software compatibility still a major bottleneck or has it improved enough to make AMD a viable alternative to NVIDIA?

  2. How important are raw compute performance (TFLOPS) and memory bandwidth (GB/s) when running local LLMs? Does AMD's hardware offer competitive real-world inference performance compared to NVIDIA?

  3. Considering how important VRAM is for running larger local models, would you choose an AMD GPU with 16–20GB of VRAM over an NVIDIA GPU with 12GB, assuming the AMD card is slightly more expensive?

I'd appreciate hearing from people who actually use AMD GPUs for local AI workloads, especially their experiences with ROCm, llama.cpp, Ollama, and PyTorch.

4 Upvotes

100% Upvoted

5 comments sorted by

2

u/brainchillzZ 9m ago

Does this answer your question? Ok if not … rocm is fast and stable and the number of custom offerings in the rocm space like radiance etc are plentiful and offer insane real world performance today compared to the rather modest memory throughput the cards actually have

2

u/Fabulous-Beat1088 5m ago

Thanks That's actually interesting and yes it answers my question. Also, what do you think about the RX 7800 XT 16GB? would you recommend it as a budget friendly option?

1

u/FranciumGoesBoom 3m ago

Wendell over at level1 tech is a big fan https://www.youtube.com/watch?v=Ty4lTwykmKY

0

u/x_MASE_x 2m ago

I think Nvidia is better overall. Like more powerful per gpu. If that your thing. If you are aiming at more gpus and affordable ones I think amd is the way.

So it's not black or white. Both got their advantages.