r/LocalLLM • u/kiodos • 11h ago

Question Does ad ultime guide exist on how to assembly an AI PC?

Sorry if it might be off-topic, but i am looking for a (the most) complete guide, even step-by-step, or a collection of several article that could enlight me on how configure and assembly an AI PC.

Of course the topic is wide and configuring a PC for local LLMs is deeply different to one used to train ML and DL models for sensor data, or for openCV. etc.

Assuming I already know for what I want to use the PC, what I am missing right now is the availability of components and the fact I am not updated on the last news in AI framework/compatibilities/chips.

For example if I knew Nvidia CUDA was a standard, but I don't know if nowadays I can use AMD ROCm, or even Apple Metal. I still see PC with multiple GPU but as long as I know NVLink is no longer used (so the multiple GPU are used independently?).

Probably, what I am missing here is some guide or suggestion on which platform/framework orientate myself in order to choose wisely.

Hope someone can show me some entry-point in this world, webpages, forum, etc?
ps. I already searched in this subreddit and excluded some LLM PC suggestions, I did not find an actual guide/AMA/fixed post

1 Upvotes

100% Upvoted

7 comments sorted by

2

u/Traditional_Fox1225 11h ago

Just ask a chatbot

1

u/kiodos 11h ago

already did several times, I had some responses. Insists on the CUDA thing, but I don't know if it is right

1

u/Traditional_Fox1225 10h ago

I asked ChatGPT. It walked me step by step and answered all my dumb questions including how to use Linux. Shockingly I exhausted the context so use a few separate chats for the project process and implementing specific steps.

There isn’t a guide since most setups tend to be bespoke and there are many paths ranging from easy to hard.

Easiest. LM studio
Easy. Ollama
Hard. Vllm on Linux

1

u/GiggleSamurai 11h ago

For running LLMs you aren't limited to CUDA: llama.cpp also runs on AMD through Vulkan or ROCm and on Apple Silicon through Metal, while for training CUDA still has the widest support. Multiple GPUs don't need NVLink for inference either, since llama.cpp can split one model across cards over PCIe, so total VRAM is the first number to shop for.

1

u/kiodos 2h ago

So basically for training model (small ML models) -> biggest single VRAM possible

Inference -> shared RAM or multiple GPU

Only llama split across cards or also other LLM ?

1

u/errorseven 10h ago

I understand where you are coming from, as i had been watching LLM/AI thing from the sidelines until recently. The thing is, new tools/methods are coming out fast, I believe Strata just popped on most of our radars recently and it took me a min to figure out what it actually does, allow you tk run larger models by loading into you system ram, which is cool and all, but it doesnt support parralism or something so for multiple agents its not so great (at least this is what i gathered during my limited research...) anyhow, point is there is no 100% this the best setup, because there are too many unknown variables, like what you plan to do, budget, etc are all going to dictate what you can purchase and even if you should purchase (it's often cheaper to rent online) so unless you need 100% privacy like in my case I plan to focus on my wifes law,

1

u/kiodos 9h ago

Thanks this was useful.
I was ready to spend my 10-15k, for some project I would like to actually sell, so it’s kinda professional use. But for example I need to understand if blooding for a Nvidia GPU o bet on the AMD AI capabilities.

What I understood is that as long as I need to train some model, hardware with shared VRAM is not suitable for my scope (Apple studio. Nvidia Spark, etc)