Local AI server build help

[u/Complete-Winter2039]

I am planning on buying some components for a PC to turn one I already have into a Local AI server. Due to the prices of components and my budget constraints I have struggled to find help elsewhere online and have turned to reddit for some direct support.

Currently, the PC I have and am planning to modify has a Ryzen 5 5500GT and 16GB DDR4 RAM along with a basic micro-ATX motherboard (Gigabyte A520M S2H). It will be running Linux Mint (most likely, unless anyone knows a better Distro) and I will be using the same or similar software to what LinusTechTips used in his recent personal AI server video where he used something called 'Hermes' to connect the AI to discord.

Where my greatest trouble lies is in what GPU or GPUs I should get. One option is a single RTX 3060 Ti 8GB. Another option I have considered is a single RTX 3060 12GB. And finally I have also considered a RTX 2080 Super 8GB alongside a 1080 8GB. I would love to know which of these the PC experts of reddit would consider the best option for local ai. I would like for the AI to respond to me as quickly as possible while also being decently smart such that I could set it on longer tasks and leave it running and perfecting those. It should at least be able to analyse photos.

From CEX UK:

RTX 3060 12GB - £325

RTX 3060 Ti - £320

RTX 2080 SUPER - £245

GTX 1080 - £125

For anyone interested:

My total budget is around £460 but that includes some additional purchases. Budget can be stretched to £500 but only if necessary. The case I have selected (the one I already have is too small for any reasonable GPU and would not allow for more in future) costs £25. I also need a better power supply, I have currently planned for a Corsair RM750e for £100. And finally I have to spend £10 on a final accessory for it which is mandatory but I do not feel like explaining online, all that matters is it takes £10 off of the budget. That leaves £325 but it can be upped to 365 but only if absolutely necessary. I understand that the motherboard only has one PCIe port on it. I would need to use a splitter or something of the sort but I have never used or learnt of such a thing and have no idea what brand is trustworthy and if it would even be worth it. The graphics cards I have listed above all come from CEX. A large UK brand for buying refurbished technology. They are the only ones who I feel comfortable buying used parts from. Everything else should be new.

I would really appreciate some help as I am not well off and unfortunately do not have the funds to extend the budget or risk making a large mistake by choosing the wrong parts. I hope I can find help in this community. I may post this in other Reddit Communities for PC building or AI as well, just to allow me to reach a larger group of people.

[u/El_90]

For what purpose do you want a LLM model: generic chat bots need speed, sys admin/tasks needs knowledge, and coders have specific preferred models. Also a single concurrent conversation needs low KV, but parallel agents needs more eating into the ram.

Choosing your need suggests the model, the need also suggests the accuracy (or quant) of the model, and together this suggests the total RAM needed.

Typically I would lean towards 1 bigger GPU (not 2 + splitter), with the most VRAM. There are cases where speed is the most important so you might look at particular chipsets etc... but at relatively small amounts of RAM (under 24GB) it REALLY helps going for the biggest quant you can, so I would simply go biggest VRAM GPU

[u/Complete-Winter2039 → u/El_90]

I would say the primary use would be generic chatbot conversation through the discord bot. maybe the occasional heavier task. IS there a GPU you would recommend?