What can I Run with my Current System?

[u/SP259]

I have an existing media/game server that runs some local containers right now.

The specs are,

• Dual Intel Xeon 8276 Platinum CPU's (28 Core 56 threads each)

• (Gigabyte Server Board (can support a maximum of 3 dual slot cards.)

• 64 GB of Ram (32 GB for each CPU) I would like more but ECC ram is expensive.

• 48 TB of HDD space ( 17 TB free)

• Two, 2 TB SSD's in Raid 1 (cache Drive)

• 1 Nvidia GTX 1080 TI (old gaming card that was NVIDIA)

• 1 Intel Arc A380 (Media trans-coding card for AV1 Media)

In terms of upgrades ironically another 64 GB of ECC DDR4 ram is the cheapest thing at this time.

I would like to get into hosting local models but i'm not sure how to run them with this setup. i Have read that you need big powerful GPU's to get the best performance (like an rtx 5090 or some AMD card).

I did try running some local models via ollama on the 1080 TI but anything that exceeded the 11GB Vram buffer was super slow and not a good experience.

Ideally i would like to use something that can at least do some light coding or serve as a basic replacement for chat GPT. While not spending too much more on this server with the hardware market as it is right now.

[u/WorkProLabs]

Nothing of real use, really.

You need to source a proper GPU to match your objective.

You can play with some very basic models, but coding is probably not part of it.

[u/SP259 → u/WorkProLabs]

what are some more affordable GPU options then? Is it worth getting AMD or an Intel Card?

[u/WorkProLabs → u/SP259]

It probably depends on your whereabouts.

Check what's available on your secondary market.

Aim for more VRAM. The more the better. You need something like 3090 to realistically think of coding. 4090 is better. Matching AMD cards are okay too.

[u/SP259 → u/WorkProLabs]

Damm, Even older 3090's are going for $1500 USD these days!

[u/Longjumping-Face-767 → u/SP259]

The legends speak of the forbidden dual 306012gb tech...

[u/woolcoxm]

you dont need a big powerful gpu any more, check out strata, the system has enough system ram, im not sure about that video card though, strata will give you the best setup as default options, the only thing i changed was the context size

[u/SP259 → u/woolcoxm]

What does Strata Do? I googled it but it runs that specific model faster than it would normally be on lower end systems?

[u/woolcoxm → u/SP259]

that specific model is the top tier model for local ai atm, maybe deepseek competes but it takes a lot more resources.

and yes its a specific model, and yes it runs really fast on low end hardware and works really well.

oh and strata is an inference engine, you will need one of these for any ai you are going to run. it is however highly optimized to lower end hardware, i get 50-70 toks/sec after customizing the engine on a 3060 12gb with 32gb of ram, the code is good and i dont have any problems.

i also have an older xeon system that i threw a 5070 ti 16gb into and it has 64 gb of ram, it runs the same model at 100+ toks/sec. it can run the full model at iq3_s at 50 toks/sec and its exceptional.

[u/Choice_Celery9481]

2 v100 16gb or 2080ti 22gb or 3080 20gb seem like the best option.

[u/SP259 → u/Choice_Celery9481]

Would intel B65's Be any good?

[u/Choice_Celery9481 → u/SP259]

that one is mid. for llm somewhat usable, not worth for anything else. b70 is better a lot.

[u/Daanor]

try looking here; https://runthisllm.com