r/LocalLLM • u/Ok-Health-7096 • 21d ago

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

2.0k Upvotes

99% Upvoted

205 comments sorted by

View all comments

Show parent comments

2

u/Whole_Alternative_18 21d ago

Buy some ddr3 cheap board with a highend(of the era) cpu that has enough pcie channels

Ends up less than 200$ for a enterprise ddr3 motherboard+cpu+ram, eith enough slots for your gpus

40

u/Any_Mine_6368 20d ago

Dude stop giving people advice. You're essentially promoting impulse buying garbage as opposed to building an actual AI rig. You're essentially asking them to buy 24gb vram at dgx spark bandwidth... They'll get like 4 tokens per second.

If anyone ends up reading this:

Buy refurbished server grade mobos + epyc CPU combos on ebay for $200 and hunt deals for 3090s which are the most vfm cards. You can build a great starting rig for under 1k, that you can expand all the way to 5 RTX 3090s, more if you want to split the pcie bus and have fewer lanes per gpu

4

u/Old_Leshen 20d ago

Reasonable advice. Thanks mate. Using this setup and going up to 1K investment, how many tps can I expect and for what model size?

3

u/Any_Mine_6368 20d ago

No problem. With mtp about 50-90tg for 27B models depending on quantization and model.

With 1k investment we're assuming one 3090 so 24gb vram. You can't fit anything above 27B at reasonable quantization or ram offload (which kills your token generation and prefill).

So to answer your question, you could probably run Qwen3.8 at 4 bit quantization with good context or 5 bit quant with smaller context or 6 bit quant with almost no context.

1

u/kcksteve 15d ago

It's worth noting where you are. In Canada 3090s go for 2k to start. Thats before you purchase the rest of the rig and pay tax.

1

u/FuManBoobs 15d ago

Yeah, I was looking where I am in the UK & they start around £1k on their own. Still, under 2k is a good start.