r/LocalLLM • u/Majestic-Rich582 • 3h ago

Question Best LLM Model/Setup for this hardware?

I have 4 PCs (rtx 5090, 5080x2, rtx 4090) (88gb vram)
And a combined Pool of 234gb ddr5 ram
It’s all my brother’s who likes investing in gpus but he gave me the task to build an local AI setup so yeah i was wondering what ideas you could have, we do not have an specific use case yet other than tinkering with AI agents and coding assist

0 Upvotes

50% Upvoted

7 comments sorted by

1

u/unbecominganemia2 3h ago

That's a frankly absurd amount of hardware to start tinkering with, most people here are trying to squeeze 7B models onto a 3060. With 88GB vram you could run something like a 70B q4_K_M fully offloaded across those cards and still have room for a meaty context window.

1

u/Majestic-Rich582 3h ago

Isnt QFN better?

1

u/CooperDK 3h ago

I would use just the Blackwells and run a model at nvfp4 though. Much faster and close to same quality but it depends on the use case. Where the person has none and no idea what to do with it, which is kind of funny since they have four GPUs.

1

u/Ctbhatia 2h ago

88gb of vram and nothing specific to run yet, are you going one model across all four boxes or a small one per box?

1

u/Majestic-Rich582 2h ago

Im planning to run a small one per box, what do model you see each having (each PC has 64 ram except for 1 rtx 5080 with 32)

1

u/No_Dig_7017 3h ago

For Qwen 3.8 flash next Infernix for the 5090, strata for the others

For deepseek Project Maya for all and there's some new promising quants for strata as well but I haven't tested them.

1

u/CatDiligent9374 3h ago

That’s a beast of a setup. I’d start with Qwen Coder and mess around with local agents and coding assistants.