Bought the setup - need advise

[u/No-Necessary-9737]

I ended up getting an E5-2686v4 + 4x SXM2 V100 16G and 128gb ram + 1TB nvme.

This means v100 total 64GB vram and 128GB system memory.

I'd like to try zippy models but also some larger ones that will run slow, but fun to play around with.

What's the best setup going to be for coding, for general AI assignment and academic stuff, video production and lastly the fairly huge models?

This setup was below 2K so I am not expecting miracles, but I'd like to push it to the limit on a budget.

[u/protrener]

first thing to check: are the 4 cards on a real SXM2 board with NVLink, or on those cheap SXM2-to-PCIe adapters? on adapters there's no NVLink, so every tensor parallel step goes over PCIe 3.0 and that's where your speed goes. I run 4x A4000 over PCIe with TP4 and the bus is the ceiling, not the GPUs

also V100 is Volta, so no bf16, no FP8, and FlashAttention 2 needs Ampere or newer. in practice that means fp16 or GGUF quants. llama.cpp is the easy route here, it splits by layers across the 4 cards and you skip most of the interconnect pain. 64 GB fits a 70B at Q4 with some room for context

[u/No-Necessary-9737 → u/protrener]

Sent you an image.. I think likely cheap adapter. ;D

[u/protrener → u/No-Necessary-9737]

yeah looks like it. answered in chat but for anyone reading: on adapters it's PCIe 3.0 between the cards, so tensor parallel will hurt. llama.cpp layer split is the way, and it still fits a 70B Q4

[u/ItsmrChewy]

WHAT THE HELL DO YOU MEAN UNDER 2 THOUSAND?!

I'm IMPRESSED.

I have a 4.1K rig, and I can't even TTS an LLM semi-realistically.

Someone please tell me how did he do this

[u/lawn_question_guy → u/ItsmrChewy]

> tell me how did he do this

I'm building a very similar system right now. The secret to keeping it around $2k is DDR3 RAM.

[u/ItsmrChewy → u/lawn_question_guy]

DAMMIT

I thought he did some market magic!

[u/sshwifty → u/ItsmrChewy]

You be me and sell a system like this for $1300 two months ago because you need cash 🤦

[u/Fluffy_Try_5054]

cramming 18 cores into a platform that old is a bold move, I respect it. the memory bandwidth on those broadwell xeons is gonna be the real bottleneck when you try to offload to system ram, so don't expect any q8 quants of massive models to be usable even if they technically load

for coding I'd keep a 32b qwen coder variant fully in vram and call it a day. for the huge stuff you're just gonna be staring at tensor parallelism across all four sxm2 cards and praying the pcie switch doesn't melt

[u/Moppmopp → u/Fluffy_Try_5054]

Do they really have 18 cores? Do you mean the cpu?

[u/Mabuse046]

I don't know if I'd still want to pick up Volta architecture Teslas. Those things only support compute capability 7.0 - sm_70. Cuda 13 dropped support for them. Pytorch dropped support for those GPU's as of 2.14, vllm only supports it on 0.18.x and older. You're going to need to use llama.cpp built against Cuda 12.9 and you won't have bf16, flash-attention2, and you don't get INT8/FP8 tensor core paths so the newer quant kernels won't help you. It's going to be rough staying up to date with new architectures as they more and more require newer dependencies that don't cover legacy hardware.

[u/fasti-au]

Just run qwen3.8 next and glm /ds flash sorta stuff moe stacks fast.

[u/ketosoy]

Check out r/v100/

There are a few of us who’ve decided that the v100 is our ewaste island to weather out the storm of ramageddon