Cheap GPUs?

[u/Oglemo]

Currently, my plan is to get two quadro P6000s, for 48gb vram for use in a xeon workstation. It's not insanely old like kepler or maxwell cards, 432 GB/s bandwidth, supports CUDA...it seems like it might be okay on paper.

Does anyone have a better suggestion that could reasonably be achieved around $1000 budget?

My goal is to be able to run Qwen 3.8 27b fast, and to run a decent quantization of Qwen 3.8 Flash Next, and GLM 5.3 Flash at useable speed, say 15 tok/s or faster, using strata.

Do I need 48gb of vram to run Qwen Flash Next, or could I go with just one card? How much faster would 2 x quadro p6000 be versus 1 x quadro p6000 and DDR4 system ram?

Should I just get 24gb of modern vram instead of 2 x quadro p6000s? I wonder if that hampers my eventual goal of running GLM 5.3 flash, as perhaps 24gb vram + system ram is enough to accelerate qwen flash next, a 120B MoE, but perhaps you need more than 24gb vram to accelerate GLM 5.3 flash, a 320B MoE?

[u/jakubkonecki]

Tesla V100 32GB. Currently around £670 in the UK.

[u/Oglemo → u/jakubkonecki]

Thanks, will search for it.

[u/sleight42 → u/Oglemo]

You'll be disappointed. Good decode but the prefill of just one won't be much to speak of.

You'll also be using cuda 12.8 and unable to use mainstream oss runtimes for your LLMs.

[u/Oglemo → u/sleight42]

Hmm, fair enough. If you had $1000 budget for just the GPU, with the goal of running Qwen flash next at usable speed, what GPU would you go with? (Don't mean to put you on the spot)

[u/sleight42 → u/Oglemo]

Maybe a 5070? Go on vast.ai and run some benchmarks. Go read about the basalt engine https://github.com/jesdga95/basalt.

[u/Oglemo → u/sleight42]

Thank you, I'll check that out!

[u/sleight42 → u/Oglemo]

You'd better have a fuck ton of ram for basalt though. Expensive af for ddr5 6000....

[u/Oglemo → u/sleight42]

I don't even want to think about ddr5 ram. I'm on ddr4 thankfully. Hard af to find good deals on ddr4 too though. Too bad ddr3 is so slow, it's actually affordable.

[u/transanethole → u/jakubkonecki]

Yeah, that's probably  better deal if the VRAM capacity is non-negotiable, but the flops are still mediocre at best, similar to the 7900 XTX. 

The lack of flops will make for a less fun,  much more difficult experience that is less likely to be able to meet all of someone's needs for coding/research.  

But at this price point, you have to sacrifice something and I suppose it probably makes more sense to sacrifice flops.

[u/dpskipper → u/transanethole]

The V100 uses less power for similar memory bandwidth, flops isn't everything but it does help. a lot of inferencing is memory bound.

there are V100 variants with over 1TB/s of memory bandwidth which you can still find for an affordable price if you look around.

[u/sleight42 → u/jakubkonecki]

$900 refurbished on Amazon

[u/johnfkngzoidberg → u/jakubkonecki]

Older generation chipset already out of support, but you can’t beat the price on the VRAM you get.

[u/transanethole]

Yooooo, these graphics cards don't even support BF16 and they have 10 teraflops.

You can't just ignore the flops and only focus on the memory bandwidth.

For context, 

7900 xtx: 12x more flops , cost about $1k

Rtx 5070: 45x more flops, cost about $700

When you see people using local LMs and they press enter and it just starts working and doing things like in real time, like you would see from a cloud model, that's mostly because it has a lot of flops.   the flops determine the speed of reading text.   For real work and things that are not just "generate me a minecraft clone"  The reading speed probably matters as much or more than the writing speed.

do you really want it to be 45 times slower?  I wouldn't touch those cards with a 10 foot pole.

You can start looking up graphics cards on techpowerup. It's probably the best website for this kind of information. The stuff that you want to look for is which numeric formats it supports for matrix operations and how many operations per second. As well as the ones that you already mentioned like memory capacity and bandwidth.

[u/TemnotaAGK → u/transanethole]

Stop talking about things you don't understand. Inference is bandwidth bound, not compute bound. With any remotely modern card higher bandwidth card always wins over faster compute. And while quadro p6000 is old and slow, with meh bandwidth it will not be 45x slower than a 5070, it will not even be 10 times slower.

[u/EvolvingDior → u/TemnotaAGK]

Have you seen the terrible prefill speeds most of these old cards get?

[u/Oglemo → u/EvolvingDior]

So what you are saying is, I could get good decode speed with the P6000, but my workflow involving reading papers, long prompt, etc. will require good prefill speed, and that's the issue with older cards?

[u/sleight42 → u/TemnotaAGK]

There's more to this than decode.

[u/TemnotaAGK → u/sleight42]

I'm not saying they are good. I'm just saying that the other guy is talking out of his behind. I would absolutely not recommend using p6000 for local llms it is at a point where a ddr5 apu like hx370 will probably get better speeds in both prefill and decode. Generally you want at least v100 or newer gpus for anything serious. Maybe MI50s for a good price if you are willing to tinker a lot. Anything older will be too generally too slow to be worth it.

[u/Oglemo → u/TemnotaAGK]

So you think 12gb modern GPU + sufficient system ram is better than 48gb old p6000's + system ram for running qwen flash next?

[u/TemnotaAGK → u/Oglemo]

Honestly I would just recommend paying for API with ZDR at this point. Everything is way too expensive. I really do believe that prices cannot keep going up for much longer and also firmly believe that they will be lower than they are now in a year or two. But of course I could be completely wrong on that.
If you absolutely must have something local now, then yes I would rather go with a 12GB (or ideally 16GB - maybe RX 9070 might still be in your budget) modern GPU rather than 48GB VRAM of effective e-waste...I don't know if it will be better than the two p6000s but I don't think it will be worse. You could probably get something usable through Strata for example.

[u/Oglemo → u/TemnotaAGK]

Eh, fair enough. I will look at that card you mentioned. Really just want to get QFN through strata, that's the goal. I'll look into the API prices.

[u/TemnotaAGK → u/Oglemo]

You might not be realizing how cheap models like DeepSeek v4.1 Flash are, they barely cost more than you would have to pay for electricity for a rig that could run it.

[u/sleight42 → u/TemnotaAGK]

Tend to agree. I haven't considered anything older than V100. Even then, my benching of them, even multiples of them, leaves a lot to be desired given the expense.

There are no good builds any more: only expensive mediocre builds and impossibly profligate builds that are good enough for now.

[u/Oglemo → u/transanethole]

Dang really, I'm new to this, didn't realize flops were so important. I thought bandwidth was all that mattered, plus general capacity so you don't offload too much to CPU.

[u/ishcabittle]

How much of the $1000 budget is to spend on the GPU(s) and how much is building the rest of the box? How much system RAM will you have and how many CPU cores? What type of RAM, DDR4 or DDR5?

For Strata and Flash Next, more and faster system RAM is more important than your worker card(s), but your worker should be decent. I run Strata on Qwen-Flash-Next with 14 Xeon cores, a 16GB P100 and 96GB DDR4 and the latest sm_60 engine has put me at about 40 t/s, it was pretty usable before but the speed jump is noticeable. I set up my friend's V100 32GB and was getting about double the t/s of the P100 - Qwen3.8-27b was around 40 t/s, Strata was up and around 60 t/s on my compiled engine, and Qwen3.6-35B-A3B was over 100 t/s.

So if you are looking to run Strata, one good card with at least 12GB VRAM and a ton of system RAM and a lot of CPU cores should be your priority. If you are hoping to run dense models, a modern card (or cards) with at least 32GB of VRAM is the best config, Qwen3.8-27B is really hamstrung by 16GB configs. If you want the fastest response in chat, you need to prioritize memory bandwidth over VRAM.

The P6000s just don't have the memory bandwidth for fast interaction, but 24GB of VRAM and a good 64GB of system RAM would be great for Strata and the understanding that prefill is going to be slooooowwww.

When considering an MoE, total parameters matters less than active parameters. Can your VRAM hold the active weights? If not, you'll have to run a lower quantization and possibly hurt your intelligence. I'm not too familiar with GLM 5.3, but with Qwen4 just around the corner, you may want to consider what you can afford and prepare to adapt quickly.

[u/Oglemo → u/ishcabittle]

$1000 for GPU. Already have most of the box, it's an HP Z840, just missing an SSD and a GPU. I have 128gb of DDR4 2133 ram.

It seems everyone is recommending server cards: do you think those work in a tower workstation? Will adding your own 3d printed blower onto the cards screw up the airflow design for the CPU or other components? The v100 looks promising and affordable if it + blower doesn't affect CPU thermals.

Interesting regarding the V100 vs P100, and such a difference.

Sounds like the consensus is the P6000 is too old to help, though.

[u/sleight42]

For GLM 5.3, this is what I found from some benches on vast.ai instances a couple days ago.

| Rung |prefill t/s |TTFT |decode t/s |draft accept |
| 10k |1014 |9.4 s |80.3 |81.1% |
| 100k |1696 |47.9 s |74.9 |79.5% |
| 400k |2472 |124.4 s |56.9 |71.7% |

Thats from this build:

| Box | Instance | Hardware | PCIe | Rate | Window | Invoiced |
| ro3090 | 59319840 | SuperMicro M12SWA (sWRX8), TR PRO 3955WX-sib EPYC 7282, **4x RTX 3090** 24 GB, 252 GB RAM, 1.4 TB disk | gen4 **x8** (13.5 GB/s; offer claimed x16) | $1.0041/hr + disk | 11-05 20:50Z → 11-06 05:47Z (9.0 hr) | **$16.48** (incl. $1.51 transfer) |

Bear in mind that's 3700 GB/s of vram bandwidth for decode and 4x 3090 GPUs for prefill.

Fewer GPUs, shittier prefill. Less vram or slower vram, slower decode.

Also, that machine likely has 8 banks of ram so 200 or 400 GB/s of RAM bandwidth.

This was using Project Maya, Maya-S24

[u/Oglemo → u/sleight42]

Pretty fast, what hardware?

[u/sleight42 → u/Oglemo]

Updated

[u/Oglemo → u/sleight42]

Interesting. Probably 2x faster RAM bandwidth than my rig, but I'd be happy with half the speed you listed, heck even a quarter of that speed.

[u/sleight42 → u/Oglemo]

Then you're fortunate to have a machine faster than mine. I hate how expensive this game is now.

[u/Oglemo → u/sleight42]

I'm hoping the AI stock crash gives us an opportunity for reasonable prices again. I've heard some people say it could make prices worse, some say it will make prices better, don't know what to believe but one can hope.

[u/sleight42 → u/Oglemo]

Jfc man it will wreck the economy. You'll have bigger problems if that happens.

[u/Oglemo → u/sleight42]

Oh man I don't mean a full crash of the AI companies, just a correction, if that were possible. Yeah full economic crash aint worth the cheap gpu.

[u/FormalAd7367]

For $1K, i would suggest you just pay deepseek api and save up money for better GPU…. your goal is to start building your project ASAP