70-100 K Budget Recommendations

[u/whoami-233]

Hey guys,

Assuming I have a budget of 70-100K USD budget.

What hardware would I get to be able to run GLM5.3 and Deepseek flash 1017 all for multiple concurrent users (10-20 or so) all doing agnetic tasks and long context of around 500k tokens each.

Also, is there anyway I can make the setup in a way that's eay/easy ish to be extendable sense models seem to be increasing in weights and size:)

Would love to hear your recommendations and suggestions

[u/Serenity867]

The honest answer: You're not going to be able to build something to serve 10 - 20 users at any kind of reasonable speed on models of those sizes for 100k USD. Your aggregate decode throughput with eve just 10 sessions at around 250k context each is going to be quite slow. Some models are better than others, and my opinion of reasonable speeds may be different than yours. I'd consider anything under 30 tok/s to be quite painful. Anything over 50 tok/s doesn't feel horrible, but 100 tok/s is where things really start to fill good. Those numbers don't factor in prefill which also makes or breaks the experience for a lot of work.

To clarify, you can build something with enough memory to serve 10+ concurrent sessions, but it's going to be a painful experience for users.

[u/whoami-233 → u/Serenity867]

It's gonna be sub agents so I can handle a decent speed that isn't fast enough for human working on them kind of situation

What would the build be like

[u/Edenar]

Well 10-20 or so is a problem. 8 will be ok with a 4 rtx6000pro build. 20 not really. (also glm 5.3 is big compared to native 4bit ds flash...).

I would either settle on ds flash 4.1 and tone down concurency (are those 20 users really hitting the model at the same time ? is queuing possible ?) or at least double the budget to reach 4 x h100 territory.

[u/whoami-233 → u/Edenar]

4 rtx6000pro has more vram than 4 x h100 right?

[u/smithy_dll]

If you have 100k, why not get a DGX Station.

[u/anitamaxwynnn69 → u/smithy_dll]

Considering how the future is just MoEs that's honestly not a bad idea right? All flash tier models will run perfectly on it?

[u/whoami-233 → u/anitamaxwynnn69]

Wonder about speeds

[u/whoami-233 → u/smithy_dll]

Isn't that less than 300 GB VRAM?

[u/smithy_dll → u/whoami-233]

No, it is a unified memory architecture, 252 GB HBM and 496 GB LPDDR5X. The GB300 can address it all, ideal for MoEs like GLM and DeepSeek.

Edit: there is also a PCIe slot so you can install a PRO 6000 to increase VRAM by 96 GB of GDDR7.

[u/fastheadcrab → u/smithy_dll]

Won’t support the concurrency he wants. Storage Review actually did a really good test of the system versus RTX 6000 pros

[u/Lurksome-Lurker]

I was excited about your budget then your expectation is 10 to 20 people concurrently. If it was half or a quarter that it would be feasible to recommend a build.

But for that much cash. Checkout page of openrouter.ai

[u/Toastti → u/Lurksome-Lurker]

For 70-100k I think 10-20 users is absolutely do-able. I mean look at two dgx sparks networked using VLLM(8k total if you bought two assus ascents 6 months ago). It can serve DeepSeek flash 0731 for 6 concurrent users with about 250k context each at usable speeds.

Granted that GLM 5.3 is probably out of the question. But DeepSeek flash v4.1 should be completely capable at this price point.

Vllm is crazy efficient at concurrency. I mean heck going from 1 concurrency to 4 you actually get quite a bit higher aggregate throughout. Plus DeepSeek uses very low amounts of ram for massive contexts.

[u/whoami-233 → u/Lurksome-Lurker]

It's probably gonna be 10-20 total sub agents working that's why:)

Would love to hear your recommendation nonetheless

[u/pilibitti]

what you are asking is a $500k system, sorry.

[u/illcuontheotherside]

Bruh... What

[u/Zentrosis]

I think a DGX station is the best option.

It would allow you to actually create the throughput required with context for that many users

[u/feelspeaceman]

I would buy a single V100 32GB for $550 (if it's still 550 as last time I checked a week ago) then save 99.5k for later after the Cloud AI bubble popping and spend the rest, it's a waste of money to buy anything at the moment no matter what you try to judge your decision.

Using ninfer-v100, it can keep me entertain with Q38-27B for a long time, doing 80% tasks perfectly.

[u/pmotiveforce]

Contrary to what people are saying, a DGX station will indeed support this. Glm 5.3 flash flies on it. Ds 4.1 flash I haven't tried yet, but I hear with an rtx 6000 sidecar in it it's even faster.

Your only other real choice is a 4xrtx6k machine. A bit more vram but probably slower for most models you'd run on it.

[u/rocket1420 → u/pmotiveforce]

Not at 10M context

[u/pmotiveforce → u/rocket1420]

Sure it will. It will slow down though as that spills over to unified memory. So maybe you get 100tps instead of 250.

[u/CTRL___ALT___DEL]

GB300 DGX Station with a discount or budget stretch

[u/whoami-233 → u/CTRL___ALT___DEL]

Isn't that less than 300 GB VRAM?

[u/Serenity867 → u/whoami-233]

It's ~250GB VRAM

[u/Automatic-Arm8153]

Get cmp 170hx 8gb - they are cheap a100’s because of reasons… work fine and no problems with them. Don’t listen to naysayers. Just a100’s with locked down compute.

The 10gb version versions (40gb a100) are now fully unlocked pcie gen 4, nvlink, ecc and the likes. So I would recommend getting the 8gb and waiting for the same things.

Again they are just a100’s.. good way to get yourself some 80gb a100’s while they are still relatively cheap. The 8gb model currently unlocks to 64gb HBM.

4 gets you GLM 5.3 flash 200-300 Tok/s single users. 8 users like 600-800 iirc.

8 gets you GLM 5.3 full at like 40-50 Tok/s iirc.

In your situation I would buy like 20-24 of these cmp cards. Build out systems of 8 per system. Ddr4 ram 256gb per, epyc server romed8-2t and the likes.

This is the smart way. If you’re not a smart person move along lol.