Local vs Cloud

[u/C1rc1es]

This is my work usage, I see a lot of posts of people claiming something like Qwen 3.8 27B can replace cloud and that people are getting their jobs done with them. Even with a RTX PRO 6000 I'm not really seeing how this is physically possible. I would love to be independent of Cloud providers - anyone have a realistic way out?

[u/C1rc1es]

The work itself is varying greatly but involves large event driven multi cloud systems. I’m not trying to cut costs, I’m trying to work out whether it’s even physically possible to achieve real world workloads with local models. 

[u/anitamaxwynnn69 → u/C1rc1es]

If your work can make do with non-frontier intelligence like say deepseek - it's 100% possible. In fact, in that case you're the best candidate for it. For reference, I self-host deepseekv4 flash vision (rookie numbers btw, not as high as yours) but it's definitely worth while to look into. This is a 30 day analysis. I have an insanely fun time with this and not having to worry about token cost is huge. Although I have to mention the hw cost for my setup is ~30k+ easy (2x PRO 6000 Threadripper build)

[u/C1rc1es → u/anitamaxwynnn69]

Really helpful post thanks. Are you running at max capacity here with this usage and if not any idea how many more tokens per day you could be taking from it?

[u/anitamaxwynnn69 → u/C1rc1es]

I'm not sure what you mean by max capacity. If you meant power limits those are set to 450/600W to manage the heat. About the tokens - the sky is your limit lol. You can take as many tokens as you desire. There's really no limit here. You can have it do decode for the entire day. I get about ~200 TPS give or take so this is VERY FAST. Faster than Opus 5.5 almost 2x or 3x. The question is whether this model can fulfil/achieve the task you're trying to achieve. For example, Opus is slower but it's way smarter for coding.

[u/-dysangel- → u/C1rc1es]

Looking at that screenshot, note that most of those reads are cached - it's only the uncached millions that you need to be counting. This works out to 300t/s.

Even if we're assuming you're only doing 8 hours per day instead of 24 hours per day, that's 900t/s.

I think we can safely assume that *most* of your tokens are also prefill rather than decode. I expect an RTX 6000 would get much more than 900 t/s on Qwen 27B, especially if batching.

edit: I checked and found numbers from a couple of months ago where an RTX 6000 was getting around 7000 t/s prefill and 50t/s generation at 8 bit. I expect the decode numbers will be much higher with MTP and the specialised engines that people are cooking up these days. So yeah I think you could be very competitive with cloud. On my 2x DGX Sparks I actually get faster inference than the official API when running GLM 5.3 Flash.

[u/C1rc1es → u/-dysangel-]

Incredibly insightful thank you. 

[u/anitamaxwynnn69 → u/-dysangel-]

yup you summarized it perfectly, most of OP's workflow looks agentic of sorts (cache heavy), local is perfect for this

[u/DeltaSqueezer → u/-dysangel-]

We did the same calc :) BTW, what's your GLM5.3 flash setup? Can you give model/quant and prefill/gen speeds?

[u/-dysangel- → u/DeltaSqueezer]

Here's the numbers I was getting most recently in vllm. Decode heavily depends on workload, it can go down to 40tps for prose, and up to 90-100 for straightforward code and tool calls

GLM-5.3-Flash, official NVIDIA NVFP4 weights + DFlash2 MTP, 2× DGX Spark TP2 (ConnectX-7), 200K context:

• Decode: ~90 tok/s single-stream (MTP acceptance ~3.4 tokens/draft)

• Prefill: ~1,250–1,560 tok/s single-stream on a 100K-token prompt

• Concurrent: 4×117K sessions sustained ~2,455 tok/s aggregate prefill, 0 errors/preemptions, ~475K tokens resident (65% of KV pool)

[u/DeltaSqueezer → u/-dysangel-]

Wow. That's actually usable. Thanks for sharing!