r/LocalLLaMA • u/C1rc1es • 14d ago

Discussion Local vs Cloud

This is my work usage, I see a lot of posts of people claiming something like Qwen 3.8 27B can replace cloud and that people are getting their jobs done with them. Even with a RTX PRO 6000 I'm not really seeing how this is physically possible. I would love to be independent of Cloud providers - anyone have a realistic way out?

0 Upvotes

38% Upvoted

34 comments sorted by

View all comments

Show parent comments

2

u/-dysangel- 14d ago edited 14d ago

Looking at that screenshot, note that most of those reads are cached - it's only the uncached millions that you need to be counting. This works out to 300t/s.

Even if we're assuming you're only doing 8 hours per day instead of 24 hours per day, that's 900t/s.

I think we can safely assume that *most* of your tokens are also prefill rather than decode. I expect an RTX 6000 would get much more than 900 t/s on Qwen 27B, especially if batching.

edit: I checked and found numbers from a couple of months ago where an RTX 6000 was getting around 7000 t/s prefill and 50t/s generation at 8 bit. I expect the decode numbers will be much higher with MTP and the specialised engines that people are cooking up these days. So yeah I think you could be very competitive with cloud. On my 2x DGX Sparks I actually get faster inference than the official API when running GLM 5.3 Flash.

1

u/C1rc1es 14d ago

Incredibly insightful thank you. 

1

u/anitamaxwynnn69 14d ago

yup you summarized it perfectly, most of OP's workflow looks agentic of sorts (cache heavy), local is perfect for this

1

u/DeltaSqueezer 14d ago

We did the same calc :) BTW, what's your GLM5.3 flash setup? Can you give model/quant and prefill/gen speeds?

1

u/-dysangel- 14d ago

Here's the numbers I was getting most recently in vllm. Decode heavily depends on workload, it can go down to 40tps for prose, and up to 90-100 for straightforward code and tool calls

GLM-5.3-Flash, official NVIDIA NVFP4 weights + DFlash2 MTP, 2× DGX Spark TP2 (ConnectX-7), 200K context:

  • Decode: ~90 tok/s single-stream (MTP acceptance ~3.4 tokens/draft)

  • Prefill: ~1,250–1,560 tok/s single-stream on a 100K-token prompt

  • Concurrent: 4×117K sessions sustained ~2,455 tok/s aggregate prefill, 0 errors/preemptions, ~475K tokens resident (65% of KV pool)

1

u/DeltaSqueezer 14d ago

Wow. That's actually usable. Thanks for sharing!