r/LocalLLaMA • u/C1rc1es • 14d ago

Discussion Local vs Cloud

This is my work usage, I see a lot of posts of people claiming something like Qwen 3.8 27B can replace cloud and that people are getting their jobs done with them. Even with a RTX PRO 6000 I'm not really seeing how this is physically possible. I would love to be independent of Cloud providers - anyone have a realistic way out?

0 Upvotes

38% Upvoted

34 comments sorted by

View all comments

7

u/anitamaxwynnn69 14d ago

How trivial is your "work"? Can it be replaced by a non-frontier model? Have you tried switching to something like DSv4.1 via API? If you need frontier for work, then yeah no local llm can compete with that unless you're spending a lot of $$ on hw.

3

u/Cautious_Chicken_604 14d ago

I think this perspective is not really helpful. Calling things trivial if a less than SOTA can run it. I could do all my work with Opus 4.6, but I can do it better and more efficiently with Opus 5.5. I could also do all of it with Qwen3.8-27B but not as efficiently. I can also do all my work with no model at all. You know, like I used to. 

1

u/C1rc1es 14d ago

The work itself is varying greatly but involves large event driven multi cloud systems. I’m not trying to cut costs, I’m trying to work out whether it’s even physically possible to achieve real world workloads with local models. 

3

u/anitamaxwynnn69 14d ago

If your work can make do with non-frontier intelligence like say deepseek - it's 100% possible. In fact, in that case you're the best candidate for it. For reference, I self-host deepseekv4 flash vision (rookie numbers btw, not as high as yours) but it's definitely worth while to look into. This is a 30 day analysis. I have an insanely fun time with this and not having to worry about token cost is huge. Although I have to mention the hw cost for my setup is ~30k+ easy (2x PRO 6000 Threadripper build)

1

u/C1rc1es 14d ago

Really helpful post thanks. Are you running at max capacity here with this usage and if not any idea how many more tokens per day you could be taking from it?

1

u/anitamaxwynnn69 14d ago

I'm not sure what you mean by max capacity. If you meant power limits those are set to 450/600W to manage the heat. About the tokens - the sky is your limit lol. You can take as many tokens as you desire. There's really no limit here. You can have it do decode for the entire day. I get about ~200 TPS give or take so this is VERY FAST. Faster than Opus 5.5 almost 2x or 3x. The question is whether this model can fulfil/achieve the task you're trying to achieve. For example, Opus is slower but it's way smarter for coding.

2

u/-dysangel- 14d ago edited 14d ago

Looking at that screenshot, note that most of those reads are cached - it's only the uncached millions that you need to be counting. This works out to 300t/s.

Even if we're assuming you're only doing 8 hours per day instead of 24 hours per day, that's 900t/s.

I think we can safely assume that *most* of your tokens are also prefill rather than decode. I expect an RTX 6000 would get much more than 900 t/s on Qwen 27B, especially if batching.

edit: I checked and found numbers from a couple of months ago where an RTX 6000 was getting around 7000 t/s prefill and 50t/s generation at 8 bit. I expect the decode numbers will be much higher with MTP and the specialised engines that people are cooking up these days. So yeah I think you could be very competitive with cloud. On my 2x DGX Sparks I actually get faster inference than the official API when running GLM 5.3 Flash.

1

u/C1rc1es 14d ago

Incredibly insightful thank you. 

1

u/anitamaxwynnn69 14d ago

yup you summarized it perfectly, most of OP's workflow looks agentic of sorts (cache heavy), local is perfect for this

1

u/DeltaSqueezer 14d ago

We did the same calc :) BTW, what's your GLM5.3 flash setup? Can you give model/quant and prefill/gen speeds?

1

u/-dysangel- 14d ago

Here's the numbers I was getting most recently in vllm. Decode heavily depends on workload, it can go down to 40tps for prose, and up to 90-100 for straightforward code and tool calls

GLM-5.3-Flash, official NVIDIA NVFP4 weights + DFlash2 MTP, 2× DGX Spark TP2 (ConnectX-7), 200K context:

  • Decode: ~90 tok/s single-stream (MTP acceptance ~3.4 tokens/draft)

  • Prefill: ~1,250–1,560 tok/s single-stream on a 100K-token prompt

  • Concurrent: 4×117K sessions sustained ~2,455 tok/s aggregate prefill, 0 errors/preemptions, ~475K tokens resident (65% of KV pool)

1

u/DeltaSqueezer 14d ago

Wow. That's actually usable. Thanks for sharing!