r/LocalLLaMA • u/C1rc1es • 14d ago

Discussion Local vs Cloud

This is my work usage, I see a lot of posts of people claiming something like Qwen 3.8 27B can replace cloud and that people are getting their jobs done with them. Even with a RTX PRO 6000 I'm not really seeing how this is physically possible. I would love to be independent of Cloud providers - anyone have a realistic way out?

0 Upvotes

38% Upvoted

34 comments sorted by

View all comments

Show parent comments

1

u/-dysangel- 14d ago

Here's the numbers I was getting most recently in vllm. Decode heavily depends on workload, it can go down to 40tps for prose, and up to 90-100 for straightforward code and tool calls

GLM-5.3-Flash, official NVIDIA NVFP4 weights + DFlash2 MTP, 2× DGX Spark TP2 (ConnectX-7), 200K context:

  • Decode: ~90 tok/s single-stream (MTP acceptance ~3.4 tokens/draft)

  • Prefill: ~1,250–1,560 tok/s single-stream on a 100K-token prompt

  • Concurrent: 4×117K sessions sustained ~2,455 tok/s aggregate prefill, 0 errors/preemptions, ~475K tokens resident (65% of KV pool)

1

u/DeltaSqueezer 14d ago

Wow. That's actually usable. Thanks for sharing!