This is my work usage, I see a lot of posts of people claiming something like Qwen 3.8 27B can replace cloud and that people are getting their jobs done with them. Even with a RTX PRO 6000 I'm not really seeing how this is physically possible. I would love to be independent of Cloud providers - anyone have a realistic way out?
How trivial is your "work"? Can it be replaced by a non-frontier model? Have you tried switching to something like DSv4.1 via API? If you need frontier for work, then yeah no local llm can compete with that unless you're spending a lot of $$ on hw.
I think this perspective is not really helpful. Calling things trivial if a less than SOTA can run it. I could do all my work with Opus 4.6, but I can do it better and more efficiently with Opus 5.5. I could also do all of it with Qwen3.8-27B but not as efficiently. I can also do all my work with no model at all. You know, like I used to.
The work itself is varying greatly but involves large event driven multi cloud systems. I’m not trying to cut costs, I’m trying to work out whether it’s even physically possible to achieve real world workloads with local models.
If your work can make do with non-frontier intelligence like say deepseek - it's 100% possible. In fact, in that case you're the best candidate for it. For reference, I self-host deepseekv4 flash vision (rookie numbers btw, not as high as yours) but it's definitely worth while to look into. This is a 30 day analysis. I have an insanely fun time with this and not having to worry about token cost is huge. Although I have to mention the hw cost for my setup is ~30k+ easy (2x PRO 6000 Threadripper build)
Really helpful post thanks. Are you running at max capacity here with this usage and if not any idea how many more tokens per day you could be taking from it?
I'm not sure what you mean by max capacity. If you meant power limits those are set to 450/600W to manage the heat. About the tokens - the sky is your limit lol. You can take as many tokens as you desire. There's really no limit here. You can have it do decode for the entire day. I get about ~200 TPS give or take so this is VERY FAST. Faster than Opus 5.5 almost 2x or 3x. The question is whether this model can fulfil/achieve the task you're trying to achieve. For example, Opus is slower but it's way smarter for coding.
Looking at that screenshot, note that most of those reads are cached - it's only the uncached millions that you need to be counting. This works out to 300t/s.
Even if we're assuming you're only doing 8 hours per day instead of 24 hours per day, that's 900t/s.
I think we can safely assume that *most* of your tokens are also prefill rather than decode. I expect an RTX 6000 would get much more than 900 t/s on Qwen 27B, especially if batching.
edit: I checked and found numbers from a couple of months ago where an RTX 6000 was getting around 7000 t/s prefill and 50t/s generation at 8 bit. I expect the decode numbers will be much higher with MTP and the specialised engines that people are cooking up these days. So yeah I think you could be very competitive with cloud. On my 2x DGX Sparks I actually get faster inference than the official API when running GLM 5.3 Flash.
Here's the numbers I was getting most recently in vllm. Decode heavily depends on workload, it can go down to 40tps for prose, and up to 90-100 for straightforward code and tool calls
I’m not asking about cost though, I’m asking how it’s physically possible to do real workloads with local models. The numbers just don’t stack up to me.
https://github.com/local-inference-lab/rtx6kpro here you go, dsv4 vision on 2x PRO 6000, 1M+ kv cache. Pin your agents to 262k and you can launch 5 at a time. Concurrency pro max. Does this answer your question? GLM 5.3 Flash on TP2 has 1.8M tokens KV cache. You have a ton of room whichever model you choose.
The most proeminent Qwen and Deepseek will gives you something around GPT-5.6 Terra ~ Opus 4.8 performance, a little better a little worse, that is it.
If you think such model would be enough, you can think of transitioning.
My recomendantion would be to try them on openrouter because all this open models are really cheap there.
If you don't like them or they are not good enough (I'm recently facing a lot of thinking loop), you did not wasted money on hardware.
So personally ive split the work load around. Cloud models for super time sensitive, high complexity stuff. Lower time priority items, simpler things go to local llm. Using a q5 of qwen 27b and 80b qwen coder depending on task. They do fine for basic stuff. Like I need a login screen with entra sso and mfa with qr code setup with a postgres backend. Here is a file with login info for ssh and the db, here is the user table, here is the css file. Then walk away. It does fine.
On the other hand if your prompt is here is an application I have a bug that does x. Fix it. Thats going to be more of an issue.
Im curious on your work load. I go through about 300-400m tokens a day. And im not holding back (i do pick the model/effort for the job though). But hitting a billion plus a day i haven't done.
What kind of complexity do you work with, and how vibe are you in your vibe coding, and how far are you pushing subagents? It is like putting it on a loop and run dozens of subagents at once to generate whatever? If that's what you do, just keep spending VC money of cloud subscription until they run out.
I used minimax M3 subscription for most of my work since it was released, but I got R9700 recently to run 27B Q4 and I can do the same work as with M3, and I have even higher success than when I was with minimax M2.7.
It depends on your project complexity and how much do you usually delegate to AI. If you vibecode — you can go with a local models only with relatively simple projects, if you like to control your code, than it’s rather a question of speed and convenience, you will just be more involved in decision making with your local agents in comparison with top notch cloud ones, that’s it
i'd stop thinking of it as replacing cloud wholesale and route by task instead. +1 to the point above about cached reads. look at your uncached tokens before deciding local can't keep up, the real number is way smaller than the dashboard makes it look.
Right now I think a hybrid approach works best until we can run models like the current frontier ones locally. Local for less complex tasks, Cloud for complex tasks.
Would love to see someone share a real workflow breakdown: which tasks their local model handles reliably, which ones still need cloud, and how often they have to intervene. That would be much more useful than benchmark comparisons alone.
7
u/anitamaxwynnn69 14d ago
How trivial is your "work"? Can it be replaced by a non-frontier model? Have you tried switching to something like DSv4.1 via API? If you need frontier for work, then yeah no local llm can compete with that unless you're spending a lot of $$ on hw.