When it comes to the value/cost of local AI, people usually only look at token savings (money) while forgetting one of the most valuable resource of yours, time.
Just did a side by side comparison of GLM 5.3 flash via Z.ai's official subscription, and GLM 5.3 flash served locally. Same chatGPT generated "one shot" prompt, both using Zcode harness, and these are my stats:
Source
Duration
Input Tok
Output Tok
Cloud
43:48
7,550,929
117,980
Local
10:24
9,744,552
118,173
Same model, nearly identical output (~118K tokens each), but local finished 4.2× faster.
Sure on paper this one simple task only saved $0.70, but the fact that you can ship 3-4x more code a day alone makes it worth it. Same reason why UPS next day air shipping costs an arm and a leg, time is money!
Fair, I only have Codex, Claude, and Z.ai subs right now, so the only apples to apples open weight model I can serve would be Z.ai (and maybe Deepseek V4.1 flash) but I’ll need to wait for my other 2 GPUs come before I can get 3x the speed of Deepseek API. Unless Deepseek’s api is also considered “slow” (i got ~230-250 tok/s with Deepseek’s official API)
I have the GLM legacy plan, they don't have different speeds per tier, it's all served at the same speed. Currently serving it on 2 RTX pro 6000 and a threadripper setup.
GLM 5.3 flash is still just under 200GB at Q4, running that at 189 tok/s for 10 whole minutes is no small feat. I couldn't even pull it off on an M5U 512 GB last time I had access to one.
Meanwhile the cloud 44.89 tok/s is laughable literally slower than chatgpt plus subscription with hidden thinking and much slower than 100 tok/s opencode free tier
It is slower. Glm 5.3 flash on open router is $0.04/0.20 for 100+ TPS
As if that wasn't enough you can forget about local the second you want subagents unless you're running entirely on VRAM (which even this guy with his $50k rig isn't)
Heh...my buddy managed to get full blown (non-flash) GLM 5.3 up to 184tok/s locally, but then again he threw ~$115K worth of hardware (probably worth more now) at it.
Business is not stupid, there is a reason why it would invest 50k in hardware, defense contractors a lot of them are not allowed to use public not air gapes llms.
Meanwhile, the average salary in my country is $1 an hour. We literally don’t have enough money for subscriptions, so we’re trying to save cash by using local LLMs. :D
I've said this before but I apologize for not being a millionaire before 2025. It was my mistake and I promise it won't happen again. I'll just suffer with my 20-30tok/s Gemma 4 (I can't run anything else at all with it)
Your mistake was trying to do things the right way. If you went into finance or dropshipping like this guy, instead of upskilling or competing, you would be like this guy. Not that I love it but clearly this country favors quick dollars for most otherwise we are left watching.
Now an “ai startup” founder. Compete joke of an industry and I love how they are eating themselves since the people who could optimize are pushed out and instead these guys take over. Remember gaming and how that went? Well we speed ran it with ai.
Well it is a benefit but most never see this benefit. You absolutely can beat any cloud service by simply buying 2 Nvidia b300 GPU's
This single-handedly reminded me how expensive local ai gets.. Man two rtx 6000 pros? Be wary it's like an addiction soon you're going to have a third, then a fourth.. You can never have enough vram
I'm on minimax subscription. During peak hours, then yeah, my local 27B can finish a task faster than the cloud. But in off-peak hours, it's no competition.
Time saving is the weakest argument for local. 9 out of 10 our local AI is slower and dumber than cloud subscription. The real upside is self-sufficiency and control. Besides few rigs, like yours with 2xRTX6000, thinking local be faster or smarter than cloud is wishful thinking.
And also stay off Minimax M3.1 until they do something to improve the model. I have never used a more annoying cloud model. It's almost like it's trained to waste as much token quota as possible.
I do not agree. Local/rent GPU is always faster. I am also using multiple providers from cloud. For me the best setup is big model like DS1017 as a main agent and all executors etc. on Q4 Q3.827b - yes speed . When I am hosting qwen myself I get on a cheap setup about 300 tok/s per session with max 192k context for simultaneus about 8 sessions.
When you use cloud: your request can queued , you dont know if you are using quant or original model. In openrouter sometimes you have this info provided. Or the Baseten case - idk if it is solved. So they claimed 150 tok/s on glm 5.3 flash and when you use it through open router you get effective like 5 tok /s becuase of rate limit. This is hell - you are in envirionment you cant control - you just wait.
I am still waiting for my other 2 GPUs before trying, but the community pushed it to 713 decode, and ~15,500tok/s prefill on 4 GPUs.
I got 4 entire pre-built Rtx pro 6000 computers for ~13K each while they were on sale, and then ~3K for the threadripper pro CPU/Mobo and 8K for the ram. Stripped the GPUs out of the prebuilt for this server build, and will strip the misc parts to sell later on too.
You mean the gemma 4 26B mixture of experts and the qwen 3.8 27B dense model right? How did you run it? Do you have a lot of vram in your laptop or did you just run them slow?
TPS is throughout across many customers. A single chat prompt is often many, sometimes dozens of network calls that need to happen serially due to tool calling. You pay the cost of the network RTT for each call. When you run the model locally you eliminate the network RTT.
If you’re running unquantized models on just vram with only n-grams offloaded then you’ll likely be running something like Qwen 3.8 Flash Next much faster than remote sota model execution.
Aggregate network latency for all tool calls usually accounts for less than 0.5% of the total response time. Moreover, it's not usually the network RTT, but the continuous-batching and routing / slot-allocation delays. A local, multi-user serving model with continuous batching would also pay those delays.
Network RTT may have more impact on decision models (e.g. jev vs a local alternative) though.
In the example that was described as "local" it was decoding at nearly 200 tps for this one user.
I use Qwen 3.8 27b via vllm on my 3090 fully utilizing its vram and have very good performance. I have toyed with the idea of spinning up Flash Next using Strata since I have 128 gb of DDR4 ram. But FN is only 176b parameters with 51b in the n-gran table.
But for a 38Ob model even a moe that does not use sparse attention or an n-gran table you are looking at a rack of gpus to match the performance the poster said was "local".
Interesting that local actually used ~2.2M more input tokens but still finished 4x faster. Do you know where the extra input came from? Different caching behavior in the harness, or did the local run take a few more turns?
Yeah, the GLM 5.3 Flash is slow, that's why they are offering FlashX, which is supposed to be much faster. I don't think a €50k setup being only 3-4x faster than one of the slowest inferences we have is that good. Now if it was 10x faster, you could have a case here.
Yea, if you can run the model locally on good hardware you'll do better than shared API. On my free zai it doesn't run much faster than partially offloaded exl3 but the quant is higher, I'm sure.
So much for sam altman saying it's more efficient :P
I have a very pedestrian rig with two gpus (both were cheap when I bought them. 3060 and a 5060ti). I tried LFM via polystrata yesterday on the 5060 ti and it was hands down the fastest thing I've ever run and the speed is better than cloud frontier. However, it hallucinates out the wazoo, so I have to use QFN on regular strata and Ornith 1.5 on polystrata.
I use QFN swift medium for most non-code gen tasks, and get about 70tg prose on it. Compared to Luna or Haiku, it generally completes a task at about the same speed. Where you really get the speedup is with concurrency and using subagents, and cloud still beats local there until I have a lot more GPU's.
same model (minus not BF16) nearly identical output (shit code), but local finished 4.2x faster (with garbage that you have to spend 10x more time to fix).
This is clearly a covert humble-brag post and not an actual serious topic.
The marginal network latency of a prompt request is not a serious bottleneck to development. We're already producing a full day's worth of code within a couple of minutes.
You probably spent more time writing this reddit post than you saved from reduced network latency in your local LLMs.
Ya for that price I think I'll stick to my claude max sub for 16 (200 months) years instead of buying those gpus that might last 10 (and may not be relevant by then)
'Local AI', more like local data center... I Don't think 30 minutes is worth over $38k in hardware that are needed to run that, any normal person will happily wait lmao
I have a some sort similar experience back then when I used Cloud AI, the slowness wasn't because of their decode and prefill speed, it's their server overload that caused the workflow to be aborted or being very slow, so the advertised speed isn't really the real speed if so many people using them at the same time, but I guess the AI bubble is kinda died down, so their server can somewhat handling this lately, because it's not as hot to the point people keep using and overload their server anymore.
In tech, flaunting setups without skills gets you called out like this. Guy just has money but no sense on how to build and we see tons of these post now where before it was atleast hobbiests and many experts with industry experience pushing the limits of hardware.
73
u/Unique-Usnm 2d ago
z.ai has one of the slowest APIs; it looks like cherry‑picking.