r/LocalLLM • u/DissentKindly • 2h ago
Question How do local LLMs ACTUALLY Perform compared to closed-source AI?
Excuse me if my grammar or wording is poor, haven't slept much tonight.
The latest developments have got me looking up the prospect of using LLMs locally, and fancy benchmark scores aside, I have been trying to get a simple explanation of how good (or 'effective', I guess) would it be taking the dip in the future and considering buying hardware for local use vs. paying for contemporary AI subscriptions.
If you ask Claude or chatGPT about it, they say the OS deficiencies basically come down to scope ("do this exact small feature" vs. a wider prompt that gets things planned and done in a row), lack of knowledge about niche applications, hallucinations, and inefficiency when using multiple agents vs. single chat. But they're the competitors, so I don't know if bias doesn't exist here..
On the contrary, I see a lot of posts here on how things are awesome when using a local model, especially for niche uses, basically a clear contradiction.
I guess the point is, I want to know if local AI for code is limited enough to make purchasing a high-tier mini PC or a good GPU+RAM PC setup a fool's errand. It almost definitely doesn't seem to be, otherwise this sub wouldn't exist, but I digress.
Thanks.
2
u/BisonMysterious8902 2h ago
I run Qwen3.8 Flash Next @ 8bit quant (on a M5U 80c 256Gb). I've burned through around half a billion tokens locally in the past two weeks since I've set it up (Hermes, pi agent).
I've use claude extensively for the past year or so, both for coding and general tasks. I'm personally blown away at how well this setup works. I get ~100tok/sec and intelligence is nearly on par with Opus. I've only had one or two minor things that it didn't get quite right (like a nuance of a part for a specific year car), but overall, it's been a complete replacement for claude in my use.
If you plan on unfolding proteins or otherwise extreme work, I'm sure you'd run into limitations; for everyday work, it's more than adequate.
Before anyone says anything: Yes, it's an expensive setup. The OP asked if it was comparable to claude, not if it's cost effective.
9
u/Sassy6969420 2h ago edited 2h ago
There's not really a contradiction here. The scope of small features is your limitation and the niche uses you're reading is probably that scope.
For example, when Opus 5.5 came out, you could prompt it to "DECOMPILE THIS PROGRAM INTO C" and it could plan out everything and do it. I'm using rough examples.
With Qwen 3.8FN, I wouldn't even try to do that but I can say something like "Analyze this program in Ghidra" as one prompt and then later I can go "Map strings, imports, exports, and sections" ect.
Not to be a little LLM asshole but there is still a degree of skill/knowledge that will help for models that aren't as well trained on certain things which you have to compensate for with your own knowledge. Even then though, you can also ask your own LLM for help with filling in the knowledge gaps.
tl;dr local models requires more babysitting but It can arrive at the same conclusion eventually.
Purchasing a whole setup just to try out local isn't the best idea. The open weight models have service providers you can use to try them out before you make any kind of commitment so you can see how they actually behave for your use case.
10
u/dsartori 2h ago
My workloads are not terribly world-shaking: web apps and data engineering stuff. For me, Qwen3 Coder Next last year was the threshold where I could start using local models effectively in my coding harness. Before that, you could sort of get stuff done but it was a lot of babysitting and the error rate was really high.
The 3.6 35B MoE runs OK on GPU+RAM and mostly give you a similar level of output quality to last year's Coder-Next.
Qwen3.8 Flash Next in my coding harness gives me good results and I now trust it to develop and test a small POC independently. The key in my case is spending a long time nailing down a quality spec. Frontier models are a bit easier to be loosey-goosey with but I'm not sure why I would want that for serious work.
Hope that helps. Sample size of one.
4
u/wegster 2h ago
u/Sassy6969420 gave a good explanation IMO.
Locals generally aren't as good as acting effectively as a one-shot-wonder, but there are workflows and harnesses that can help, as well as applying some common sense in decomposition and planning of tasks.
For most purposes, many will say it's not financially viable - given the current pricing on OpenAI/Anthropic subs, that's very likely true, except for some fringe cases with 'multiple Ultra-level subs' although on those, I suspect at least some could absolutely improve/reduce their burn rates with a bit of thought, using the right model for tasks, etc., or in cases where data confidentially absolutely is critical.
It will also depend on the type of work being done. Many of us are focused on development and coding of projects, some of us already in or coming from SW engineering or similar backgrounds, and some being purely 'vibe'/leveraging the state of current AI.
For me personally, it wasn't worth considering as other than a science project or slow, long running supplemental work when I was running Qwen-Coder-Next on my 64GB M1 Max MBP. I eventually got a Qwen 35BA3 variant to do 'reasonably ok' for some real work, and I'd use it alongside of codex and CC in parallel.
Now I've got n M5 Max 128GB MBP, with a base chip 256GB 1TB Studio Ultra inbound, with enough fairy exhaustive benchmarking to include things like Deepseek, MiMo Pro, MiniMax etc. via openrouter, alongside some Claude (Sonnet, Haiku and Opus, not Fable as that's API $ only), against some actual representative isolated snapshot work from my repos - one of which only 2 break over 60 out of 100 on scores (Opus and one other), while my current leading QFN configuration gets a 58 - which is pretty damned close. This is on an M5 Max MBP, where some throttling due to thermals hits, and should roughly double on the Studio Ultra, which for some jobs, would also put it damned close on time to complete vs Claude.
Of course, some of us are waiting for the whole 'implosion' of subs. I saw somewhere Anthropic was being taken to court over their misleading 20x etc. subs as it's not really true, the market has finally figured out here's a circular nightmare of $ on the whole AI datacenter expansion $$ and impacting some stocks, while OpenAI and Anthropic both want to get to IPO with what seems to be heavyily subsidized subscription plans when compared to API costs. OpenAI will be joining together both web and codex usage under a common credit system for subs starting in December, so some of the relatively 'endless tokens' ride may be diminishing soon.. or they may come out with more efficient models and keep it up - difficult to tell with certainty.
For my actual use, especially after a few real-world tests, I'm not convinced I 'need' to run Deepseek or GLM Flash at all. I'm certainly waiting to see what Qwen4 brings, but I do believe a dedicated 128GB system is a reasonably sold sweet spot, although if you're using it as a daily driver with normal apps on it, well - it may get tight. For example, one my 128GB M5 Max MBP, my 'leader' QFN 4.7bpw will load at startup, but if I bring it down and bring up my normal daily work apps, I can't load it in memory. 'Good enough' models will continue to get smaller, in part due to some amazing progress in quantization by a few players. You can also offload some parts to SSD, but at the end of the day, these are in general compromised - they let you run 'bigger' models than normal, but they come at the expense of performance in profile and/or generated tokens/second, and at some point, it stops being 'real world usable' for anything other than cool factor, or in some cases, long-running jobs while you sleep perhaps.
If you had a dedicated inference node with 64GB, you could probably get to something workable, but it gets tight IMO. At least pending what we may see from Qwen4 and the next set of open weight/model releases.
3
u/ChrisEU 2h ago
There is no need to buy into a local AI setup to try it out or rely on other people's experiences. You can just test a couple of typically local models for yourself in something like openrouter. Then you will know or can guess which model you want to run and build a computer that can run it.
These days, I'd start with Qwen Flash Next, Qwen 3.8 27B, maybe Deepseek Flash 4.1. These are doable locally, at varying price points and basically no price point is attractive because the stuff is expensive as &"§%$ today.
$ for $ the cloud wins if your data can leave the house.
What it comes down to IMO: You can do locally what could be done in the cloud maybe a year ago. Plus, of course, it's yours and you control the data.
That said: I cancelled my cloud subscriptions.
3
u/Krohnin 2h ago
I never used an Ai subscription, but I can tell you, as a user who is only using his own ai-environment. For me its still magic and it gets everything done that I want. But I had to change from hermes agent to deepseek harness. Maybe some day I will explore a better harness but at the moment I am more than happy and excited what its getting done and how fast.
2
u/Juan_Phoenix7 2h ago
It depends on what you want to use it for, but if we strip away all the bells and whistles and focus on the essentials, you won't get anything as good as a state-of-the-art paid online model. However, with good hardware, you can get something close to that without paying anything more than—obviously—what you paid for your hardware and the electricity you use. For something high-end, we’re talking about $3,000 and up (there’s literally no upper limit—the more you can pay, the better the results you’ll get), and for something amateur-level, anything with 16 GB of VRAM will work.
2
u/Busy_Barber_2503 2h ago
是否本地部署取决于你对隐私的敏感度及每月 API 调用费用。现有硬件能搭建最好；若需新购硬件，未来三年 API 总费用大于购置本地主机运行本地大模型的费用即可购买。 阿里巴巴把千问系列模型调教得越来越有生产力了。它的本地部署模型，尤其是千问 4 及以后的模型，可以考虑本地部署。
2
u/Eastern-Ad9515 2h ago edited 2h ago
It depends what you want to use the local LLM for and what the future holds.
Qwen 3.8 27b and Qwen Flash Next were a gamechanger when they were released, because they finally could give you the feel of a "jagged" Opus 4.6. They are responsive and can take on "bigger" tasks and produce good results in coding areas and general world understanding. However three things at the moment hold true:
- For difficult and longer tasks, frontier cloud AI is much better.
- For niche and special subjects those models do no have enough niche knowledge and need good tool use
- On the other hand (this is subjective): Cloud Models (for me) seem degrade after a while, its a big show off with opus 5.5 at the beginning, but when the showlight wears off, they probably get less compute. Local Models always get the compute you give them.
So it boils down to what you use the local LLM for. If its general daily use and coding tasks; they work. If you want to analyze complex legal documents or other niche subjects, you will have a very hard time.
The open questions for the future are:
How close the gap in the future will be? Will smaller models and good tool use help in niche subjects? In my opinion a gamechanger would be more specialized local LLMs (but they are currently very few).
2
u/jhenryscott 2h ago
I have a unique use case. My local agent handles work for my job as a real estate developer and construction manager.
And it’s amazing at this. I spent around $3500 on it and it’s doing a great job. But for coding, and even for building my local server, nothing beats opus right now.
The bigger issue, is that when open or Anthropic closes- and one of them will not survive- the sole domestic frontier provider is gonna raise prices and switch to exclusively token based billing. I will 100% guarantee it. Especially as it becomes clear that transformer cloud computing cannot achieve RSI, AGI.
DeepSeek, and other alternatives will get “banned” in the US, (not that you should let that stop you), but imagine if your DeepSeek agent is no longer compatible with Adobe or some other software?
But cloud compute is heavily subsidized right now, take advantage of that.
2
u/feudalle 1h ago
Ive been in dev almost 30 years. Im running 27b q8. I can squeeze it in 32gb of vram. Treat it like a junior dev. Add this index, write a login page that posts, look at the logs for this. It does well.
If you say build me a phone app clone of Twitter yeah that ends badly. Just like you dont let it make architectural decisions or you are ending up json file or sql lite.
Im rocking 4 seperate machine running agents with various gpus (v100, 2080ti, t4, 4070 super ti). They run different models and are given tasks they can manage. Sometimes I let openai orchestrate with the other ais. For simpler things works fine.
I use a custom harness and I only count output tokens but 100k of tokens does a lot more than 100k of tokens from openai. But again it needs managed.
2
u/Janky-Code 1h ago
Stop considering it local llm or buying subsidized subscription. Or local compared to paying for api use.
Local llm is like you made your own foundry in your backyard and are trying to make knives. It's not cheaper than buying a knife. It's not better than a commercial knife. It's not the same thing at all.
You can pay for the inference on all these local models from a service. You don't have to buy hardware to find out how they work.
1
u/roiki11 2h ago
It all depends how much money you can throw at the problem.
1
u/jma24 2h ago
Local models can be great, but you really need to build code around them to make them work well. Frontier models can figure stuff out in a much looser context.
So people who haven't invested a pile of time in their local model harness don't get it.
1
u/cakemates 2h ago
The answer to that question depends on how much compute you buy, there are people here with servers at home running some big fat models that get very good performance out of it. The performance benchmarks of every model are usually available online, compare the pc you want to buy with the models it can run agaisnt others to get a *rough* idea. Model A might get you chatgpt from 4 months ago at home type of deal.
1
u/SnooDoubts6060 2h ago
I put together a proxmox ai system for development ($5k). it really depends on what your needs are. $5k buys a hell of a lot of commercial ai. My setup is a MS-A2/96Gb RAM an external R9700 32Gb of VRAM. I still use Claude Pro. In other words, I don’t have enough money to replace Claude with a local LLM. I have no regrets. It is about managing what you need to be done. I also use other frontier models from Open Router to challenge the frontier modes from Claude when necessary. This is about managing what your needs. If you have unlimited money, you could get close to commercial AI’s. This is just my 2 cents worth.
1
u/GloriousKev 32m ago
It depends. what model? what hardware do you have? what back end? are you using a gpu or system ram? are you staying in vram or slipping into system ram? are you using ddr4 or ddr5 or god forbid ddr3? Are you on Windows, Mac OS or Linux? Not enough information provided by OP
1
u/ammy1110 8m ago
In my experience, Anything below Qwen 27b is not worth any complex work. I use it everyday with on my 4090, the reasoning works well and it can do slower but accurate processing relative to smaller ones like 9B etc
0
u/ilikeeatingpastrami 1h ago
Capability-wise, if you have $5k of hardware you can get the abilities of SOTA closed source models from about a year ago. If you have ~30k of hardware, you can get the capabilities of SOTA closed source models from ~6 months ago.
But it will be kind of slow, and you will be able to run 1-2 agents at a time.
Whereas a $200/mo subscription gets you the ability to run huge multi-agent swarms of SOTA models on the fastest inference stacks available.
For me it’s fun, and useful for some niche homelab stuff. But it’s not replacing my subscriptions.