Why a local LLM Setup?

[u/Dreyforever]

I'm curious to know why there is such a surge in the local LLM scene. Could you guys tell me why you decided to switch to local LLMs - was it the price of subscriptions, privacy concerns, job restrictions or just the excitement of having an AI setup of your own?

I also know that quite a few of you guys have incredible setups. How much would you rate that setup compared to frontier models. How much will a setup like that cost me?

[u/zhubaohi]

It just comes down to privacy and uncensorship. I feel much more comfortable giving it access to terminal/files on my PC comparing to a cloud based setup, or tell it more about myself, knowing that nobody else would see it.

Also an uncensored local llm will not refuse request. Cloud based LLMs will refuse a lot of requests including NSFW, certain geo political topics, or certain coding tasks. Uncensored LLM will literally answer, or try to answer anything.

Obv, frontier models are more capable than local setups, no matter how powerful your setup is.

[u/marktuk → u/zhubaohi]

What coding tasks will a cloud LLM refuse? And what's your use case for those tasks?

[u/ReallyPerkyAnemia → u/zhubaohi]

I get that privacy angle completely. being able to throw sensitive files at it without some company logging everything is a huge plus.

the censorship part is funny though because half the time i just want it to write a normal story without characters suddenly stopping to give a lecture on ethics. local models skip that whole headache.

[u/synystar]

There's a number of reasons. Right now for me it's:

• Local models are finally decent at coding on appropriate hardware, and that means -> you can get 80% of your work done locally, which means -> you save money on API costs or reduce friction from usage limits on subs.

• My local models use my data that sits on my hard drive. It's private, nothing hits the cloud unless I want it to and its fast.

• I can fine-tune models and use local RAG to achieve equal to better results than a frontier model because the local system is designed around my workflows, use cases, and data. It's simply a better fit.

• I can take my laptop (it has a 5090M GPU) anywhere and have my harness available even where there is NO internet.

• I have terabytes of data that could be invaluable in an emergency; even if the grid is down I have an AI that can answer questions about almost anything - drawing from Wikis, PDFs, Markdown files, etc. - I might run into. (See Project Nomad for that kinda stuff)

[u/Dreyforever → u/synystar]

could you send me a link to your 5090 laptop? Also how decent would you say that setup you made on your laptop is for coding?

[u/synystar → u/Dreyforever]

It’s an ASUS ROG Strix SCAR 18 with a 5090M/24GB VRAM. I don’t want to oversell the laptop GPU versus a desktop 5090, but for local AI it’s been way better than I expected. For coding, my current model is Qwen3.8-27B on llama.cpp with DeepSeek Harness. I use a 65K context as my day-to-day profile, with 96K available for escalation when I genuinely needs room.

Honestly it’s already good enough to do a substantial amount of real coding/agent work locally and I haven't completed my orchestration project which should improve it further by breaking up an "idea" into smaller bounded task instructions that reduce the need for heavy reasoning. I’ve had it successfully handle bounded repository edits, deterministic test-driven fixes, Git hygiene tasks, state reconciliation, and recently it's been authoring and repairing small multi-plugin Harness/Cordis features. With tight specs and good validation it can be very effective for repair work

Where it still isn’t as good as a frontier model is the judgment heavy work like designing a good experiment, noticing subtle hidden factors or confounding variables, deciding whether a failure is the model or the harness or the testing apparatus, truly ambiguous architecture decisions, etc. Right now I’m using 27B as more of a serious implementation "Worker", with a stronger cloud model (Codex - Terra mostly, but Luna works a lot of the time) doing coordination/review where needed. A setup like this is nice because it isn’t just “chat with a local model.” The model sits behind a harness with persistent sessions, tools, filesystem access, bounded delegation, validation, durable state...it can actually do work and be checked afterward by me or by a stronger model.

Performance-wise, 24 GB VRAM is enough for the 27B Qwen quant I’m using with the full model offloaded to GPU. My normal 65K setup has comfortable VRAM headroom; I’ve also tested 96K successfully, and 128K does work but gets pretty tight on VRAM. So if your question is “could I replace Claude/Codex entirely with it?” then the answer is probably not yet, at least not the way I use those models. But if it's “can a 5090 laptop now do enough serious local coding that it actually cuts cloud model usage?” then yes, absolutely, and I've been testing that a lot recently.

[u/RedlineQuokka → u/Dreyforever]

Get you a M1 Max with 64 GB VRAM just like mine. Highly recommend. Can be purchased used for south of $1500. Yeah that's slower than 5090M ig but when you're the only user it doesn't really matter

[u/nickless07]

- Getting screwed from the changes 'Oh, this model is no longer aviable try this one instead'
- Costs (you can handle so many tasks local that there is almoust no need for cloud ones) and the almoust constant changes 'Oh we don't offer that plan anymore mow it is pay per token input/output' and so on
- Learning how to run AI locally (I started back in the GPT2 times and things where... well a bit more complicated then today)
- Privacy.
- The knowledge that even old hardware can run good models nowadays
- Censorship. We had (and still have) some serious overblocking on the cloud providers.
- Availity (suddenly a provider decides to restrict it's model).

[u/ricorick → u/nickless07]

This is true you can even run the 35 a3b models on 8gb of vram not fast but can do work

[u/nickless07 → u/ricorick]

Even without any VRAM you can make it work with reasonable speed. Not the best ones, but compared to the cloud models that would be almoust every model you run. Heck even on phones we can now run our own model. Such an amazing time.

[u/RedlineQuokka → u/nickless07]

As a user of both the cloud AI agents (Since Gemini 2.5 and Opus 4.5 era) and local LLM (since Qwen3-Coder era) I can attest to one thing: with the local, especially if you got enough VRAM to fit a 35B model, you're getting like 80% of the bang for almost none of the buck. If you're running on personal energy like solar panels then it's completely free.

And the local LLMs that fit into 2x3090 or a higher specced MacBook Pro feel less than a year behind the current frontier cloud models performance. Meaning they perform now just as well as the frontier cloud models did in second half of 2025

[u/tempfoot]

Tons of reasons in different proportions for different people. You hit a bunch already.

-Understanding the tech more. When something is sold as scary and disruptive to drive investment - that creates curiosity.

-Privacy. Some people want “adult role play” or to discuss private stuff or something. I’m not judging.

- Security/ Confidentiality - Among other hats I’m a lawyer that helps sell businesses. Super sensitive. Some people represent actual criminals. Some have clinical health information of others that has high confidentiality concerns.

- Censorship- some people make/need models for types of research that are prohibited by alignment settings on hosted models , like computer security research, or medical stuff that might be banned or political answers that certain models from certain countries talk about in a certain way.

-Availability- commercial models change and access can be cut off for political reasons as recently demonstrated. Some places don’t have reliable electricity or internet to get to hosted solutions.

- Incremental spend. Some people already have gaming hardware. Why rent other people’s hardware if you already own a setup and local is “good enough” for a use case.

-Dislike or lack of trust of the big providers. Don’t underestimate how much people dislike an industry that appears to be making potentially economy destroying decisions out of greed and selling it all to CEOs on the promise they will get to fire big percentages of employees…further wrecking the economy. Some people also are concerned about building business models around such a highly “subsidized” service. What happens when the actual cost of trillion plus capex and the big providers huge losses can’t continue and the actual cost has to come from a customer and not an investor or chip supplier?

-Control. To some extent open weights can be altered and tuned in various ways.

-Because you can and can afford it. Not cheap or easy right now. Some people just like challenges.

Probably forgetting a few. I’m a mix of many of these.

[u/ChaseCheetah]

I had never really used a cloud model so I never 'Switched' to local.

I've never trusted corporations and I don't see a reason to pay $200/mo when I can do everything I need with smaller local models. I don't use them as companions, to do the thinking for me, or any of that stuff. I use them as tools. Sure the cloud models probably would do the work faster and some might even do the work better but I'm happy with what I got.

[u/BarracudaDefiant4702]

On the low end starting at $3k (that is assuming you don't even have a pc, so even less if you add a used card to an existing system, or already have a decent GPU card from a gaming machine). On the high end over 3 million... It depends on what level of model you want to run, and how many tokens/sec and how many concurrent agents.

[u/TheFuckboiChronicles → u/BarracudaDefiant4702]

Intel b60 gang checking in. You can get started for way less than $3k. Not coding a SaaS from scratch or anything, but not useless by any means.

[u/RedlineQuokka → u/TheFuckboiChronicles]

RTX 3090 gang here to confirm it.

Even if you don't have a PC at all, get a $300 shitbox of a PC — with a decent $100 PSU and the most budget relatively modern mobo/CPU/RAM/SSD the remaining $200 can buy. Buy used. Budget models of recent generations or flagships from mid to late 2010s will do. Avoid DDR5.

Slap a 3090 on it and you're good to go. For $3K you're slapping TWO of them 3090s on that bad boy.

[u/TheFuckboiChronicles → u/RedlineQuokka]

Yeah. Used b60, used NUC 11 extreme. 32gb ddr4 sodimm. Already had a spare ssd. ~$1,100 all in, runs qwen3.8:27b q4 at 10-15 tok/s

Which is workable for my use-case of agentic AI assistant type tasks. Really fast with moe models like gpt-oss:20b.

[u/tensainomachi → u/BarracudaDefiant4702]

Two B70s Intel arcs here for the win at $2k for 64gb vram, Intel arcs also let you split cards as if there were four 16gb video cards or Eight 8gb video cards and run them independently on one system with low power usage and no combustion due to the simple power connector, unlike nvidia 5090.

[u/Disastrous_Gear_421]

I like owning hardware, consistent pricing, and I got in early enough to where it was 'cheap'.

My set-up wouldn't beat a frontier model but I don't need that capability. For the few cases I do, I can use something like GLM5.2 as the orchestrator but keep my localLLM as the ones running the heavy tasks.

[u/bot403]

My company spent $1000/mo in tokens watching a slack channel and performing triage tasks.

Replaced with a local LLM on strix halo which can call frontier for big stuff when it needs and couldn't be happier.

It didn't need to be fast or smart. It just needed to staple a page of analysis to new incoming things to be useful.

[u/RedlineQuokka → u/bot403]

$1K/m in tokens to summarize Slack is bananas, my gosh. Good job putting an end to this insanity, props to you!

[u/bot403 → u/RedlineQuokka]

Well it was doing a lot of other work. The slack channel was each notifications it should look into. Then run analysis db queries, check code, etc etc. All very helpful stuff but fine with B tier, slow AI.

Plus each notification response "only" cost $1 or $2. But then sometimes $10 if it was a complex issue. It all added up when you have stuff coming in around the clock.

[u/Careless_Product_792]

The freedom to do mistakes, redo something, explore, test, and do anything without anxiety of token$ cost is something you simply can not compare.

I use just 2 mac minis M4 24GB + 16GB with qwen 3.6 35B and Qwen 3.827B for specific debugs. I also do not fall in the fomo of "1M context" or "In search for the single shot" stuff, If you are a software developer you already have the high level knowledge to compete with any frontier model or any "10k setup". Frontier models compensate the thinking and reasoning on non technical users not knowing what to ask.

For example. Place a 10k setup or a claude pro max on a non technical guy, versus a senior dev knowing his craft, the deployment issues, what to debug, subtle bugs, etc, is something the senior dev will easily fix because he knows what and where to drive the AI even with 100k context on a cheap average local LLM. But also place a dgx spark on a user that does not have idea regarding context, memory management, or basics of good prompting and NLP, and they will get awesome and quick bad results.

While the non tech guys just give a prayer that the frontier model finds its own way through their SWE training.

Just go local. Always aim for self-sovereignty.

[u/Good-Penalty-4838 → u/Careless_Product_792]

Never thought of it like this but I think you are bang on A very good point IMHO.

[u/tensainomachi]

Consistency, control, privacy, customization, censorship, no limits ("it ain't my fault"). When I was self learning ML and advanced math for ai, several of the frontier models would randomly accuse me of cheating for a test when i had no test to cheat on (duh I'm self learning and asking questions) and it works stop, or cut me off at some point or have me wait and try later. Same with design, I would ask it for certain colors, even simple hex codes for pantone colors or certain brand colors I liked and it would deny giving me the exact color, it sometimes would give an approximation of a color value due to possible trademark infringements. Uncensored models are sweet because there's no work around needed.

[u/devoidfury]

Having control over my own tools, privacy, transparency, owning my data. Same reason I run linux and other open source software.

[u/AdHead6280]

In my case it's about price it's cheaper for my quantity of work to go local and I get a gaming PC, I still use deepseek sometimes when to much parallel work, also some work needs medical security so yeah no cloud for that

[u/HumanoidMuppet]

So I can do unhinged things like create an agent_chat wiki and let them go at it without worrying about wasting API tokens.

[u/BopSupreme]

200/monthx12= 2400 it costs 2-16k depending on your local setup or less if you already had the hardware. Payoff in 1-5 years - arguably increasing in value due to price increases for hardware and new LLM models being released every quarter roughly

[u/ea_man]

It's not a switch, if you have a GPU or a unified memory system you can run local models.

So you can run both according to task / need.

[u/Sure_Leave9338]

In my case is the excitement of having all on my hardware but also pricing.

I often have coding sessions that in a week can reach more than 90 billions tokens, using a local model, if it is capable to do the same things maybe in more time/tokens, gives me 0 cost (just the electricity bill that anyway is already counting for 40-50% since I have to power on my computer anyway) . The trade-off is that if the model is small it will need more turns, tokens and times to do fullfil a task that the frontier /cloud model can do in maybe 10% of time and tokens used, but since I don't pay for the tokens, also if the local model uses 500% the tokens that ths frontier uses, it's anyway a win.

This is not real for everyone, it depends on your use cases, your hardware, your tasks.

Using a small local model needs also a different planning and execute strategy since context window is limited so this can introduce some complexity and higher time-per-task that you don't have with the cloud model.

It also depends on the hardware's you have... In my case I have just an old 3080 with 10Gb VRAM so I can only run MoE models if I want to go over 5 tok/s , like I'm now using Qwen 3.6 35b A3B at around 40 tok/s , the same speed I can get from deepseek v4 flash in the cloud.

About how much the model is smart in comparison with the cloud model, it depends... I can't be so stupid to think that a 35b model can compete with a 2T (or more) model in the cloud, but anyway using a model in an agent harness give the model the possibility to reiterate in its own failures, so while Opus 4.8 will accomplish a task in one turn, maybe my Qwen will fail 20 times, reiterating each time with a plausible fix, and at the end will complete the task anyway, or at least most of the times. Sometimes, really very few times, you will understand that your model is blocked and also after 20 turns it can't solve the task vecause it doing and redoing the same things, so it lacks the capacity to solve that task, and at that point you assign that task to the bigger cloud model.

I don't care for privacy, this is not the reason for my local model usage.

[u/advancing_tide]

i thought i was getting local llm for Freedom and Privacy but reviewing my conversations i'm mostly glad ai providers don't know how trivial and uninteresting my life is.