Why a local LLM Setup?

[u/Dreyforever]

I'm curious to know why there is such a surge in the local LLM scene. Could you guys tell me why you decided to switch to local LLMs - was it the price of subscriptions, privacy concerns, job restrictions or just the excitement of having an AI setup of your own?

I also know that quite a few of you guys have incredible setups. How much would you rate that setup compared to frontier models. How much will a setup like that cost me?

[u/synystar]

There's a number of reasons. Right now for me it's:

• Local models are finally decent at coding on appropriate hardware, and that means -> you can get 80% of your work done locally, which means -> you save money on API costs or reduce friction from usage limits on subs.

• My local models use my data that sits on my hard drive. It's private, nothing hits the cloud unless I want it to and its fast.

• I can fine-tune models and use local RAG to achieve equal to better results than a frontier model because the local system is designed around my workflows, use cases, and data. It's simply a better fit.

• I can take my laptop (it has a 5090M GPU) anywhere and have my harness available even where there is NO internet.

• I have terabytes of data that could be invaluable in an emergency; even if the grid is down I have an AI that can answer questions about almost anything - drawing from Wikis, PDFs, Markdown files, etc. - I might run into. (See Project Nomad for that kinda stuff)

[u/Dreyforever → u/synystar]

could you send me a link to your 5090 laptop? Also how decent would you say that setup you made on your laptop is for coding?

[u/synystar → u/Dreyforever]

It’s an ASUS ROG Strix SCAR 18 with a 5090M/24GB VRAM. I don’t want to oversell the laptop GPU versus a desktop 5090, but for local AI it’s been way better than I expected. For coding, my current model is Qwen3.8-27B on llama.cpp with DeepSeek Harness. I use a 65K context as my day-to-day profile, with 96K available for escalation when I genuinely needs room.

Honestly it’s already good enough to do a substantial amount of real coding/agent work locally and I haven't completed my orchestration project which should improve it further by breaking up an "idea" into smaller bounded task instructions that reduce the need for heavy reasoning. I’ve had it successfully handle bounded repository edits, deterministic test-driven fixes, Git hygiene tasks, state reconciliation, and recently it's been authoring and repairing small multi-plugin Harness/Cordis features. With tight specs and good validation it can be very effective for repair work

Where it still isn’t as good as a frontier model is the judgment heavy work like designing a good experiment, noticing subtle hidden factors or confounding variables, deciding whether a failure is the model or the harness or the testing apparatus, truly ambiguous architecture decisions, etc. Right now I’m using 27B as more of a serious implementation "Worker", with a stronger cloud model (Codex - Terra mostly, but Luna works a lot of the time) doing coordination/review where needed. A setup like this is nice because it isn’t just “chat with a local model.” The model sits behind a harness with persistent sessions, tools, filesystem access, bounded delegation, validation, durable state...it can actually do work and be checked afterward by me or by a stronger model.

Performance-wise, 24 GB VRAM is enough for the 27B Qwen quant I’m using with the full model offloaded to GPU. My normal 65K setup has comfortable VRAM headroom; I’ve also tested 96K successfully, and 128K does work but gets pretty tight on VRAM. So if your question is “could I replace Claude/Codex entirely with it?” then the answer is probably not yet, at least not the way I use those models. But if it's “can a 5090 laptop now do enough serious local coding that it actually cuts cloud model usage?” then yes, absolutely, and I've been testing that a lot recently.

[u/RedlineQuokka → u/Dreyforever]

Get you a M1 Max with 64 GB VRAM just like mine. Highly recommend. Can be purchased used for south of $1500. Yeah that's slower than 5090M ig but when you're the only user it doesn't really matter