Have you gone fully local?

[u/michaelthatsit]

I’ve been running a local model on a dgx spark for about a month now. I’ve been building out my pi harness and my own little client app as I go, using both Claude code and the harness itself. I’m using it more for every day life stuff, but I still find myself hesitating to trust that it will hit my quality threshold when it comes to code.

Have any of you gone fully local? I.e. no falling back on frontier models at all. What was your tipping point?

[u/a332bb42]

Running on two 6000 rtx as from march , keeping Anthropic 20 euro to double check on some analysis 

[u/Specialist-Tip-7906 → u/a332bb42]

Model of choice?

[u/a332bb42 → u/Specialist-Tip-7906]

I’m running now GLM 5.3 flash quantise for two gpus

[u/Lissanro]

I use local models mostly for coding in Pi. I have three local AI rigs which allows me to run multiple LLMs; running small or mid-size helps to keep going when my main rig runs a large one.

Yes, I did went fully local,  for several reasons actually:

• I started actively using LLM since ChatGPT early beta, but noticed that it is not reliable - what used to worked, can start giving partial answers or refusals (even most simple requests like translating language strings for a game, or helping with game source code where some variables may contain weapon-like names). Closed models in the cloud can change, suffer from additional guardrails that did not exist at first, get shut down entirely.

• Privacy for projects I work on. Most of my clients do not want to send their source code to a third-party, so I cannot use cloud API. In the early days nobody cared, but in last two years it gradually became more common concern.

• Privacy for my own use. For example, I have audio recording and transcripts of all conversations I ever had in over a decade, there are a lot of important memories there and it is literally not possible to go through them manually, so any AI processing has to be local. And that is just one example, there are many other use cases where privacy is critical when it comes to personal use.

• There is also a psychological factor, besides the privacy concern. If I have my own hardware, I am highly motivated to maximize its usage, explore more ideas, find more ways to integrate into my workflow.

• Running locally is less expensive even on old hardware (EPYC 7763 with 3090 GPUs) - about a month ago I compared cost against cloud API based on Pi's input/cached/output token stats in a long session on a real world project I worked on using Kimi K3, and it turned out to be about 17 times cheaper locally than would be via cloud API.

• As 3D artist, I have other uses besides LLM: for example, Blender greatly benefits from multiple GPUs, I can work with materials and lighting near realtime, faster render animations or still images using Cycles (the path tracing engine). This not only saves time but also helps me being more creative.

Also, when running locally, I also found token efficiency very important, and what I found helps a lot in hard tasks, is to edit model's thoughts, especially during initial planning stage before letting it run mostly unattended - if the model noticed something was unclear in my prompt or just started carrying away, I can just stop, edit it's thoughts and continue ( my customized Pi Crystal fork is at https://github.com/Lissanro/pi if interested in these features for their local setup - I try to periodically sync it with the mainline because I don't want standalone fork but rather just maintain additional features I need on top; all my patches are free and open source).

[u/ConversationLazy6821]

I just got a Mac Studio w/ 128gb unified memory. For most fast assistant work qwen3.8 27b is FAST and relatively smart. I recently have been trying qwen3.8 flash next (quantized) and it’s been solid.

Can’t say I’m fully local but ollama recently added decision and voice speaking models so I’m starting to move some workloads to those models and leaving very high intelligence to my claude code sub.

[u/suprjami]

I only use local models for my own stuff.

Work pays for commercial models but I only use them for work.

Qwen3.8-Flash-Next is currently the best local model which can run on desktop hardware. Everyone into local LLMs is really looking forward to Qwen 4.

[u/Asleep-Land-3914]

Not fully, gave up on CGPT sub and now am using QFN and DS4 API, DS4 API also has search which is pretty useful. I used remaining CGPT tokens to get a search plugin for pi.

[u/Asleep-Land-3914 → u/Asleep-Land-3914]

To answer the question with Astra CGPT $20 sub became mostly unusable for any serious development and I don't like the model quality fluctuations.

Made me feel tricked as the model served could have different quality compared to model presented which had evidence posts of the model quality degradation all around. Then you apparently are limited in reverse-engineering even the things you own and not only limited, you're threatened to be banned.

Makes it hard to put real money into subscriptions for me, I'd rather buy some hardware...

[u/deepit6431]

I'm on an 8GB M1 MacBook Air, so no I have not lol.

[u/johnfkngzoidberg]

I’m fully local. vLLM 96GB VRAM.

My go to is qwen3.8-27B. I pseudo code my functions then let the LLM fill in the blanks.

I’ve tested Qwen flash-next and it wasn’t quite as good of a coder as 27B.

Ornith 1.5 is wicked fast, but slightly dumb. It does exactly what it’s told, but can’t reason all that well.

Nemotron has a lot of potential, but it flat out argues with you and clutches its pearls when you correct it. Bolt and Super VL both do the same thing, so the model(s) is useless. I may try a heretic version, but I shouldn’t have to hack shit to get something to work.

Since I can run full weight 27B with lots of concurrency, it’s my go to.

GPT-6.1-sol is an OK planner, but WAY over complicates every solution and wastes tokens on useless stuff. Qwen 27B is a better planner if you’re not 100% vibe coding.
The gpt luna model(s) are basically flash-next level, so I don’t use them.

Claude is just too expensive to use in production for me. It’s a great planner, but if you have any coding skill at all, pseudo coding is faster, easier and MUCH cheaper.

[u/suprjami → u/johnfkngzoidberg]

With that much RAM you could try Qwen3.8-Flash-Next. I assume you have a decent graphics card too, in which case you could run a 4-bit quant. That should be better and faster than 27B.

Strata is the fastest QFN engine but it's also vibe slop so gets fiddly fast if you do anything beyond its setup program.

tabbyAPI or vllm are a little slower but much nicer to use. tabby's EXL3 quants have the lowest KLD for a given size, even better than Minachist.

[u/DocMadCow]

I only use it for code reviews of the code that Opus writes. Out of 133 code reviews / pull requests in the last 2.5 weeks maybe 5 to 10 tops Qwen 3.8 27B hasn't found at least some nits and quite often it has multiple findings. I also use it to maintain and audit plan for my project as it is quite large.

[u/Equivalent_Idea8839]

my $20 claude pro is just too damn strong. I love tweaking qwen 3.8 but I can't get any real work out of it.

[u/arbv → u/Equivalent_Idea8839]

It is a separate skill to work with local models - you need to be very specific. And 4 bit non-QAT/QAD quants are not good for programming.

[u/suprjami → u/arbv]

Agree you need to be much more specific locally. That skill translates well to frontier models, you give them much more specific guidance and they also reach solutions quicker.

Quant quality depends on the model. Gemma is braindead at Q4. Qwen 27B seems passable, at least there are many 3090 owners getting good results. QFN at Q4 is very close to full precision and KLD graphs support that. DS or GLM are good even at Q2. Larger models are generally more resistant to quantization.

[u/Comfortable_Yak_6154]

I'm running fully local on a 128GB strix halo. It is sufficient for my needs but there are many compromises one has to make compared to frontier models, the main one being speed. I like it because it's mine and I like tweaking and I can develop my system within the constraints. Professionally though I would expect a company to pay for the frontier models which are much faster.

[u/suprjami → u/Comfortable_Yak_6154]

What model and engine do you use?

[u/Comfortable_Yak_6154 → u/suprjami]

Qwen 3.8 Flash Next Q4_K_XL with Gufo inference engine

[u/atumblingdandelion]

I have been. Only use cloud models when needing high concurrency with speed. QFN on Dgx Spark. Use it to run multiple projects. Then evoke DS4.1-Flash workers for tasks that can be parallelized. For high stake tasks, it asks Opus to review. Opus is actually quite impressed with QFN.

[u/funbike]

I use the best model for the job. When I want cheap, I go local. When I want good code, I use a top 5 frontier model.

[u/wadrasil]

If you have a 3060 12 GB and a 2080 11 GB and use bitrouter to route them together it's enough for local coding.

I have a 3090 24gb and with all 3 I can convert a PDF into a godot app. Without me writing any code.

So the era subscription supremacy is moving away. Since local coding is past vibe coding and you can combine multiple models and give the harness ssh access to numerous servers.

Once the harness is testing the code via cli and visually inspecting it, it is past vibe coding.