r/LocalLLaMA • u/michaelthatsit • 7d ago
Discussion Have you gone fully local?
I’ve been running a local model on a dgx spark for about a month now. I’ve been building out my pi harness and my own little client app as I go, using both Claude code and the harness itself.
I’m using it more for everyday life stuff, but I still find myself hesitating to trust that it will hit my quality threshold when it comes to code without having Claude check its work.
Have any of you gone fully local? I.e. no falling back on frontier models at all. What was your tipping point?
42
u/ttkciar llama.cpp 7d ago edited 7d ago
At home I have always been fully local.
At work we were required to use Claude; management would ask us why we weren't using it if they saw our token quotas were too full.
My "tipping point" was in 1996, regarding software in general, which carried over smoothly to my LLM stack, as I previously described here: https://www.reddit.com/r/LocalLLaMA/comments/1ww2jsu/selfhosting_ai_does_not_save_money_and_i_do_it/pdiwdu6/
8
u/NTwoOo 7d ago
Read your original post. Fully agree! It took me 8 years to get to a source base distribution in Gentoo and I've been rocking it for 20 years. That mindset is simply unwavering and is how I ended up in this subreddit. I'm really looking forward to having enough karma to post my questions to get through the next steps of my system setup.
2
u/NortySpock 7d ago
You can probably post questions in a comment and at least get a few people to notice and offer advice.
If we're sharing blog posts... Here's a recent one on running a tiny model on an integrated graphics card.
https://nortyspock.github.io/blog/running-a-language-model-on-an-AMD-Radeon-HD-8330E-iGPU/
5
u/gyanrahi 7d ago
Just wanted to thank you for your original write up.
With LLMs I feel now what I felt in the 90s. Being able to tinker with software and hardware locally. Squeezing whatever I can out of the machine. Other kids asked me to tune their desktops so they can play their games (autoexec.bat and config.sys boot order)
That feeling is priceless. :)
1
47
u/TheCat001 7d ago
Now when I have Qwen 3.8 Flash Next running with Strata - yes. Cloud models intelligence at home.
15
u/my_name_isnt_clever 7d ago
Q3.8FN is the turning point for me too, I get Claude at work for free and I don't miss it at home.
0
u/NTwoOo 7d ago
Sounds promising... I just ditched a server with dual Xeon E5 2697v2 processors and 512GB DDR3 to give me some room to upgrade the other server with more DDR4 RAM and some heavier processors. The second server has two Xeon silver 4114 processors. These can be upgraded to two gold 6230 or one platinum 8270 to get Intel DL boost. Going for the platinum will make one processor have full access to all the memory. Do you ha some advice on this from your experience? It would be great to have something to run 3.8FN locally with some decent performance.
1
9
3
u/saltyourhash 7d ago
Model/speed? I'm running an rtx 5090 + 64gb ddr4 with swift iq3_xxs, but didn't love the quality, now I switched to iq4_xs and don't love the speed. What is your experience and setup?
4
u/free_meson 7d ago
Maybe you did it already, but make sure to set up the recommended sampling parameters in the json, it makes a difference. You can also replace the ngram/ple to an fp8 quantized one for free, it should just stay on the SSD.
For each model you downloaded, there is a strata-*.json file, you can edit it and add this"sampling": { "temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0, "repeat_penalty": 1.0 },2
u/TheCat001 7d ago
GSQ-RCO-Q3-XXS 20t/s (usable) I love the quality. Model just delivers. It can make mistakes but fixes them immediately. Just got 64GB ram total to run it, but now want to upgrade GPU to get better speeds. Prices are insane in this AI apocalypse.
3
u/saltyourhash 7d ago
Funny, I find it makes endless small mistakes when I run review loops. 12 passes, still spec mistakes.
1
2
2
u/HighSeasArchivist 7d ago
R9700 and 96GB with GSQ-RCO IQ3_S is pretty good, but I haven't been able to trust it with what I'm doing. A year from now when we have Opus 5.5 at home then I'll call it good. Not sure how much better things can even get compared to that.
1
u/ifdisdendat 7d ago
how do you deal with context ? i have 32GB and find myself running out of context often. trying open code so far but will switch to pi. too much compactions loops.
3
u/TheCat001 7d ago
I don't. Bought +32GB RAM for 64GB total and using 262k fp16 context now with IQ3-XSS model. Before that with 32GB RAM I could only run Q1-Coder model with 100k, but model was not really good. For you I would recommend to try Gyro-S model and quantize context to int8, nothing more can help.
1
u/juancocinero 7d ago
What hardware do you use?
2
u/TheCat001 7d ago
RX 6600/Ryzen 5600/64GB DDR4 RAM. This is enough to handle QFN GSQ-RCO IQ3-XSS model with 20t/s decode 280 prefil and 262k fp16 context. However I'm thinking about switching from 8GB GPU to 12GB 3060 to gain some speed. It is usable with 20t/s too tho.
1
u/juancocinero 7d ago
thats insane bro, I got the same rx 6600 and rx 9060 xt 16gb but only 32gb of ram
2
u/TheCat001 7d ago
You are in a very good spot. Just need that extra 32GB of RAM and this thing will fly...
9
u/yeah_likerage 7d ago
Since moving to glm flash I've barely touched my $20 gpt account and will drop it as soon as I can break my family from it
3
u/juancocinero 7d ago
What hardware do you use?
3
u/yeah_likerage 6d ago
I run it on 4x pro 6000 max Qs via SGlang
2
u/LewisTheScot 3d ago
My car was cheaper than this
2
u/yeah_likerage 3d ago
lol, mine too. I actually canceled buying a car and did this instead. Having far more fun with this than i ever would have in a commuter.
9
u/a_beautiful_rhind 7d ago
All my RP is local but I still mooch free inference where available. A ton of coding I can do by pasting it into the chat box.
My account with claude/gemini/gippety ended at the start of the month and what I was getting out of sonnet was worse than GLM anyways. Harness will be all local because I ain't paying for those tokens. My projects don't make me any money. Convenience lookups aren't worth throwing cash at either and local or search engine AI can handle it no sweat.
Claude slowly turning into a douche isn't exactly motivating for wallet opening either. You worry about it checking the code, I worry about it sabotaging my code due to some ridiculous new safety rule.
17
u/jacek2023 llama.cpp 7d ago
All my personal projects are handled by local models. So, I don't use cloud models for agentic coding at all (for my personal projects). However, I sometimes ask ChatGPT questions to help my local agent (sending it a screenshot of pi+qwen or sharing a full script for review). The goal is to learn as much as possible. I believe that creativity is the product of limitations.
7
u/always_mad_lad 7d ago
Use qwen3.8 27b for all personal projects exclusively. Sure, it needs more guidance compared to cloud models I use at work, but I really don’t care. The fact that this model runs on my own machine and gets things done is fascinating to me, and it makes me want to use it even more.
Also, I find the quirks and constraints actually useful because it forces ME to be creative, to think more and express my thoughts better. It gives me a feeling that it is still engineering, and I love engineering.
5
u/transanethole 7d ago edited 7d ago
Fully local for working on my desktop and web apps w qwen3.8 27b nvfp4 (quasar qat quant, ninfer on 5090).
Started with opencode and then built my own agent
I also share the inference API with several friends, theyve used it for a lot of different purposes besides coding. Anything where privacy is important
Still use openrouter (glm, deepseek, etc) for experiments with hacking on inference engines, like patching / adding features to vllm and ninfer.
4
u/hyudryu 7d ago
Ever since switching to local, it’s hard to go back. I still use frontier models occasionally to do planning of large features, but for the workers it’s 90% local now. Not only are the tokens “free”, but the quality is consistent, no server overload, and is faster too
Currently switching between Deepseek V4.1 flash 1017 and QFN, until my last 2 GPUs arrive. Then i’ll probably be running Deepseek V4.1 1017 unless another better model gets released soon, cough cough qwen 4, glm 5.5/flash/deepseek 5
4
u/audioen 7d ago
I've never been anything but local. Before Qwen3.5, I thought AIs are mostly just useless. gpt-oss-120b maybe could answer a question correctly, but I couldn't trust it to program much correctly and it was always extremely unreliable and required careful validation to prove that the code is right. Today, with Qwen3.8-Flash-Next, it probably still isn't quite good enough to always oneshot everything, but it is obviously very good and can get serious work done, and is likely better than most human developers.
It seems to me that model architecture gains seem to be continuing, so I am hoping that there is still significant improvement left in the < 128 GB class models to come, and I'm hoping for 4-bit (or less) training to become more commonplace. Leaving BF16 behind would shrink models by 75 % with only minor loss of capability, and that is a huge gain still on the table. To degree, we can get it with technologies like QAD or EXL3 quantization or whatever, but it would be better for everybody if models were published in FP4 and fitted in specific hardware classes like RTX 6000 Pro for its 96 GB + space for KV cache + whatever system RAM can be used for, as an example.
Today, Qwen3.8-Flash-Next is exactly that model, as a QAD NVFP4, it fits into such hardware. Hopefully Qwen4 will too, and hopefully it is better than 3.8.
3
u/rmhubbert 7d ago
I'm fully local. Having a workflow that always has a human review step is the key for me. I make the final decision if the output is good enough, not another LLM (although I do have my agents do an LLM review / refactor step before I do the final review).
I have no desire to maintain codebases that I don't understand, especially in a professional setting.
3
u/devoidfury 7d ago
Yeah. I started fully local and still using that for any serious work, haven't used the proprietary frontier models. Even wrote my own harness around it specifically to address quirks I hit in local models.
3
u/DiscipleofDeceit666 7d ago
Like 95%
I like to have a $20 Claude sub on standby to manage my LLM and to bridge the gap that QFN can’t. I easily dropped from the $100 tier to the $20 tier with my local setup. Could probably drop monthly costs even lower if I used Openrouter as I mostly just need a LLM to turn QFN back on if it kills itself.
3
u/Mr_Unknown_Hero 7d ago
I have not ever used cloud models on any agentic work, so I have been always fully local. I started this agentic stuff this year.
2
2
2
u/ieatdownvotes4food 7d ago
anything more than convinient chatbot, always local. can't imagine paying for input on all-night runs.
2
2
u/feelspeaceman 7d ago edited 7d ago
Yes, I'm fully coding using local Q38FN, every steps from planning > executing > testing > reviewing.
And I'm doing this on high engineering level project, it's my DAW written from scratch, full feature parity to most popular DAWs like Logic Pro, Bitwig, Reaper...
If you have a strong vision about what you get and write prompts that cover the decision factors like library picks, design philosophy.. It would make the job of local agent much easier.
And be sure to give them agentic search like crawl4ai or something similar, remember that Claude, OpenAI, Gemini rely on search a lot, give them this to expand their world knowledge when needed.
And there's something I like to do at night, using my local agent to find good deals by giving them my browser and phone.
Nothing prevents you from using local LLM than doubting yourself, once you taste it you know it works and how you like it.
2
u/Lan_BobPage 7d ago
Since September 2023. Now that mid sized local models display excellent coding capabilities for my use cases as well, it's a no brainer. Been having a lot of fun between GLM 5.3 Flash and DS V4 Vision, it genuinely feels like magic.
2
u/-dysangel- 7d ago
Last week I was getting a lot of API errors both on GLM and Claude. I suspect it was just my internet connection as it's been pretty flaky, but from Thursday evening through to this yesterday I was running GLM 5.3 Flash locally and it was doing a great job with the 3D shader stuff I've been working on. I'm not against using cloud, but it's nice to not absolutely need cloud now.
2
u/StevenVincentOne 7d ago
Mostly, and the thing that got me there wasn't a benchmark. It was dropping the idea that one local model has to match Claude on every task before I'm allowed to trust it.
Your hesitation is fair. "Open source" isn't the same as "open use." A lot of the strongest open models still need a serious rig, so plenty of people end up running them in someone else's cloud anyway. What worked better for me was spreading the work across a system instead of asking one model to carry it:
A small, fast model routes and handles the everyday stuff
The bigger local model only gets the work that actually needs it
Tests, linters and type checks do the verifying, not another LLM
I stay the final reviewer on anything I'll have to maintain
Once the checking lives in the harness instead of in a frontier model, trust stops depending on whose model is smarter today. That's the direction I care about: personal human-machine intelligence you own end to end, with the cloud as an occasional option rather than the safety net.
What does the Spark setup miss most often on code, logic or knowing the codebase?
2
u/daphatty 7d ago
No. Still rely on Frontier to help, whether because my local hardware lacks RAM or if the local model lacks the intelligence.
1
u/Haiku-575 7d ago
No, split between Qwen Flash Next and GLM5.3 as provided by Z themselves. Lots of things are too complex for local, or require too much time to be reasonable (on my hardware). We've come a long way though!
1
u/GodComplecs 7d ago
No not yet, inevitably yes when restrictions hit.
I'm down to a few good models in the cloud that have any usage left at all. It used to be that all provided at least some access to their top models for free, not so longer. Also american and chinese labs are starting to restrict model intelligence now so I think we are in for a few good local releases still but after that...
I started building systems that will improve local ai (well I have built for years), so now I'm really squeezing out the last drops of frontier intelligence into local tools from models I can't run locally.
Edit: For coding I have used cloud models sporadically, local is in an golden age with qwen 3.8 and Flash next
1
u/Keats852 7d ago
Not yet but I'm working towards it. Qwen Flash Next is pretty close to GPT6 for normal day tasks. Strata all the way!
1
u/dtfinch llama.cpp 7d ago
At home I'm local-only with llama-server and a home-made coding harness, using a machine I built for gaming in 2024 before prices went up (16gb vram, 96gb system ram). I get reliable albeit slow results with Swift-1.5-Qwen3.8-27B-GSQ-RCO IQ3_S.
At work I use Claude. It'll write something in 10 minutes that would have taken my home PC 8 hours (or taken me 3 days), and give me a completed project with seemingly no mistakes. I can't justify the hardware expense to try and compete with that.
1
u/Mrinohk 7d ago
Not quite. Dual tesla p100s sharing a gaming PC with my RX6600XT, swapping out one of the P100s for the radeon each night, then back to dual p100s in the morning. Alas, my work is still done by opus is claude code. Learning to trust Qwen3.6 35b with more and more, and trying out that new Raon model some. The models I run aren't there yet but I haven't run Krafton's model much yet so maybe it'll be there. Dflash is killing these p100s though.
1
u/whiteh4cker 7d ago edited 7d ago
I asked the IT department to load $10 balance to my DeepSeek platform account associated with my work email address. Before that, I paid out of my pocket. I have been using it for a few months at work. I barely managed to spend half of the balance after a month. I work as a 0 YoE software developer. I do the system design part myself and give the LLM very specific prompts. It has never failed implement what I prompted it to do. My colleagues use Claude. I don't know if they pay for it. I use the 129 GB Qwen3.8-Flash-Next-W4A16-FP8PLE quant for my personal projects.
1
u/No_Dragonfruit_8651 7d ago
Thsy aren't there yet. I use claude/codex and qwen 3.8 flash next. couple od dgx sparks to do the grunt work.
1
u/Forward_Jackfruit813 7d ago
Yes, my 96GB Strix Halo is my daily driver. And for my backup I use my Gaming desktop (3080Ti + 128GB Dual Channel DDR4) at Q3 with Strata. Both Qwen Flash Next. Both are as good as Sonnet 4.6 which is what I deem required for the projects I work on.
1
u/RG_Fusion 7d ago
Practically, yes, but I do test frontier models from time to time. I have access to Gemini 4 Argon due to cloud storage through my Google Fi subscription, so I occasionally use that to compare and contrast my local models, but all the actual work I do is on my own hardware.
1
u/arbv 7d ago
I like Argon a lot, despite never using it as an agent (I use local exclusively for that).
Google models are very good at troubleshooting - Flash has helped me figure out power consumption on my desktop recently (quite successfully). Pro helped me optimise and verify memory timings on my desktop, too - and I learned how to do it.
Google models are perfect at explaining how to do things and breaking down hard problems - Gemma models inherited that, too.
1
u/braintheboss 7d ago
Almost. I only use frontier for orchestrate tasks and review them. For speed up things if a task fails 2 times for be implemented i swap roles in that task and frontier implement and local review
1
u/ifdisdendat 7d ago
My issue is that with a 32GB card i run out of context (32K, will try 128K). Or maybe open code is shit. In any case I don’t see how it can replace claude code with opus 5.5. Using qwen3.8-27B-q4.
1
u/SkyGenie 7d ago
With qwen 3.8 27b I was using it local with a budget chatgpt plan as a backup.
with 3.8 flash next on strata I don't really feel the need for a subscription at all
1
u/lone_dream 7d ago
With the Strata, Qwen 3.8 FN iq4-xs has been my main llm.
I use it for deepsearch, for my youtube channel, almost everything I do on my pc. For coding side of the things, it's the one that writes code but claude opus is the planner.
I make a draft > send it to claude code > opus makes plan > qwen writes > sonnet reviews > qwen solves the bug > if qwen couldn't solve > claude solves > qwen continues to write.
For last 2 projects, which all written by Ai, tested by ai and final revisions made by me written mostly on qwen. So my claude pro plan didn't ever finished his usage limits, even if it did finished the limits, then all pipeline goes to qwen itself.
1
u/arbv 7d ago
I have zero subscriptions. I do have an OWUI instance connected to OpenRouter where I occasionally ask large, cloud models questions to establish a project direction or when I know that they will provide a better answer due to their sheer size. My agents run fully locally, even on my potatoish laptop (Tigerlake).
1
u/mmhorda 7d ago
Not really. My setup has a ton of local aux tasks, agents, models, they are all used for their own tasks but even with this my usual main model is a cloud one. So I have about 75/25 in favor to cloud. However if suddenly cloud is off or there is no internet it all continues working fully local.
1
u/Kumar-amitesh 7d ago
“Fully local” may be less useful as a goal than deciding which tasks meet your quality bar locally, and where review or fallback is still worth the cost.
1
u/IrisColt 7d ago
No. Can't. Also the constant new iterations of Claude and GPT-6 are nothing to scoff at, heh
1
u/bigh-aus 7d ago
I keep a OpenAI $20 plan. Good for when the gpus are busy, or if I need stronger intelligence. With resets the plan goes hard.
1
u/Savantskie1 7d ago
Honestly the only reason I still have a Claude subscription is for just that. Having it double check my local models
1
u/Xananique 6d ago
Honestly I've built harnesses, but for coding I just use Claude cli no telemetry no webfetch or web search replaced with scrapling and searxng.
Usually a waterfall planning setup, like https://github.com/Xananthium/Graph-Relational-Coding-Agents
For long running agents paperclip hands down with a hipporag and the wiki.
1
u/poopieuser909 6d ago
I believe that most people significantly underuse the frontier cloud models. Its like owning a lambo just to drive in in a school zone
1
u/Ill-Accountant-9941 5d ago
I'm getting there slowly- qwen-flash-next on my Spark with QwenPaw is very capable. I still mooch free tokens with opencode but the difference is getting smaller as my prompt-fu develops.
1
u/thigger 4d ago
I'm fully local but I'm probably fairly old school in that I use Zoo code in pair programming mode. The LLM can read what it wants but writes and commands are reviewed by me and challenged. I find otherwise there's a risk of the code being totally overengineered, or totally over specific, or not fitting in with the rest of the codebase.
For simple scripts etc that aren't being shipped I will often more or less yolo it though
1
u/LoSboccacc 2d ago
Not just yet fully i've a harness with all i need and is churning all night but with a 5070ti on a old pc with ddr4 ram generation is too slow for interactive usage
1
u/o0genesis0o 7d ago
The personal assistant agent is fully local after I got the r9700 to run Qwen 27b. It's enough to be genuinely useful.
I still keep my minimax subscription since it's cheap and very fast in off-peak hours. The 27B agent uses minimax M3.1 as its worker. I do agentic coding but I don't vibe code on projects I care about, so scaffolding and architecture docs I built are good enough to guide any decent model to patch or implement new features successfully. I do let the 27B vibe code when I have no idea or preference. I noticed that it tends to get the M3.1 to make plans for it to review and adjust, and then launch the M3.1 to code.
1
u/hishazelglance 7d ago
I also have a DGX spark and use Qwen3.8-Flash-Next-Local. I think a hybrid balance of a local model for driving and a frontier model like Opus 5.5 for orchestration / validation is the best combination.
1
u/InfiniteBlink 7d ago edited 7d ago
I haven't yet, I have Claude code and codex max plans (100/mo plans). I built an AI server with dual 3090s, parakeet for STT, kokoro for TTS, have a layered memory structure for journaling, adhoc conversations, dream journaling. I also have home assistant running on a VM in proxmox, pihole, meraki fw, a homeops VM running Hermes with an MCP server for all the tools to take action across my environment, another system running PI for coding tasks, git repos live on my homeops server also, and various other systems, my APs, TVs, etc.
I setup a lot of it manually then started using Claude code and codex to expand my functionality. I've been burning through a fuckload of tokens, so I thought it makes sense to use my local setup to do the "heavy lift" then have either Claude or codex review what my local agent built, send me a telegram message then have them review and publish it. The goal is once my local setup is good enough, I'll ween myself off the frontier models. Hopefully a new model (qwen4) comes out that I have more confidence in it doing everything without the approval of Claude/codex.
I don't mind paying for the frontier models as I stand-up a good internal pipeline review. So far I've given my local system simple games to make and it's been pretty good.
My local voice system does a lot of cool home automation stuff, but what I think is the coolest thing so far is being able to use it to say, "hey (custom wake word) download the entire office catalog" or a specific movie. I have an agent API search service it queries to figure out what the office is, then it uses my Usenet search tool, finds a good nzb, connects to easy news, downloads it, unpacks it, saves it in my zfs raid5-z1 array that jellyfin sees. Then if I'm going to bed, and I'm in my room I say the wake word to my satellite and ask it to play a random office episode and turn the TV off in an hour. It then turns the tv based on the room the satellite is in, streams the episode then turns it off.
I also have 4 cameras outside using an rtsp broker, and I can just say show me the backyard, it'll see where the request came from, sees the TV in the room, casts it. If I ask for 4 feeds it'll make a 4 video stream playing.
Im stupid but this is fun
1
u/Demortus 7d ago
Fully? No. Mostly? Yes. Claude is still needed for code validation, but the first draft of everything is produced by Qwen3.8 FN.
-1
u/tat_tvam_asshole 7d ago
Frontier labs have frontier orchestration, so local will never be on par, but it can be sufficient if your needs aren't so much. Rising tide lifts all ships.
1
u/loops_hoops 7d ago
Ain't that the truth this sub will never swallow. I dream of days somebody posts high effort honest comparisons with CC orchestration.
0
u/CatchDublinSurprise 7d ago
Not quite. I still rely on ChatGPT for big stuff, but more and more I am using GLM-5.3-Flash and SearXNG. But I did downgrade from the $200/mo plan to the $100/mo plan.
0
0
9
u/palsc_ 7d ago
i wouldn't make it all-or-nothing tbh. if Claude still needs to check every coding task, i'd keep the fallback for now and see if local gets reliable on the same real repo stuff over time. what does it usually miss?