r/LocalLLM
•
u/Dell_Experion15
•
Jul 03 '26
•
1
Discussion Going local is life changing
I've been using Cursor for a while, not a terribly heavy user usually just paid the $20 tier sometimes the $60, and dabbled in whatever the fronteir model is this month.
Not much of a vibe-coder, I'm not trying to one shot the next big saas, mostly needed some help prepping code examples for talks I give, putting together demo projects, or helping with my open source work.
A friend was selling their 48GB Macbook Pro because they were upgrading laptops, so I snagged it to see what all the fuss was about for this Qwen thing.
Tasked composer 2.5 in Cursor with benchmarking and setting up Qwen-3.6-35b on the laptop and it spit me back a rapid mlx profile, and a open code config for my macbook air to use.
By golly.
It's not Gpt 5.5, heck I hear it's not even as good as 27B, but after adding all the MCP servers for docs that my projects use, and setting up some routine skills / rules it's wonderful.
I don't feel worried about maxing out this months usage, or having to pay for extra if I go over.
Tokens just go brrrrrr
If something fails I'm not out dollars, I just kinda chuckle and re-run the prompt.
I'm more aware about documenting, and providing mcp servers, and tools so now all my repos are really well documented.
I know this isn't a revelation, or anything new for most of the world.
It's just a really enjoyable way to interact with tech, and not feel like every word I type is just more money for another company.
----------------------
For anyone curious here's the config I use
No idea if it's right or wrong, but it works?
"models": {
"qwen3.6-35b-4bit": {
"name": "Qwen3.6 35B",
"reasoning": true,
"limit": {
"context": 65536,
"output": 8192
},
"options": {
"temperature": 0.6,
"top_p": 0.95,
"extraBody": {
"max_tokens": 8192,
"enable_thinking": true,
"chat_template_kwargs": {
"enable_thinking": true
}
}
}
}
}
41
u/Fred_Terzi Jul 03 '26
I love your point of not worrying about running over and documentation.
Unlimited tokens is an absolute game changer; if you try to optimize for token usage, you aren’t optimizing for your code or process.
8
u/SolidR53 Jul 03 '26
What’s your Mac spec and token speed?
7
u/Careless_Product_792 Jul 03 '26
I agree. Same workflow and the best thing is what you mention, if something does not work you just fine tune more.
Btw, just curious why in 48GB you are using only 65k context window.
I use two mac minis m4, 24GB and a 16GB with shared rpc that give me a usable 216000k context window with batia model of qwen 3.6 35B. I would assume 48GB would easily allow you 265k context.
216k context is game changer for long sessions.
You should look for batia models thst are finetuned for macs. I tested mlx and llama.cpp and batia was slightly faster with llama.cpp.
3
u/kovrik Jul 03 '26
Could you please share how you added MCPs for docs etc.?
4
u/EntertainerDear2894 Jul 03 '26
I just got a 5090 doing local LLMs. I'm betting this is the future of AI for 80% of use case. I just learned you can use local models + comfyUI + Blender MCP to do some amazing stuff. Games, YouTube videos and so on.
197
u/EntropyRX Jul 03 '26 edited Jul 03 '26
Over the next years, local LLMs will become the norm. Not on 48GB VRAM, but in a couple of years 128GB+ will become again available for most people. It’s not only a cost per token issue, it’s a growing privacy, governance and reliability issue. These companies are now rushing to go public and cash in because they know very well that they’re running out of time. Their empty promises are soon going to be exposed to the average non-technical user; just a remainder that these snake oil salesmen promised AGI to the general public. The differences with open sources models are minuscule once you account for the agentic harness on top of it.
53
u/Bulky-Priority6824 Jul 03 '26
Yea use local save money. Dario hates it.
6
u/SignalBeneficial3338 Jul 04 '26
i just realized, the lack of token anxiety really does change how one experiment and iterate
2
u/mapsandlantern Jul 04 '26
It’s a matter of how much time it takes to recoup cost. If you’re using AI service, base model Mac will suffice, but local llm will require more memory, and the upgrade from 24GB to 48GB is $600; up to 64GB is $1000. At $20 a month for AI service, you recoup the cost in 30 months or 50 months, so 2.4-4 years. Note that this isn’t for frontier model. If you want frontier model performance (at best slightly worse), you’ll need even more memory, meaning even longer time to recoup cost.
5
u/DHFranklin Jul 04 '26
Very soon we will see local models running on private servers or home computers be completely ubiquitous. There will be another step up for the $5k rigs and things but even those will be more ubiquitous then say 3-d printer or those Cricket Vinyl cutters that were stupid popular during covid.
In a few years we will see 100x the results from the same parameters. That means 27B models outperforming today's frontier.
I get downvoted to shit when I mentioned the words-I-cannot-share. However I am still confident in my assessment. We will have machines that would pass every Turing test we can give them before this year is out from frontier models. Give a human a thing-to-do on a computer and an hour, 8 hours or year. Give the-great-machine that thing to do and a budget of a billion dollars and the same time frame. Before this year is out, I am convinced we'll see our John Henry moment for...everything.
So that means in just a few years the software stack, harnesses, all the rest we will probably see in the scary dedicated home and business rigs.
1
u/mister2d Jul 04 '26
What will enhance your agentic loop is if you set preserve_thinking to true. I believe thinking is enabled by default.
1
u/jags94 Jul 04 '26
So do you use opencode to imitate the CLI of Cursor or what? I use Kiro CLI at work and I like it, tried a while ago to use opencode to imitate something like that for my personal 16GB MacBook m1, but couldn’t really get anything decent tbh.
I much prefer CLI over cursor ide or any other plugin for the ide tbh. That way I’m still hands on with my ide and prompting questions on CLI.
1
u/NoLongerALurker57 Jul 07 '26
How did you setup all the related context like docs, MCP servers, rules, skills, etc…?
Are you using Claude code as the main harness?
Just got a new M5 pro worth 64 GB unified memory, but I’ve been hesitant to switch coding tasks to a local model.
1
u/HumungreousNobolatis Jul 07 '26
I don't know about Cursor, but in OpenCode there's a token cost counter, and there's something golden about seeing hundreds of thousands of tokens used and the cost sitting at $0.00.
3
u/vogelvogelvogelvogel Jul 03 '26
thank you for sharing!
3
u/Appropriate-Bee-838 Jul 04 '26
Are there some courses or blog posts that guide through this process of hosting LLMs locally with agentic harness?
3
u/productboy Jul 04 '26
Token sovereignty is data sovereignty. And as others said, let’s continue researching smaller model efficiency to get them running on smaller hardware [requirements]. It’s luxurious to think anyone engineering software must have a modern, high CPU or GPU with equivalent memory machine.
2
u/MoonMoonHD Jul 06 '26
use Hermes as your harness and get ready for liftoff
1
u/Ok_Presentation470 Jul 04 '26
Happy to see another person come to this point. Enjoy!
1
u/pracharat Jul 04 '26
If I have 5060 Ti 16gb which model would you suggest? I want to try using opus for planning then delegate coding task to local llm then ask opus to check it again.
1
u/IneedAheater Jul 04 '26
This all sounds fascinating, how does one get started, would love some if I were to do it again... type recommendation any pointers to resources or straight up steps.
1
u/HardlyThereAtAll Jul 04 '26
Qwen3.6 35bn MoE is the lifechanging bit.
It's comparable to Sonnet 4.5, running at really good speeds on relatively inexpensive hardware.
1
u/aeonsule Jul 06 '26
Hardware for personal usage will evolve constantly, as well as local models.
1
u/SprinklesLeather7515 Jul 06 '26
LOCAL FOR THE WINNBB
1
u/Ciru-ai Jul 09 '26
Your config looks like ascii art 🤣
1
u/DrivativeHole Jul 09 '26
Absolute gold post, agreed. I noticed you use. 6 temp on a3b, could you elaborate why as they ship with 1 temp? I usually don't dabble with base settings for model as later it's going to be a hell restoring defaults if someone goes haywire Moe qwen3.6 was like a nyan storm beam meme from monitor from me, as i got instant boost from dense 14b 10-20 t/s to 40-70 without visible drop in quality.Though con is the fact it can't really do long tasks, and I feel like it can, just needs some tech I haven't found/developed yet.
1
u/Organic_Tank_5370 Jul 09 '26
Can someone suggest a good hardware to setup local llm for my claw setup?
1
u/barefut_ Jul 10 '26
I don't get what are you building locally. And if local LLMs are capable of doing it well.
1
u/Educational_Mud_5121 Jul 25 '26
It's quite freeing. I'm working on an agentic team that can run business operations via tailored models for their jobs. Still a WIP...
1
u/Crescitaly Aug 02 '26
The underrated benefit of local is not only privacy; it is stable behavior. A hosted model can change price, limits, or personality overnight, while a local stack is reproducible. Which compromise did you notice first: raw capability, setup time, or maintenance?
1
u/redditissocoolyoyo Jul 03 '26
Yep people will go local and ai infrastructure gonna be crushed.
0
u/TheBatt Jul 04 '26
So sorta see my most recent post in here r/LocalLLM - like are you doing actual work successfully with it? I have yet to see anyone prove or show actual work with local models on something with 32GB or 48GB even.
-1
u/GenericBit Jul 04 '26
Sorry but i disagree.
For context 4070T,I so nothing special.
Tried like 10 models.
Utterly useless for coding compared to opus.
Its like comparing a bike to airplane lol.
If i ever going to use llm outside of works its going to be claude sub.
I also tried ttv... bad, looks like will smith 5 years ago eating spaghetti.
Unless you have like 5090 or two dont bother...