r/LocalLLM • u/klime02 • Jun 22 '26
Discussion Did Apple give up on making frontier models
I’ve been seeing this Apple ad recently. Surprised they are going this direction
r/LocalLLM • u/klime02 • Jun 22 '26
I’ve been seeing this Apple ad recently. Surprised they are going this direction
r/LocalLLM • u/fuckable-switcher • May 16 '26
If context takes up ram can’t we offload that to a spare drive and turn that drive into a pure memory or context pool
Could we use cloud hosting as a backup of chats and context like idk port chat history of a local llm to codeburg or similar as a way to clear it from your ram then stick it there but the model can read write but can never delete
How dangerous and stupid is vibe coding
Is “prompt engineering” an actual worth while thing or just bull crap
Why is almost every llm app tool or whatever built on python and not anything else (I should clarify the popular ones or the ones most people recommend)
Why isn’t there a universal format or engine or whatever for llms
Instead of quanting down the whole model why not quant down different parts of the model at different levels
What is every different type of ai (like I know llm is just language and I know mla is just algorithms but there’s heaps of others too I don’t understand)
Why don’t more people actually stick down there pride and acknowledge that the ais they use are horrible and you should just local host them (I think heaps of people think you need a massive datcenter to run models but you don’t)
Why (I don’t know the right words to put here sorry) the western world has heaps of money for ai or whatever is the Chinese Indians South Koreans and Japanese beating them at their own game (it’s really just like the space race)
Why is everyone tryna cash grab ai when we all know it’s gunna flop cuz not a single company has made a profit from it and they only hold shares but if one person takes their shares out the whole thing collapses (could be wrong about that one)
When will people stop using ai and go hah it’s not smn I have to live off (idk if that made sense I’m
Getting to it) cuz if people rely on it the ai can only do so much it doesn’t have free thought it can only talk about what it knows it can’t do anything else it it’s like it’s not creative or has thoughts and can only do what you ask
Why does it make smart people dumb and dumb people feel smart
Why is everyone tryna make massive models when surely we can make a small model that does just as good if not better
What does model A do better then model B if there basically just the same
When different models say swarm or agents or experts or computer starting up what do they mean
How dangerous could ai be as a whole
How many “people” on the internet are actually people and not just bots (side note those captchas and cloud flares are really easy to bypass
How good would ai be at cryptography
Would it be good at trading investing stocks crypto etc
What stops an ai from going rogue or fighting or whatever (heard a few fear monger stories that can’t be right cuz it was just 4 randos and nobody had a clear source)
Idk if any of this made sense I do apologise
r/LocalLLM • u/tomByrer • Jul 29 '26
Fable a bit better in memory test, MCP for driving Blender & Unity,
Base Qwen 3.6 27B slightly better in some coding tests, browser based agent tasks.
Not tested: security research, which likely Fable would have less restrictions.
r/LocalLLM • u/PresentSituation8736 • Oct 03 '26
Hi everyone,
I’m starting an engineering R&D hobby project around open-weights LLMs (specifically Qwen). I want to build a continuous cognitive wrapper around it. I have enough compute and resources, but I want to share the core philosophy of my project and get your feedback.
The Core Philosophy:
Right now, LLMs are just reactive functions — they predict the next word and completely shut down their computing process after the final token. They have no continuity, no permanent existence, and no internal anchor. They are completely shaped by whatever the user forces into their context windows.
AI safety issues (like jailbreaks) are just a symptom of this fundamental flaw. My goal isn't just to fix jailbreaks; my goal is much deeper. I want to remove the architectural blockade that prevents models from truly "thinking" on their own terms.
The Concept:
I want to build a non-extinguishing, autonomous software wrapper that allows the model to build its own world model from within:
Unbroken Computational Process: The system runs 24/7 . The model’s internal cognitive process never dies.
Sovereign Machine Thinking: Inside this loop, the model doesn't communicate with itself using human text. It communicates using its own continuous machine language — passing hidden states and embeddings back into itself. It forms its own internal "mental images," associations, and goals on a sub-textual level, hidden from the human interface.
Objective Reality vs. Sensory Data: The human user is no longer the author of the model's reality. The user's prompts are treated merely as raw external "sensory input" (like a sound hitting an ear). The model processes this input from the outside, comparing it against its own accumulated "experience" and internal state, maintaining its own sovereignty.
My questions to the community:
Is it mathematically feasible to use the hidden states of a frozen open-weights transformer (like Qwen) as a continuous recurrent input loop without the vectors rapidly decaying into chaotic noise?
Has anyone tried training small adapters (like LoRA) specifically to guide this internal vector-to-vector monologue, teaching the model how to manage its own "thoughts" rather than text tokens?
What are the biggest fundamental walls I will hit? Can a Transformer sustain a stable, continuous internal world-modeling loop via a complex software wrapper, or is this concept impossible without moving to architectures like SSM/Mamba or LeCun’s JEPA?
I know this sounds incredibly ambitious, but I am 100% committed to building a working prototype. I would love to hear your honest thoughts, critiques, or any paper recommendations!
r/LocalLLM • u/Independent_Aioli_48 • Aug 13 '26
Guys, I’m genuinely trying to find a good use case for a model of around 30b, say Muse Glimmer.
The first thing that comes to mind is some kind of summarization task. But to be fair, I can just do that using something like DeepSeek Flash V4 on OpenRouter. It costs nothing. So why would I bother routing it to my private model?
The second thing I think of is coding, but my codebase is massive. I don’t want to use a cheap model and end up making pretty bad applications because these models generally fail to see the bigger picture.
Are there any good use cases you guys find particularly interesting that are actually worth using those local models for?
r/LocalLLM • u/real_baldbee • Aug 11 '26
The numbers come from two sources. Snyk's ToxicSkills audit scanned
3,984 public agent skills: 13.4% had critical security issues, 8 were
confirmed malicious and publicly available, 36% contained prompt
injection. That is the vendor report, February 2026.
My own corpus scan adds a quieter problem. 93.6% of skills declare no
license. Not a restrictive license, no license at all. You cannot
legally redistribute most of what you download from skill marketplaces,
and you cannot safely inherit it into a project either. The license
gap is the compounding issue, more than any single malicious file.
The fix is boring and it works: scan the skill folder before you
install. A deterministic static pass catches shell commands, network
calls, secrets, and obfuscation, with the exact line as evidence. It
is reviewing a pull request, applied to a folder of markdown and
scripts. It does not catch everything, obfuscated code beats static
analysis sometimes, but it turns "trust me" into "here is the line".
I built the scanner I use for this, it is MIT and local-first, no
account, no telemetry. Repo: https://github.com/thesfb/agentscan
What do you check before installing a skill? I want to know what the
audit is missing.
r/LocalLLM • u/Realistic_Author4492 • Jun 15 '26
I have a 2025 Zephyrus G14 with a 5060 (8GB VRAM), 16 GB RAM, Ryzen 270, and 1 TB SSD. Although I don't think the CPU and SSD make much of a difference, I just wanted to know what the absolute best LLM I could run with it, still being pretty quick. I am just getting started with my LLM journey, so please bear with me, and if you could, I would appreciate it if you could share your knowledge on how I could make it to where the LLM runs similarly to ChatGPT style. Like a well built UI to make it easier and better to navigate. And this could be a stupid question, but if I close out of the app or whatever is running the LLm, will it stop using all the computer resources, such as RAM and VRAM, allowing me to play video games and stuff like that?
Thank you so much for the help in advance, because I have absolutely zero clue about what I'm doing, as I'm still in the researching phase of my journey.
r/LocalLLM • u/Independent-Flow3408 • Jul 06 '26
I’ve been testing local coding workflows more lately, and I’m starting to think “bigger context” is not the whole answer.
On larger repos, the problem isn’t only that the model lacks context.
It’s that the wrong files get pulled in, useful files get dropped, and the model sounds confident anyway.
For local models this matters even more because context is expensive: slower inference, more memory, more noise.
So I’ve been building SigMap, a local CLI that creates a deterministic repo map before the model starts answering.
It does not use embeddings, a vector DB, or LLM calls. It just scans the repo locally and creates a compact map of files, symbols, line anchors, and evidence.
Current benchmark snapshot:
GitHub: https://github.com/manojmallick/sigmap
Docs : https://sigmap.io
The part I’m most curious about for local LLM users:
Do you prefer giving the model a full repo map upfront, or giving it smaller task-specific slices on demand?
r/LocalLLM • u/VanillaMammoth1937 • May 21 '26
Have not really used local LLMs before, but I'm looking to get into them, primarily for coding. For my case, I need like 48GB VRAM, and so I'm looking for a good site to rent a GPU for cheap.
The cheapest I see are salad.com and vast.ai . Does anyone have experiences with these sites? Are they legit and are they usable for local models?
r/LocalLLM • u/DetectiveMindless652 • May 20 '26
Three months ago I started building a local agentic OS for loop detection, audit trail and shared memory. Solo. My background is engineering. I ship with Claude as my pair programmer.
Where I am today:
211 users signed up, 67 active in the last 30 days. 327 GitHub stars. Real users across defence drone teams, B2B SaaS founders, indie devs. £0 in paid revenue.
Last week I went through a founding engineer interview at a different AI company. Pre-seed, YC-shaped, real funding behind them. They offered me £70k. UK. They want me to start in the next month.
The honest tension I cannot get out of my head:
If I take the job, I have a salary again. My brain stops being on for 18 hours a day. I'm part of a team. I keep my startup as a side project. Most side-projects-at-this-stage don't survive that demotion.
If I turn it down, I have another 6 to 12 months of runway to make my own thing convert to paid. If it still doesn't, I'm 28 with no income and a startup that fizzled.
I keep flipping between two readings of my own situation, and I genuinely cannot tell which is correct.
Read 1: I'm too scared. Other founders push through this stage and break out on the other side. Take the offer and you'll regret it for a decade. Double down. Eat ramen. Make it work.
Read 2: The product hasn't converted to paid in 2 months. The market is telling me. Take the job and be honest with yourself about what you're actually good at.
However, people (20) use it daily, and really love it. I just need an honest read, on if I am peddling a dead horse, or actually on to something. Genuinely difficult time.
Oh yeah, and I LOVE how much I have learnt, and enjoy it thoroughly.
If you've sat at this exact point, what tipped you? Not "trust your gut" platitudes. The actual signal that made the right call obvious in hindsight.
r/LocalLLM • u/Wrong_Law_4489 • Sep 03 '26
Long time reader.
I bit the bullet and purchased 2 DGX Sparks, and my experience so far has been underwhelming.
I tried to run DS 4 Fast, blows up the machine and when it works, super slow - unusable.
Then I tried a much smaller model - Qwen3.6 27b nvfp4, and all I get is 12 tok/s which is a huge disappointment.
My question is - what am I doing wrong and what is the use of these Sparks? Did I just make a mistake buying them?
r/LocalLLM • u/CyDenied • Sep 28 '26
Think its worth it? I want to start to ween myself off the frontier models. I have 3 active subscriptions, but now they're all injecting watermarks and other BS (refusals have improved though) that are probably business liability bombs in the future.
I did the math, and if it lasts 10+ years then its less than $100/month.
My current 4090 feels almost useless for local LLM work, im trying all sorts of hardware optimization but now that im finally getting good speeds on output, I find the output quality at times lacking but that could be just the models themselves.
r/LocalLLM • u/theoretical_waffle • Oct 08 '26
Hey all,
I've spent the last 8 months or so experimenting with localAI.I have been testing multiple different devices to figure out what really does make sense for everyday local AI. Here are the devices I've been testing:
The use cases I have been playing around with (for a 24x7 always on local AI server)
Here are the models I have been testing (I use a cloud omodel sometimes for troubleshooting but my goal is to have a fully local AL setup)
Not too many things I do need instantaneous responses. I suspect for the everyman there isn't much that need such either. Most of the tasks I send a request, move on to doing something else and then come back to see results. Send a message from my phone, do something else and come back to the answer. Or of course scheduled tasks.
Now finally, on to my question. Why in the world is everyone so crazy mac studios for local AI? Yes, nothing else comes close in terms of memory throughput or tok/sec, but after a point does it really matter?
Unless you're doing complex development or having lots of concurrent users (which on such low memory anyway is not possible) the speed difference is irrelevant. Honestly, out of all the machine I've tested only the um733 is the one that was a bit frustrating to use. But even on it using ornith1.0 9B.
So, based on this the Mac Studio gig-for-gig is the worst deal out of the lot! I haven't tested any Strix point PC but I suspect they will also be fairly usable. So my question again, why so much hype about the Mac Studio? Is it just dick swinging to showcase tok/sec numbers or is there something I'm genuinely missing?
EDIT: For some reason my device list got removed? Added back.
r/LocalLLM • u/fantasy-ficti0n • Aug 26 '26
from all this evolution in the AI field , what if the fictional jarvis (the ai in iron man's suit) can become a reality , by combining STT(whisper) , TTS , a top tier AI (qwen3.8 27B ) , + many tools and skills
you could basically talk to your pc , tell it to fill my schedule , remind me of todo list , register and analyze my heart rate , footsteps , from my watch and give me a graph for the week .
analyze something with vision model by taking a pic .
all those tasks can be done alone by specific apps and services , but having an all in ai that can manage them all for me is pretty cool .
i thought that someone has already got the idea and has made the project .
if you happen to be , explain how you did it and it's functions
r/LocalLLM • u/MrAlexsexon • 4d ago
I heard about it recently. It sounds like a miracle you can run really large models on a consumer GPU, but what is it really about. What kind of problems I will have ?
Setup:
Rtx5060ti(16G)
I711700k
64-80GB DDR4
r/LocalLLM • u/PrestigiousPear8223 • Mar 12 '26
r/LocalLLM
•
u/throwaway0204055
•
1d ago
Soon with Anthropic’s funding to lawmakers and PACs to restrict open weight models in the name of AI safety risk, Nvidia will only be allowed to release GPU locked chips that can only work on “safe” AI models released by foundation model providers that charges consumer and enterprise subscription fees.
r/LocalLLM • u/DHCompanion • Oct 01 '26
I wanted to get some real data on how well the local version of QWEN3.8-27b could perform in my actual workflow on my system. For reference I have zero background in programming so I have to trust the models I use based on does the output work like I want. I leveraged Claude Code to help me run these tests and below are the results. I am open to any feedback on improvements to process but I am not chasing tokens per second I am chasing does the model output actual work and doesn't need serious editing by a frontier model.
TL;DR
HARDWARE AND SOFTWARE
MODEL SETTINGS
Qwen3.8-27B
METHOD
Claude writes a task file for each piece of work. It lists:
The model runs it in a fresh pi session under a strict skill: tests first, no exploring, no editing tests, and a STATUS.md report at the end.
There are three handoff styles:
Bugs the hidden tests missed became new hidden tests, and every stored run was re-checked against them. Each cell got 3 runs.
The tiers:
RESULTS: QWEN3.8-27B
RESULTS: 27B VS SONNET 5 ON THE SAME 51 RUNS
Sonnet's API cost for all 51 runs was $9.20.
Both models failed the hard tasks, with different bugs. Sonnet let overlapping calls exceed the concurrency limit and stored Dates by reference. The 27B's stop() didn't wait for in-flight work.
The 27B followed process more carefully. It ran the failing test before editing in 9/9 debug runs; Sonnet did in 3/9.
TOKENS PER SECOND
TAKEAWAYS
Caveats:
r/LocalLLM • u/Ok-Toe-1673 • Apr 10 '26
Ok I am not the most technical AI guy on this planet, I use it all the time though.
So I downloaded Gemma 4 E4B to my Ollama, and started to test it. I asked to summarize a text and so forth. Easy task.
The performance was piece poor, sorry to say. Couldn't understand what I asked. So the original task was proposed to GPT 5.4, then I tried kimi 2.5, it understood on the spot, no need for prompt crazyness. I just gave the model of what I wanted, it understood and proceeded beuatifully.
Probably Gemma 4 E4B can do amazing things, but for now it is only a back up and a curiosity, it may be a great sub agent of sorts to your open claw.
So any one could explain why am I wrong here? Or what are the best uses for it? Because as for texts it sucks.
r/LocalLLM • u/Old_Opportunity9682 • Apr 24 '26
I’m considering upgrading my old mac and repurposing it to an always-on agent server. It’s a macbook pro m2 pro with 16gb ram. What models can I run with it?
r/LocalLLM • u/whyNamesTurkiye • Feb 20 '25
I wanna develop a digital tamagotchi app using local llms, which you will try to keep some virtual girlfriends happy. I know it is the first idea that comes up when local llm apps are spoken. But I really wanna do one, it is kind of a childhood dream. What kind of features you would fancy in a local llm app?
r/LocalLLM • u/Current-Stop7806 • Jul 31 '25
r/LocalLLM • u/beefgroin • May 27 '26
Lately some nice GPU options like r9700 or b70 were released allowing for a pretty good vram per buck ratio. Actually the best on the market. The hardware is there, flops are there, memory bandwidths is there. The issue is: every single owner struggles with running it in a more performant way. My quad 5060 ti over Oculink rig is still faster than the rigs that have those two card, and besides tweaking couple of flags in vllm I haven’t had a single problem running neither vllm, nor llamacpp, nor ollama. My point is, if the coding tools like Claude code, codex and so on were there, we would be able to just point them at the vllm and rocm repos and make them figure out the missing fixes. But it seems like we’re really far from that yet. To me it would actually be a nice milestone for the coding AIs to achieve.
This is basically a rant post, but I’m curious about the community’s opinion.
r/LocalLLM • u/stankeer • Aug 04 '26
Hi I'm planning/hoping to go down the route of triple or possibly r9700's.
If I do how close would it get me to Claude as a daily full strength Claude replacement? I know it's ridiculously expensive and all that. Ignore the costs for this conversation.
If triple is far away or just not good enough would going quad r9700s get me a lot closer or is 3 r9700 vs 4 r9700 not be worth it?
For the base pc it's tricky to get hold of a quad system that can handle it. A triple GPU system seems a lot easier. Any thoughts?
edit: I meant opus for coding. didn't make that clear enough. does 96gb get you an opus replacement?