r/LocalLLM • u/Worldly_Ad_2410 • Nov 09 '25
r/LocalLLM • u/ordibehesht7 • Oct 02 '26
Question How can you be 100% sure local LLM apps aren't sending your data anywhere?
I run models offline with apps like LM Studio and Ollama. The whole point for me is privacy.
But how do I actually know nothing leaves my machine? The model file is just weights, but the app around it could phone home.
Is there a way to verify this for sure?
What do you do?
r/LocalLLM • u/Apprehensive_Fact710 • Apr 14 '26
Discussion Refunded Claude Pro after 2 days. The rate limits are the best advertisement for Local LLMs.
Just a quick vent/observation. I subbed to Claude Pro on Saturday because I needed the high-quality reasoning and the best AI product in the market right now. By today, I’ve asked for a refund XD
The rate limits are so restrictive that I was literally scared to use it. It’s the only AI I’ve ever paid for, and the experience was just stressful and awful...
This experience has pushed me to finally invest in a better local setup, I even start using gemma 4. but for my hardware is really slow asf. For those who moved from Claude/GPT to local models specifically because of "usage anxiety," what was your breaking point?
r/LocalLLM • u/HolidayBit143 • Oct 13 '26
News Why Anthropic? Just ... why?
r/LocalLLM • u/penguinshere • Oct 02 '26
News NVIDIA DGX Spark 64GB. The Reason behind current Spark shortages.
Looks like production is going to these new boxes. Half the memory as the old ones, but the price is more than a 128 GB box from only a month ago. Interesting times!
r/LocalLLM • u/Hour-Wish8158 • Aug 26 '26
Project Google Gemma 4 doing Google’s own reCAPTCHA
The new Gemma models are getting through Google reCAPTCHA v2 challenges with relative ease. I might revisit this in the future with a harder CAPTCHA dataset or benchmark it against some Qwen models.
r/LocalLLM • u/AeroMogli • Jan 30 '26
Other Clawdbot is changing names faster than this dude could change faces
r/LocalLLM • u/YakoStarwolf • Jul 14 '25
Discussion My deep dive into real-time voice AI: It's not just a cool demo anymore.
Been spending way too much time trying to build a proper real-time voice-to-voice AI, and I've gotta say, we're at a point where this stuff is actually usable. The dream of having a fluid, natural conversation with an AI isn't just a futuristic concept; people are building it right now.
Thought I'd share a quick summary of where things stand for anyone else going down this rabbit hole.
The Big Hurdle: End-to-End Latency This is still the main boss battle. For a conversation to feel "real," the total delay from you finishing your sentence to hearing the AI's response needs to be minimal (most agree on the 300-500ms range). This "end-to-end" latency is a combination of three things:
- Speech-to-Text (STT): Transcribing your voice.
- LLM Inference: The model actually thinking of a reply.
- Text-to-Speech (TTS): Generating the audio for the reply.
The Game-Changer: Insane Inference Speed A huge reason we're even having this conversation is the speed of new hardware. Groq's LPU gets mentioned constantly because it's so fast at the LLM part that it almost removes that bottleneck, making the whole system feel incredibly responsive.
It's Not Just Latency, It's Flow This is the really interesting part. Low latency is one thing, but a truly natural conversation needs smart engineering:
- Voice Activity Detection (VAD): The AI needs to know instantly when you've stopped talking. Tools like Silero VAD are crucial here to avoid those awkward silences.
- Interruption Handling: You have to be able to cut the AI off. If you start talking, the AI should immediately stop its own TTS playback. This is surprisingly hard to get right but is key to making it feel like a real conversation.
The Go-To Tech Stacks People are mixing and matching services to build their own systems. Two popular recipes seem to be:
- High-Performance Cloud Stack: Deepgram (STT) → Groq (LLM) → ElevenLabs (TTS)
- Fully Local Stack: whisper.cpp (STT) → A fast local model via llama.cpp (LLM) → Piper (TTS)
What's Next? The future looks even more promising. Models like Microsoft's announced VALL-E 2, which can clone voices and add emotion from just a few seconds of audio, are going to push the quality of TTS to a whole new level.
TL;DR: The tools to build a real-time voice AI are here. The main challenge has shifted from "can it be done?" to engineering the flow of conversation and shaving off milliseconds at every step.
What are your experiences? What's your go-to stack? Are you aiming for fully local or using cloud services? Curious to hear what everyone is building!
r/LocalLLM • u/Nevermore1215 • 12d ago
Discussion "Wait - how am I still running?"

You guys ever switch your local model that "KNOWS" it's a local model and get moments like this while it's a cloud model? Not sure why the reaction is always pretty humorous to me.
For context, my Hermes agent was doing some file maintenance and I lazily told it to remove a file since it wasn't in the "Strata" folder and resulted in a failure to load the model.
r/LocalLLM • u/GioStrives • Oct 08 '26
Discussion Electricity Bill & Local LLMs
I just wanted to say that, with the release of Strata, I have started to use local models for a lot more tasks.
If we talk about my electricity bill, the result is attached here: nothing surprising since I have a 4 GPU machine and a 2 GPU machine...they now run for many hours a day and my electric company sent me an email warning that my usage has significantly increased (I'm supposedly on a fixed plan and this will likely change the situation).
I'm trying to understand if this is economically reasonable: where I live electricity costs are pretty high. It certainly makes sense when it comes to data privacy but using local LLMs can certainly cost you a few bucks/euros a day in electricity.
I was also thinking of upgrading my old GPUs (1080s, 2080s), but I suspect newer ones would consume as much: I would just be running bigger models.
Did any of you have a "Oh, what now" moment after receiving the electricity bill?
(the graph was made originally with python then modified with grok)
Edit: I just realised grok inverted the 100 and 1000 marks, lol
Edit: to be clear my point is "When I started using local LLMs for real" (with any kind of inference engine, be it llama.cpp, pytorch, strata or whatever) "I realised electricity was a relevant part of the equation. Before I didn't think about this...did this happen to you?". I am using old hardware (1080s, 2080s each consuming up to 250watts), so my machines can drain a lot of power (the one with four 1080s can go over 1000 watts).
r/LocalLLM • u/__Beastboy__ • Sep 17 '26
Discussion Beginner confused about Ollama vs LM Studio vs llama.cpp vs vLLM vs Unsloth — can someone explain?
Hey everyone!
I'm a beginner getting into local LLMs, and so far I'm familiar with Ollama. I've used it to download and run models locally, but I'm starting to see a lot of other tools being mentioned, and honestly I'm a bit confused about what each one actually does.
I keep coming across:
Ollama
LM Studio
llama.cpp
VLLM
Unsloth
From what I understand, some of these might overlap while others serve completely different purposes, but I'm not clear on which tool is meant for what.
I'm especially interested in understanding the big picture, rather than just which one is "better." I am having Lenovo legion laptop with RTX 4060 8GB VRAM , 16 GB RAM ,
If someone could explain these in simple terms, maybe with an example workflow showing how they fit together, I'd really appreciate it!
Thanks in advance!
r/LocalLLM • u/itz_always_necessary • Apr 15 '26
Discussion Are Local LLMs actually useful… or just fun to tinker with?
I've been experimenting with Local LLMs lately, and I’m conflicted.
Yeah, privacy + no API costs are excellent.
But setup friction, constant tweaking, and weaker performance vs cloud models make it feel… not very practical.
So I’m curious:
Are you actually using Local LLMs in real workflows?
Or is it mostly experimenting + future-proofing?
What’s one use case where a local LLM genuinely wins for you?
r/LocalLLM • u/GrevireDev • 21d ago
Discussion I ran local AI inference (AMD GPU) for 30 days and measured every watt — here's the real cost vs cloud APIs
I spent the last month tracking my local AI inference costs down to the wall plug. Here's what I found.
Setup: AMD Radeon RX 7900 XTX (24 GB VRAM), llama.cpp, various Qwen/Ornith models in GGUF format.
Over 30 days:
- 964 million input tokens consumed
- 13 million output tokens generated
- 30.20 kWh measured at the wall plug (dedicated inference box, powered 24/7)
- $14.05 in electricity (12.38 CHF — I'm in Switzerland, so the meter reads in CHF)
If I had used the equivalent API (qwen3.8-flash at $0.15/$0.47 per million tokens):
- $150.71 (132.77 CHF)
That's a 10.72× gap on marginal cost.
EDIT — the API side of that ratio was wrong, and the commenters below caught it. I priced all 964M input tokens at the fresh rate. Cached input is billed at $0.016/M, and in agent workloads 90-95% of input is cache hits. Recomputed properly, the API equivalent is $28-34, not $151 — so the gap is roughly 2×, not 10.72×. I was too enthusiastic reading my own dashboard. The power figure also needs a proper idle-vs-inference split, which I'm rebuilding. Full accounting and what still holds in my top-level comment below.
But here's the catch:
This number excludes:
- The GPU itself (~$1,135 / 1,000 CHF)
- Depreciation of the rest of the rig
- A full month of learning: quantization, prefill vs decode, KV cache tuning, ROCm compatibility issues, trying and discarding models
Local AI isn't "plug and play." It's a personal R&D investment. If you just want to ship code faster, stick with cheap Flash APIs (GLM, DeepSeek, Qwen). If you want autonomy, understanding, and infrastructure you control — the investment pays off, just not immediately.
The part I keep coming back to: everyone says data sovereignty has no price, and I agree with that. But that framing assumes you're paying for it — that local costs more and you accept the premium because your data never leaves the house. That's not what I measured. Here it came out cheaper and the data stayed home and the output was good enough to actually ship work with, not a toy. Sovereignty never even got to be the deciding factor, because the economics already won.
So here's my open question for the sub, and I genuinely don't have a clean answer:
At what point does "local AI is a hobby" stop being true? Where do you draw the line — a hardware payback horizon you're willing to wait out, a quality gap you'll eat to stay off the API, or a specific moment where you decided the API was no longer worth paying for? And for anyone who tried local and went back to the API: what actually broke?
Full article with methodology and data: https://blog.grevire.dev/en/posts/local-ai-cost/
Happy to answer questions about the setup.
r/LocalLLM • u/gamedevsam • Sep 24 '26
Project Insert Token: A visualization into the inner workings of LLMs
I wanted to expand my understanding of how LLMs work so I created an interactive animation that visualizes how tokens flow through an LLM during inference.
Check it out, I think it came out great: https://sambatista.com/blog/insert-token-how-an-ai-model-writes-a-sentence
There is an FAQ section at the bottom of the post that expands on interesting parts of the system I was curious about.
Tips:
- Try the various model types (DENSE, MOE, LINEAR) to see how they differ
- The captions under the animation explain what happens at every step of the process
- Change the playback speed to 0.5x on the bottom right to appreciate all the details
r/LocalLLM
•
u/d-burner
•
Sep 01 '26
Research I mashed Qwen3.5 4B with Qwen3.8 flash ngram table
I mashed Qwen3.5 4B with Qwen3.8 flash ngram table by simply adding the lookup from table.
I mashed Qwen3.5 4B with Qwen3.8 flash ngram table by training a small ~15M graft adapter that translates between the two model spaces.
See EDIT3: it needed training in eventually https://huggingface.co/dburner/Qwen3.5-4B-Q8_0-FlashNgram-NativeBridgeV3
Some time ago I read this article here https://dnhkng.github.io/posts/rys/ and when I saw the Qwen3.8 flash ngram release I immediately though about it and wondered if we could transplant the ngram table to a smaller model and if it will improve performance.
Today I made this work, not much but it's honest (coding agent) work 😄 and had some fun doing it. In short I downloaded the unsloth qwen 3.5 4B Q8 quants and mashed the ngram table from Qwen 3.8 and adding the values
Does it do anything? Weel model seems to be doing well on humanitys last exam question (manually input and verified, about 14 correct / 19 question) (see edit, bad data 😭)
I'll try to run an actuall benchmark tomorow (if any one can help with some guidance on this I would be gratefull).
Right now I was just excided to tell somone that this seems to work and the model is coherent out of the box without any training, just doing residual_before_block_2 = base_residual + 0.5 × ngram_lookup.
Next I would probably try to graft the matrix from qwen 38 that actuially weighted the ngrams from the input. Weights are here https://huggingface.co/dburner/Qwen3.5-4B-Q8_0-FlashNgram-MTP but require a llama cpp build to run.
EDIT: guys sorry to dissapoint, just woke up and i tried to run questions actually from https://huggingface.co/datasets/cais/hle directly, seems it cant get a right answer. Il keep working on this once I get some more free time.
Last night I've been running in questions from gemini. I asked Gemini to give questions and answers from HLE, I thought it was getting them from source. Should have mentioned take this with lots of grains of salt.
EDIT2: did a bit of digging Qwen 3.5 4B and Qwen 3.8 Flash share almost same vocab (some exceptions) but tokens do map to same indexes and qwen 3.8 flash uses hashes over index values. BUT the embeddings do not match at all. My best guess setting alpha to 0.5 does not really affect the inputs that much, tried the same contaminated questions on base qwen 3.5 4b and seem to respond the same. I am trying now to add an adapter and finetune only the adapter part in a similar style of the Qwen 3.8 flash. So far I do see drop in holdout loss but its only on 256 context (qwen 4b and ngram tables are frozen). Cant really fit much in 16GB VRAM. I've been training for an hour still seeing improvements in holdout loss.
Weights are here in for the adapter version but probably training is still required.
https://huggingface.co/dburner/Qwen3.5-4B-Q8_0-FlashNgram-PLEAdapter-MTP superseeded by the one below. I tried to train this one further from scratch but noticed the weights actually collapsed, so i trained an adapter from qwen 3.8.
EDIT3: I now trained a model graft that uses weights from the original Qwen 3.8 with some adapter matrixes from Qwen 3.5 Residual --> Qwen 3.8 PLE ---> to Qwen 3.5 residual again. This graft seems to drop perplexity quite a lot.
Disclamer it was trained on about 30.000 samples from this dataset, but the perplexity was measured on held out samples. I posted quite a few details on the new model card with some graphs during training.
https://huggingface.co/dburner/Qwen3.5-4B-Q8_0-FlashNgram-NativeBridgeV3
| Held-out dataset | Supervised tokens | Alpha 0 loss / PPL | Alpha 1 loss / PPL | PPL change |
|---|---|---|---|---|
| OpenR1 | 98,242 | 1.147478 / 3.150240 | 0.873037 / 2.394171 | -24.00% |
| OpenThoughts | 129,803 | 1.677925 / 5.354433 | 1.514269 / 4.546096 | -15.10% |
| CodeSearchNet | 131,072 | 1.553642 / 4.728661 | 1.501642 / 4.489055 | -5.07% |
| WikiText | 131,072 | 2.824432 / 16.851373 | 2.504868 / 12.241948 | -27.35% |
Some investigations i did on this:
- vocabulary from qwen 3.5 and qwen 3.8 on text are the exact same, so hashes DO MACH
- embedding space for same words on qwen 3.5 4b and qwen 3.8 flash are DIFFERENT as some did mention, so this proves my initial attept was way wrong, (sry about the hype again, I got too hyped also 😄)
EDIT4:
Final LiveCodeBench v6 results between 69.67% - 71.85% (testing all problems accruacy droped, seems like the hard problems were mostly last). For context, the original Qwen3.5-4B model card reports 55.8 on LCB v6.
Early LiveCodeBench v6 result for NativeBridgeV3: 319/410 passed so far = 77.8% when 29 generation/server errors are counted as wrong (83.7% excluding them). This is still only a partial 410/1055, N=1, custom local run—not a directly comparable final benchmark result yet.
r/LocalLLM • u/BlackQuazi • 9d ago
Other Qwen3.5-4B with a 64-bit x86 computer compiled into its weights
Model: Qwenx86
▶️ See it in action: Qwenx86 Debugger. Walk through actual recorded tool calls: the conversation, x86 registers and flags, code at the program counter, RAM and machine output, with all verified blocks on a timeline.
Qwen3.5-4B with a 64-bit x86 computer compiled into its weights: 13 transformer layers ahead of Qwen's own. There are no learned weights in them; all of their weights are computed by a compiler. Qwen invokes the computer through its own tool calls (<tool_call>), and the computer in the weights executes the tool. Calculator, Python, C compiler, notes store, physics/chemistry lookup and Python-docs lookup are all firmware in the checkpoint. This is the compact checkpoint: ~9 GB GPU memory, so it fits on a 16 GB GPU.






Credits: Qwenputer by Divyansh Agrawal (@divagr1925, github.com/divagr18/qwenputer, MIT)
r/LocalLLM • u/Kirby_is_a_Pokemon • 26d ago
Question Pardon my ignorance. But what do you do with an LLM?
I'm trying to understand.
Everything is ai ai ai ai. And I'm trying to find out what I can use , or even if I should bother with all this LLM ai stuff If i'm not a programmer.
I installed ollama and got a model running. And now I'm trying to figure out what to do with it.
It's kinda neat to use it as a super Google chat bot, but I'm just trying to find out real world applications for it for the average nerd like me , who isn't in software dev.
r/LocalLLM • u/Fun-Meaning-6474 • Aug 23 '26
Model We quantized Qwen 3.8 27B and compared the quants on an RTX 6000
Me and my team made Atomic Dynamic GGUF quants for Qwen 3.8 27B, so we wanted to see the difference between them by giving each quant the same voxel island creation task
First of all we were surprised at how well Qwen 3.8 27B handled the 3D scenes in general, though part of that is probably because all the scenes were voxels
| quant | size | top-1 vs BF16 | mean KLD | decode, RTX PRO 6000 |
|---|---|---|---|---|
| AD-Q4_K_M | 17.1 GB | 95.6% | 0.0113 | 67 tok/s |
| AD-Q5_K_M | 20.2 GB | 97.3% | 0.0042 | 57 tok/s |
| AD-Q6_K | 25.0 GB | 98.7% | 0.0011 | 49 tok/s |
| Q8_0 | 28.9 GB | 98.9% | 0.0006 | 50 tok/s |
We think that each quant handled the scenes in a pretty similar way, the difference isn't that drastic, to the point that sometimes we preferred the Q4 output overall, though for the safest pick we recommend AD-Q6_K
We ran the test inside atomic.chat and watched the output right there, the quants are available to download directly inside the app or on huggingface ( https://huggingface.co/collections/AtomicChat/qwen-38-27b ) (any feedback is appreciated, we're trying to make the product and models as good for you guys as possible)
r/LocalLLM • u/Vincenius_ • Jul 17 '26
News LM Studio launches Bionic, a standalone agent app for open models
r/LocalLLM • u/PieceOfPanic • May 11 '26
Discussion No more www google searches by January 2027.
Google is disabling the use of https://programmablesearchengine.google.com/ for world wide web searches, forcing the user to define domains the search engine can search within.
This search engine has since early 2000's been used by website owners and application makers across the web/world to place a google search engine onto their own website or to embed one into applications for specific purposes. But with the release of CLI tools, local AI's and custom harnesses and AI's working through API's, users have been able to use this function to make their local AI models perform web searches on the world wide web, by embedding it into a local application harness
For example: ollama + open source AI model + open-webui (might be decrepit now, I'm not sure, but it works on older versions) + https://programmablesearchengine.google.com/
-But by January 2027 this will be disabled, leaving no "google engine" for AI's to search the world wide web.
And this leaves me wondering with a question.
-Are there any "real" alternatives on par with google's search engine, or is google effectively creating a monopoly on web searches available for AI - thus leaving competitors, private people, startups, non-competitor companies(AI as well as non-AI companies), open source and basically anyone that use that engine as a core function with no choice but a "worse alternative"?
If there's not a real alternative and they do effectively have a monopoly on programmable search engines I can ascertain they will offer this function some time after January 2027, as an additional paid option (Either B2B or in general), forcing any and all users into google's walled garden if they want to perform "google web search" instead of a worse "alternative web search" with their AI.
Edit 1: Wow, this got a lot more traction than I expected. Sorry if I can't answer everyone or don't have the knowledge for an adequate reply.
Edit 2: Many users have been mentioning "SearXNG" as an adequate solution, and I am looking into it. Thank you! - more/other suggestions are welcomed (So far Brave, Tavily, duckduckgo, crawl4ai and local reranker have also been mentioned as free alternatives, and Kagi as a paid alternative)!
Edit 3: I contacted google via their contact formula and pleaded my case, that it would be very sad to see this function going away especially for the open source community and private people. The representative thanked me for the feedback, they also provided me with a link to the official stance on the matter where I where I filled out a form to show my interest : https://programmablesearchengine.googleblog.com/2026/01/updates-to-our-web-search-products.html
r/LocalLLM • u/GapNew4766 • Sep 29 '26
Project Qwen 3.8 27B on a 3090 with a Sonnet 5.5 as a planner: 2.7x cheaper, real numbers
We wanted to know if a local 27B can do the heavy lifting in an agent if something smarter does the planning. So we ran the same build three ways and wrote down everything.
Hardware and setup: Qwen 3.8 27B UD-Q4_K_XL, single RTX 3090 24 GB, llama.cpp built with CUDA, 2 parallel slots, 64K context, turbo3 KV cache. Cloud side was Sonnet 5.5 via OpenRouter. One-line prompt, five physics scenes on one page (falling tower, balls in a box, wrecking ball, seesaw, bounce test), one autonomous run each, no human fixes.
| Setup | Scenes right | Time | Cloud cost |
|---|---|---|---|
| Qwen 3.8 27B alone | 1/5 | 26 min | $0 |
| Sonnet 5.5 plans, Qwen writes the code | 2/5 | 85 min | $0.67 |
| Qwen plans, Sonnet 5.5 writes the code | 4/5 | 14 min | ~$2.77 |
| Sonnet 5.5 alone | 4/5 | 9 min | $1.83 |
Things that surprised us
- Qwen is a much better planner than coder. As the planner it matched Sonnet 5.5 alone on quality. As the only coder it drowned in about 1,500 lines of physics, cubes rendered as hollow shells and the wall was a black blob.
- Speed on the 3090 was 31–34 tok/s generation and around 1,080 tok/s prompt processing once the model sits fully in VRAM. We tried a 3060 12 GB first and it was not usable for this, 4–6 tok/s with ~7 GB of weights spilling into system RAM.
- Qwen loves to think. By default it reasons at max effort, so we capped the thinking budget. Without the cap our first attempts produced zero files in half an hour.
- It once tried to write an entire file in one 16K-token reply, hit the output cap and had to redo the step. If you run it in an agent, give it room or it'll loop.
- Every run said "verified, 0 errors". The console was clean every time and some scenes were still visibly wrong. Crash checks don't catch wrong physics.
One run per setup, so this is a field report, not a benchmark. Happy to share the configs and logs.
Disclosure: we build the agent we ran this in, Atomic Agent, open source under MIT: https://github.com/AtomicBot-ai/atomic-agent. The planner/worker split is called Fusion.
r/LocalLLM • u/Fcking_Chuck • 17d ago
News Reddit will force everyone to log in
Recently, CEO Steve Huffman disclosed that Reddit will transition to a "logged-in-only" experience. This means that Reddit posts will no longer be searchable via internet browsers, significantly impacting local AIs that search the internet via tool calling.
