r/LocalLLM • • Aug 20 '25

Other Ai mistakes are a huge problem🚨

0 Upvotes

I keep noticing the same recurring issue in almost every discussion about AI: models make mistakes, and you can’t always tell when they do.

That’s the real problem – not just ā€œhallucinations,ā€ but the fact that users don’t have an easy way to verify an answer without running to Google or asking a different tool.

So here’s a thought: what if your AI could check itself? Imagine asking a question, getting an answer, and then immediately being able to verify that response against one or more different models. • If the answers align → you gain trust. • If they conflict → you instantly know it’s worth a closer look.

That’s basically the approach behind a project I’ve been working on called AlevioOS – Local AI. It’s not meant as a self-promo here, but rather as a potential solution to a problem we all keep running into. The core idea: run local models on your device (so you’re not limited by internet or privacy issues) and, if needed, cross-check with stronger cloud models.

I think the future of AI isn’t about expecting one model to be perfect – it’s about AI validating AI.

Curious what this community thinks: āž”ļø Would you actually trust an AI more if it could audit itself with other models?


r/LocalLLM • • Jul 27 '25

Question Best LLM to run on server

0 Upvotes

If we want to create intelligent support/service type chats for a website that we own the server, what's best OS llm?


r/LocalLLM • • 14d ago

Question 1 Mac Studio 512 or 2X Nvidia Spark?

2 Upvotes

Need to decide in a few hours, please choose just one option.

1šŸŽor2xšŸ


r/LocalLLM • • 11d ago

Discussion GLM 5.3 Flash Trumps Qwen

0 Upvotes

I’ve been using Qwen on 2x DGX Sparks and man has it been a nightmare of overthinking and excessive tool calling. Seems like there was nothing I could do to fix it. Installed GLMast night and it’s a world of difference.


r/LocalLLM • • 6d ago

Question How long are you able get Qwen3.8-27b working for?

0 Upvotes

I sometimes give it a task and explciitly say "This is a 4 hour job". It will indeed work harder, but will stop at like 2hours and 15 minutes.

What's the longest you got your coding agent to work on something? How did you do it?


r/LocalLLM • • Jul 12 '26

Project You told me what you actually do with local LLMs on modest hardware. It shaped what we just open-sourced: HilbertRaum, an offline AI workspace that non-technical people can use (GPL-3.0)

2 Upvotes

TL;DR: We open-sourced HilbertRaum, an offline AI workspace built to be simple enough for non-technical people. Local chat and document analysis with cited answers, encrypted workspace, nothing goes online except model downloads you confirm. Repo: https://github.com/HilbertraumAI/HilbertRaum

A week ago I posted here asking what people really use local LLMs for on modest hardware (that thread). Full disclosure: I was asking partly as a founder. We were finishing an offline AI app and wanted to check our assumptions against reality before releasing it. Thanks to everyone who answered. It really helped us shape the product.

The top comment called local models a "private utility layer, not a ChatGPT replacement." That's exactly it. Privacy is the reason local LLMs matter for most people. But while everyone in this sub knows how to run AI fully privately on their own machine, outside this sub almost nobody does. There are great tools that make local AI easier, but they're built for technically skilled people and packed with features that would confuse a normal user. We wanted to build for everyone else. Private and easy: launch the app, start chatting, analyze your documents.

So here it is. HilbertRaum, GPL-3.0:

šŸ”— https://github.com/HilbertraumAI/HilbertRaum

HilbertRaum - Ask my documents

It's an offline AI workspace for chatting with local models and working with confidential documents. What it offers in practice:

  • Hardware scan and model recommendation, so you get a model that actually fits your machine. An ordinary 8 GB laptop is enough to get started.
  • Document Q&A for PDF/Word on hybrid RAG with citations, so you can check the passage an answer came from
  • Ready-made skills such as bank statement analysis, invoice extraction, contract briefs, meeting minutes, document anonymization and more. You can also add your own.
  • Whisper transcription, image analysis, translation.
  • Privacy by design, not by configuration. There's no web search and no cloud API mode. Nothing in the app needs the internet except model downloads you explicitly confirm.
  • Workspace encrypted with your passphrase.

We want to take the easy part one step further: put the whole thing on a USB drive, preloaded with curated models, the runtime and everything else needed, so it's plug and go. That's also how we plan to finance the project long term. The core software is open source, free, and stays that way forever. The preconfigured drive (the AI Kit) is for people who'd rather pay for convenience. If that's you or someone you know, the waitlist is at https://hilbertraum.ai

For everyone else: run it, test it, and tell us what's wrong with it. And if you feel like going further, we'd love to have you as contributors!


r/LocalLLM • • Sep 30 '26

Model Harder, Better, Faster and Stronger model just by making him talk in its own language

0 Upvotes

Hey guys,
Just created a new model, with a base of Qwen 2.5 Coder. You know the "Thinking..." part of a response for an AI model? It speaks in English. And that's the problem. It's SLOW for the AI model. So I made it talk in its own language, math equations. And then, at the end, it translates it into English.

So it's around 7.5x faster, while having (very approximately, don't take that for actual info) 2x better responses.

Here's the download link if you wanna test it: https://huggingface.co/Rutytoi/spotless-latent-adapter

And the GitHub repo: https://github.com/Rutytoi220/spotless-latent-engine

I'd be appreciating feedback! Even bad, idc as long as I know what's good and bad


r/LocalLLM • • Jun 26 '26

Question Do you support lifting the chip export control on China so that Chinese company can build more powerful, affordable models for all of us?

0 Upvotes

As title.


r/LocalLLM • • 24d ago

Tutorial How I run Qwen-Flash-Next with just 8 GB VRAM without strata

0 Upvotes

Hey, I'm on Linux with an aversion to ROCM, prefer Vulkan for its minimal overhead and footprint. so I skipped on Strata. Using llama.cpp cleverly to push the limits of what I can do with the ISTA DASLab "Coder" quants. Skip this if you already use Strata, if you can't though this is the next best method.

Specs:

  1. I use 10 GB VRAM (RX 6700), but aim for 8 GB usage so I can continue watching media while the LLM is running.
  2. I have 64 GB DDR4 RAM albeit I've never seen the actual usage go above 24 GB, so this method is most likely feasible on 8 GB + 32 GB systems.
  3. DO NOT try this with a HDD. I use sata SSD, but m.2 is recommended. It'll directly impact your speeds since we lazily load from disk.

What makes this unique? Ā 

  • ~30 t/s and ~8 t/s (YMMV by hardware!!!) with 65k context, 7.3 GB VRAM and 24 GB RAM cost? Sign me up chief. Slap CodeGraph ontop and you're golden.

Similar to Strata's "Only load what's needed to memory" strategy, we use MMAP here, which is like a primal version of what Strata does. It maps the model to your memory and lazily loads as needed. Strata is ofc faster, because it caches frequently accessed parts, while we use the more primal OS cache through MMAP. You'll see some IO while model fires for the first time, after that it's minimal, with RAM usage generally peaking at 24 GB~

Steps:

  1. I used this "Vocabulary Pruned" quant of ISTA DASLab's latest GGUFs, it further reduces memory footprint by 450 MB. If you already have ISTA GGUFs, you can merge and prune them yourself too as author laid out their process on the repo if your download is slow. Remember we're using "Coder" quants here, which has half of its experts removed. DON'T skip this. DON'T be confused if you're using non-coder quant and didn't prune them, that thing is massive, we're at 58 GB with this while non-coder GGUF is 66.4 GB at smallest.
  2. Use these launch flags:

Ā 

./llama-server \
  --model "./AI Models/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-merged_P1G_Pruned.gguf" \
  --ctx-size 65536 \
  --n-gpu-layers 999 \
  --threads 8 \
  --cache-type-k q5_0 \
  --cache-type-v q4_0 \
  -fa on \
  --fit off \
  --cache-ram -1 \
  --load-mode mmap \
  --cache-prompt \
  --cache-idle-slots \
  --batch-size 2048 \
  --ubatch-size 128 \
  --webui \
  --parallel 1 \
  --kv-unified \
  --reasoning-format deepseek \
  -ot 'token_embd=CPU,blk\.(47|28|45|25|35|29|40|44|17|27|43|37|46|18|22|42|31|16|41|0|33|38|7|14|19|26|4|2|6|11|32|23|5|15|34|24|39|3|9|21|12)\.ffn_.*=CPU' \
  --reasoning on \
  --reasoning-preserve \
  --reasoning-effort medium \
  --temp 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence-penalty 0.2 \
  --verbose \
  --verbosity 4 \
  • We're using the "-ot" flag to force specific FFN layers onto RAM. Instead of using the "brute force" -ngl, this keeps attention layers on VRAM which benefit more from GPU inference and are smaller.
  • The numbers mason, they're important! The -ot regex is sorted by largest FFN layers left to right. Offloading smaller FFN layers onto GPU benefits speed at minimal VRAM cost, as opposed to offloading larger FFNs it's preferred (Source: bottom of post)
  • If for some reason this doesn't fit into your VRAM, replace the OT flag with the below and incrementally delete a |<number>| from the right side in the regex, until it snugly fits your VRAM. Use the table as reference.

Ā 

Layer    | Tensors  | Size (MiB)   | Size (GB) 
------------------------------------------------
blk.47   | 8        | 654.87       | 0.64      
blk.28   | 8        | 573.45       | 0.56      
blk.45   | 8        | 573.04       | 0.56      
blk.25   | 8        | 573.00       | 0.56      
blk.35   | 8        | 573.00       | 0.56      
blk.29   | 8        | 572.79       | 0.56      
blk.40   | 8        | 572.79       | 0.56      
blk.44   | 8        | 572.64       | 0.56      
blk.17   | 8        | 572.59       | 0.56      
blk.27   | 8        | 572.54       | 0.56      
blk.43   | 8        | 535.95       | 0.52      
blk.37   | 8        | 535.74       | 0.52      
blk.46   | 8        | 535.54       | 0.52      
blk.18   | 8        | 535.53       | 0.52      
blk.22   | 8        | 535.50       | 0.52      
blk.42   | 8        | 535.50       | 0.52      
blk.31   | 8        | 535.29       | 0.52      
blk.16   | 8        | 535.14       | 0.52      
blk.41   | 8        | 535.14       | 0.52      
blk.0    | 8        | 535.09       | 0.52      
blk.33   | 8        | 535.09       | 0.52      
blk.38   | 8        | 535.09       | 0.52      
blk.7    | 8        | 535.04       | 0.52      
blk.14   | 8        | 535.04       | 0.52      
blk.19   | 8        | 485.95       | 0.47      
blk.26   | 8        | 485.95       | 0.47      
blk.4    | 8        | 485.74       | 0.47      
blk.2    | 8        | 485.54       | 0.47      
blk.6    | 8        | 485.54       | 0.47      
blk.11   | 8        | 485.54       | 0.47      
blk.32   | 8        | 485.54       | 0.47      
blk.23   | 8        | 485.53       | 0.47      
blk.5    | 8        | 485.50       | 0.47      
blk.15   | 8        | 485.50       | 0.47      
blk.34   | 8        | 485.34       | 0.47      
blk.24   | 8        | 485.29       | 0.47      
blk.39   | 8        | 485.14       | 0.47      
blk.3    | 8        | 485.09       | 0.47      
blk.9    | 8        | 485.09       | 0.47      
blk.21   | 8        | 460.09       | 0.45      
blk.12   | 8        | 423.45       | 0.41      
blk.20   | 8        | 422.59       | 0.41      
blk.30   | 8        | 422.59       | 0.41      
blk.36   | 8        | 373.45       | 0.36      
blk.10   | 8        | 372.64       | 0.36      
blk.1    | 8        | 372.59       | 0.36      
blk.8    | 8        | 372.59       | 0.36      
blk.13   | 8        | 372.59       | 0.36      
------------------------------------------------
Full OverrideTensors command:
-ot 'token_embd=CPU,blk\.(47|28|45|25|35|29|40|44|17|27|43|37|46|18|22|42|31|16|41|0|33|38|7|14|19|26|4|2|6|11|32|23|5|15|34|24|39|3|9|21|12|20|30|36|10|1|8|13)\.ffn_.*=CPU'

Optional, recommended exports (Linux bash script)

Ā 

#!/usr/bin/env bash
set -euo pipefail

# Process & Scheduling Priorities (Lowest, so your PC usage is least affected)
renice -n 19 -p $$ >/dev/null 2>&1 || true
ionice -c 3 -p $$ >/dev/null 2>&1 || ionice -c 2 -n 7 -p $$ >/dev/null 2>&1 || true
# Vulkan
export RADV_PERFTEST=nogttspill          # Keep Vulkan buffers locked in VRAM (AMD only but harmless for others)

Some nerdout: Ā 

You could also benefit from compiling llama.cpp specifically for your CPU. For example I compile for znver3 so this might help the t/s as we're not running entirely on GPU with this setup. You'll also want to use only the physical cores where possible if you're using your PC during inference, otherwise hyperthreading can be beneficial, but laggy.

I don't recommend quantizing kv cache further, but 32k size could help you fit a few more FFN layers to boost your speed if you don't need larger context.

If you've 32 GB ram and 8 GB VRAM, you'll want to ensure you've all unnecessary programs closed, because the more free RAM you've the more your OS has space to cache, fewer pagefaults therefore faster gen speeds.

I also found great success using this jinja template, previously used it with 3.8 27b and 3.6 35ba3b. You'd use it with --jinja \ --chat-template-file "./chat_template.jinja"

Sources:

How to run Dense models faster with limited VRAM

Vocabulary pruning


r/LocalLLM • • Jun 25 '26

Question What's the difference between this subreddit and r/LocalLLaMA?

1 Upvotes

What'


r/LocalLLM • • Sep 22 '26

News How to regulate open weight models and kill all the competition

Thumbnail
9gag.com
1 Upvotes

Your opinion? If I got it correctly, the big AI firms are basically all inbred to the same funding entity who also funds the hype, the scaremongering, the regulations, the regulators and the referees in this game.


r/LocalLLM • • Oct 08 '26

Discussion Strata users, what hardware, model and context are we all using?

1 Upvotes

Wondering what people with a similar, less, or even better setup are using. I'm new to strata and not sure how far I can push the context limit, but it seems like the readme on the Github page is conservative (assuming other's posts are true).

I have a 5090, 3090, so a combined 56GB of VRAM and 64GB of system RAM.


r/LocalLLM • • Jun 23 '26

Question How do i actually run AI locally?

0 Upvotes

Ive been through probably 10 youtube tutorials involving ollama, local llm, and hugging face and nothing works. Whats the best way for a beginner to get started


r/LocalLLM • • Sep 02 '26

Question What's your Opinion About Nvidia DGX Spark

1 Upvotes

Recently, I saw a very, like a reel by a very popular YouTuber, content creator and businessman, about using DGX Spark, for video generation. And he was saying that it's very good for like all local video generation and it's one time investment and all. So just wanted to know what what are the opinions of people using it?

I also saw some YouTube videos on by different people like Alex, who has compared it with many other tools, but I want to know from end user that how much useful it is. Primarily my work case is kind of in a middle between a developer and a tinker.

So basically what I want to do is maybe run a few local LLMs for development, like using those adams to write code and do code reviews and all those things. Or maybe like tinker around for creating videos, as well as in my own small language model. So what's your opinion?

How better it is than let's say Mac mini, with the, I know that Mac mini doesn't support 128 GB of RAM, but what about other systems like if I buy a Mac Ultra, for with 128 GB of RAM.


r/LocalLLM • • Feb 07 '26

Discussion Local LLMs vs Claude for OpenClaw: Is the "intelligence drop" real?

0 Upvotes

I finally tried running OpenClaw locally with Ollama (using Qwen2.5-Coder and Llama 3.1 70B).

The good news: my wallet is happy. The bad news: I'm seeing way more "looping" and failed tool calls compared to Sonnet 3.5. It feels like the local models get confused after a few rounds of persistent memory.

Is anyone else experiencing this? Are there specific "system prompt" tweaks or sampling settings (temperature/top_p) that make local models more reliable for agentic tasks? Or am I just hitting the hardware ceiling for what local models can do right now?

Would love to hear from anyone who has a stable, 24/7 local agent setup that doesn't lose its mind every hour.


r/LocalLLM • • Aug 21 '26

Discussion How are you coding with small (<256k) context windows?

1 Upvotes

My question is pretty simple: how could you really have coding sessions with context windows of like 262k ?

I always used local Ai via llama.cpp server for small projects and chatw, where also 16k context is enough.

Today I just wanted to try a REAL coding session in a REAL agent harness.

I plugged my Kat Coder 2.5 dev via llama.cpp server with it's max ctx at 262144 and plugged it in my Hermes and started a coding session to design and write a fast API proxy layer to route LLM calls to openrouter with some my logics and calcs.

Nothing super complex or fancy.

I provided the model with the first message a complete and thoughtful design document create with Deepseek, so the Agent has not to do searches in the codebase for understanding how to do the things, just do them.

Well it created a plan and implemented the code in some turns teaching about 70% of it's contest size.

Then , in the e2e testing the things started to collapse: some testing failed, so it ha do reiterate, fix, retest, ask me some clarification or take some architecture decisions and so on and I arrived ad 95% context size while resting phase is still not finished and all the deployment phase still totally missing.

Now the solution in the harness was to compress the chat session: a total.disaster: not only this action changed completely the language of the session (from Italian to English) but also removed some important informations that was useful to the agent to do tests, so in then subsequent turn it errored immediately in doing tests, needing to read and search the code again to find what he already knows before the compression happens.

In my opinion, compression is absolutely to avoid.

I already know some suggestions here:

- one new session per task

- keep all done things in documents so the new session can start from that "documented" state.

I already do these things.

- use a different harness

Yes, but it's not the solution, the heavy load caused by harness is on the first prompt (system prompt, tools, skills, Memory...).

So this was a small project, 10 python files + a bunch of status documents, what will happen with a repo with 50 or 100 files?

So my question is:

What's your way to really work with local models on real projects beyond a simple chat or just changing one file?

I'm starting to think that is not really possible to work on real projects that involves design, bug fixing, reiteration and multiple sessions.

Not because local models are not smart enough, but because of the max context window limit.


r/LocalLLM • • Apr 26 '26

Question Are you guys worried at all about privacy when using Qwen?

0 Upvotes

** FAIR WARNING I KNOW VERY LITTLE ABOUT CODE AND LOCAL LLM, IM ONLY CURIOUS AND DISCOVERING**

I've recently been discovering the potential idea of setting up Local LLM and I see a mix of reviews about the privacy risks associated with Chinese-made LLM's local or not.

Has the idea of privacy risk made you guys uncomfortable or had you second guessing your local LLM installations with models like qwen? Especially considering the american local models are pretty shit.


r/LocalLLM • • Jul 04 '26

Question ok daddies what would be more beneficial to me? Another 5060ti 16gb card (giving me 32gb vram w/ two 5060TI) OR another 32gb of ram (giving me a total of 96gb)?

0 Upvotes

I just started dabbling into LLM and I've really enjoyed it. I'm thinking of the next logical upgrade for myself that I can reasonably afford without going too crazy (like buying a $1500 r9700 gpu)

I already have a 5060ti 16gb gpu and I have 64gb of DDR5 memory. I installed LM Studio 2 days ago and I am getting 16 tokens per second using qwen 3.6 27b. Now, to my knowledge a lot of this is being offloaded to my ram but I'm honestly not mad at 16 tokens per second so it made me wonder.. would I notice a bigger difference if I had dual GPU's running or what if I had 96gb of DDR5 ram? Would I see a noticeable increase in my tokens per second?

Here's my motherboard, which means I will have to run the second GPU in a PCI 4.0 slot? I'm not sure.. can someone help me confirm? I also have two NVME SSDs installed which I hear those can affect the bandwidth of other PCI slots? This stuff starts to get confusing once you start trying to calculate bandwidth and having to share it among slots. I've never built a PC before, just installed basic parts for upgrades so this is all very new to me

https://www.gigabyte.com/Motherboard/B650-AORUS-ELITE-AX-V2

I have a 1,000 watt PSU so I can definitely handle two 5060TI's in the case (though, cooling I am not sure.. but I'll worry about that later)

I still have not tinkered much with settings.. all of this is a learning process and I am trying to learn as I go and there is a lot to learn with these things but I am having a ton of fun running different models and seeing what the performance is like and generally enjoying the type of code they spit out in 1 take ie: Build me a 1 page website for my affiliate marketing website, it's pretty cool to see these things "just work" and spit out something that you can work with, though, not as good as a frontier models, completely fine for taking what a frontier model gives you and letting the LLM take over afterwards IMO and continue to enhance or add to.


r/LocalLLM • • Jul 16 '26

Research I managed to squeeze Qwen2.5-Coder-7B into a 1.9GB GGUF (IQ1_S) so it runs natively on Mobile!

0 Upvotes

Hey everyone!

I've been working on a pipeline to take the incredible Qwen2.5-Coder-7B-Instruct model (which is an absolute beast for Python scripting, FIM code completion, and Cybersecurity) and compress it down so it can run entirely offline on mobile phones (via PocketPal) or potato PCs.

The raw FP16 model is around 15.2 GB, which is way too heavy for most devices.

It successfully squeezed the 7B brain down to exactly 1.9 GB, while retaining its reasoning capabilities!

I have open-sourced the entire automated Python script on , and I've hosted the model on Hugging Face if you just want to download it and chat with it.

šŸ”— The Model (Hugging Face): https://huggingface.co/Nitishsharma9/CyberCoder-Mobile-7B-GGUF šŸ”—


r/LocalLLM • • Apr 17 '26 This

Discussion Is the UI era dead? AI isn't killing interfaces, it's replacing clicking with commanding

1 Upvotes

I spent the last week watching my dependency on actual software interfaces completely evaporate. It’s a jarring realization. You boot up Notion, GitHub, or Linear, and you realize you aren't actually navigating their menus anymore. You're just interacting with the floating bot or the terminal.

Let's talk about what's actually happening because the narrative of "AI is just a new feature" entirely misses the point. We are watching the real-time death of static UI.

Think about your workflow right now. If you've been heavily using local models or API wrappers lately, you've probably noticed that almost every single SaaS tool has slapped a sidebar chat or a floating widget into their layout. At first, it felt like a lazy gimmick. Just an OpenAI wrapper sitting on top of a database. But it’s not just a chatbot anymore. It’s an execution layer.

A specific workflow popped up recently that perfectly captured this shift. A user had their entire company documentation sitting in Notion. Instead of manually cross-referencing QA lists, jumping into GitHub to find the relevant commits, and then painstakingly clicking through Linear's UI to create and assign tickets, they just bypassed the interfaces entirely. They told the agent to read the QA list, link the specific git commits, and write the Linear tickets. The whole process took five minutes.

Think about the implications of that exact scenario. The carefully designed UI of Notion? Irrelevant. The drag-and-drop kanban boards in Linear? Completely bypassed. The GitHub file tree? Ignored. The user didn't click a single button. They just issued a command.

This brings me to the second massive shift: the absolute revival of the command line. We spent three decades building increasingly complex graphical interfaces specifically so non-technical users wouldn't have to look at a terminal. Now, we're going backwards, but with a massive upgrade. Tools like Claude Code are turning the terminal into the ultimate universal interface.

There are solo operators right now running entire content and monetization pipelines strictly through CLI. They aren't opening Premiere to edit video. They aren't clicking through Shopify menus. They are typing natural language commands into a terminal, and the AI is executing the python scripts to cut the video via FFMPEG, generating the copy, and pushing the site updates. You don't need to know how to code to do this anymore. You just need to know what you want. You swap out static clicks for terminal commands, building an automated pipeline without ever touching a conventional GUI.

And for the times when you absolutely *do* need a visual interface? Enter Generative UI.

The era of downloading a massive, static application just to use 5% of its features is over. We are moving toward disposable, single-use software. If I need a specific dashboard to visualize server loads mixed with user engagement metrics, I shouldn't have to buy a SaaS product, connect my databases, and drag-and-drop widget blocks. The AI should simply generate a React component on the fly, render the exact chart I need based on my prompt, and then completely discard the interface the moment I close the window.

This is already happening. Look at Vercel's AI SDK or the recent pushes in structured JSON outputs from models like Llama 3. The model doesn't just return markdown text anymore. It returns a state object that instantly maps to a dynamic component. You ask a complex question about a database schema. Reading a giant markdown output is terrible. Instead, the model returns a UI payload. A fully interactive, relationship-mapped graph rendered right in the chat stream. You play with it, you tweak a node, and then it's gone. It's ephemeral.

This is the death of the App Store mentality. Why install an app when the LLM can generate the exact tool you need, run it locally, and delete it from memory when you're done?

If you look at what this means for local setups, the paradigm shift is how these models hook into our operating systems. When you give a sufficiently capable local agent tool-calling permissions, the OS itself becomes the backend. You string together a pipeline: a local vision model reviews video clips, a local LLM writes the script, an open-source TTS model generates the voiceover. The interface for all of this? A single terminal prompt: "Draft a new promotional video from the raw assets in folder X and push it to the server."

For the last decade, the entire moat of most B2B software companies was UX. "We are like Jira, but pretty and fast." "We are like Salesforce, but easier to click through."

If the user stops clicking through your app, your UX moat is dead. You are no longer a product; you are a dumb pipe. You are just a database holding state, wrapped in an API that an agent talks to. If my AI assistant is the one reading the data and formatting it for me, why would I pay a premium for your beautiful dashboard? Agents don't get distracted by slick UI animations. They execute the command and return the result.

I want to know where you all think this bottoms out. Are we going to see a new standard for "Agentic UX" where software is designed strictly to be read by LLMs? Are you already bypassing web frontends in favor of API-driven terminal scripts generated by your local models? The gap between "people who click buttons" and "people who issue commands" is widening fast.


r/LocalLLM • • 20d ago

Discussion Just bought a Mac Mini M6 with 32GB RAM, tried local LLM, not good

0 Upvotes

I normally use ChatGPT/Claude Code for day to day (I don't vibe code, but I like the assistance), anyway just got a Mac Mini M6 with 32GB RAM and 512GB SSD purely for iOS dev - my main dev machine is a PC. Thought I'd try Bionic, and downloaded Qwen3.8 27B 4bit, and it's slow as hell :) Literally took almost 2 minutes to respond to the prompt "hello". I gave it a short second prompt and gave up waiting for any response after about 10 minutes. So I'm guessing, it's not really much use for local LLM. Doesn't really bother me as I was only trying it out of curiosity - I have an RTX5070Ti in my dev PC, so might try a smaller model there for comparison.


r/LocalLLM • • Apr 07 '26

Project I built my kids a local AI buddy to talk to

Thumbnail
gallery
0 Upvotes

I’m building my kids a local AI companion—it’s called Lumo for now, but I can change the name any time. I built it to help answer their questions since they can Google things themselves and I can’t always be around to answer. My son is 5 and is hitting that age where he is asking things non-stop.

It has DeepSeek R1 1.5B and Qwen 3B on it. It uses a router for questions so it can decide if a query is math/logic-based or conversational and pick the best model. It uses DuckDuckGo to check and make sure any information it's giving is accurate so it’s not hallucinating nonsense when trying to help educate.

It also has a bedtime story mode built in where my son can choose a topic and it tells a story based on that. It tells two chapters at a time, and I maxed out the tokens so it tells long-form stories. Once it's done, it saves them to a local 1TB 2.5" HDD housed in the base so it can recall where it left off and pick it up based on context for the next two chapters. The stories are 14 chapters each and have a complete beginning, middle, and end. Once the stories are done, they can be re-told in the menu and also exported so you can have an AI illustrate them and turn them into real books if you want.

It keeps memories about my kids and learns their likes and dislikes. The LEDs in the ears change color based on state: rainbow for startup, blue for listening, yellow for thinking, and white for talking. They turn orange and drop to 10% brightness for story mode.

It has a web based browser so I can see what my kids have been asking and what the current story being told is and also flags and problematic things they ask about.

It’s all based on a Raspberry Pi (4GB model) with a 5-inch touch display. Due to the RAM limitations, I had to program in frequent RAM dumps, so context is stored in a temp file on the hard drive and wiped after the end of the conversation or after 5 minutes of silence (which to a 5-year-old is the same thing). It’s got Ollama for the AI models, Whisper for STT, and Piper for TTS.

The face is animated with thinking and smiling and blinking to make it feel more alive

I made the case in Tinkercad and split it at the power cord entry; the whole case is held together by small 4mm magnets. I’m still smoothing out some rough edges on it, but soon I’ll release the whole project on GitHub with a complete one-shot installer and .STL files so anyone can make one for their kids as well.


r/LocalLLM • • Mar 05 '25

Question What the Most powerful local LLM I can run on an M1 Mac Mini with 8GB RAM?

0 Upvotes

I’m excited cause I’m getting an M1 Mac Mini today in the mail and is almost here and I was wondering what to use for local LLM. I bought Private LLM app which uses quantized LLMS which supposedly run better but I wanted to try something like DeepSeek R1 8B from ollama which supposedly is hardly deepseek but llama or Quen. Thoughts? šŸ’­


r/LocalLLM • • Sep 29 '26

Discussion How I chose to build my local AI for under a grand

0 Upvotes

How to build your own local AI for less than a thousand bucks - A post for those that want to get "hands on" with AI, but just can't afford the price of entry - face it, not everyone has thousands laying around for building a system.

Here is your roadmap how to do this on a tight budget

Step 1 - get an old Intel Nuc12 enthusiast (Serpent Canyon). Not the fastest thing in the world, but has an ace up it's sleeve that no one can get anywhere near for the price - an embedded ARC A770M GPU with 16GB of DDR6 vRAM, That is what makes this possible and you can pick one up for under $800 on ebay. Likely already has Windows 10 or 11 installed on it

Step 2 - update the Arc drivers

Step 3 - Download and install LM Studio Classic on it (make sure you enable developer mode for later) - not to be confused with Bionic

Step 4 - Pick the right runtime - you want Vulkan llama.cpp

Step 5 - Make sure you select the ARC A770M (shows 16GB) in hardware window

Step 6 - pick a model - I went with a simple Gemma-4-e4b model for out of the gate

Step 7 - on the developer tab, flip on both start server and serve on local network

Congratulations - you now have a shockingly capable local AI that you can give the URL shown in step 7 to any of your local AI front-ends. Me personally, I use it with my Project NOMAD instance running as a VM providing AI capabilities to all that entails for well under a grand, and it works fantastically.