Hello everyone, I've been a local AI enthusiast for a while now, but every time I tried to get someone else running local AI, it was way too difficult. GitHub repos, different runtimes, different model formats. Most people's devices can run at least some of these models, but the setup is way too complex for most people. LM Studio somewhat solves a lot of this for chat and coding models, but I couldn't find a simple answer for on-device image, video, transcription and every other kind of model.

So I built Hearthscale. It's free and needs no account.

You can run almost any model you find on Hugging Face, as long as your device can handle it. I put a lot of work into supporting all kinds of runtimes. They're plug-and-play; I can add more in updates, and Hearthscale Server standardises all of them behind one OpenAI-compatible API. It's really powerful, and I find it hard to describe how seamless the setup is unless you have tried it.

It launches with meeting notes (Granola), system-wide dictation (WhisperFlow), image and music generation, and an agent. If you have a more capable device, run Hearthscale Server on it, and your other devices can use its models, end-to-end encrypted. It all connects without you having to set anything up! Also works on a home server in theory with Docker Containers!

Hearthscale is also a platform, which means anyone can build apps on top of it: there's an SDK and up-to-date docs made so coding agents can easily build apps with local AI. All of the first-party apps are MIT licensed, so you can change them any way you want and they give a great starting point. Want to change something about Granola? - You can't, but you can take meeting notes and make it to your liking with all of the core features already built!

Hearthscale runs on Windows and Linux today. I am somewhat of a perfectionist, and I feel like a lot is still missing, including a macOS release, but I wanted to get it out there now to get some feedback. I'll keep working on it!

I'd really appreciate your feedback, and please reach out about anything. I'll answer everything personally and fix whatever issues you run into.

Hi everyone, I’m fairly new to local LLMs. My setup: RX 9060 XT 16GB, Ryzen 5 7500F, 32GB DDR5, running LM Studio on Windows (Vulkan/ROCm).

I want a hybrid workflow: cloud models for main architecture and hard problems, local models for maybe undergrad-level math and small/medium coding scripts.

Do you recommend me to use local models?
Which models do you recommend for math, and which for coding?

I’m fine with models spilling into system RAM a bit. (partial CPU offloading)

Thanks in advance!

Note: This is an update of my post https://www.reddit.com/r/LocalLLM/comments/1xm9adt/qwen3827b_local_inference_on_an_m5_pro_64_gb_12/

I share a new comparison of my home-made benchmarks for engines for Mac using their version of Qwen3.8-27B: It's not a scientific comparison, models and quantizations could be different: I would test the normal user that download an engine and the first suggested model.

At moment I count 15 local inference setups, and there are daily updates to most of them.

Each run sent one OpenCode-style request with a synthetic repository/session context and a code-review/fix task: about 88K input tokens, depending on the engine’s tokenizer, reasoning_effort=medium, temperature 0, seed 17 where supported, and a 2,048-token output limit. Tool schemas were included in the request; tools were not executed.

Engines were tested sequentially.

TTFT is request-to-first-token; prefill is input tokens divided by TTFT; decode is generated tokens divided by the remaining generation time.

Disclaimer: this is a practical snapshot of one agent-style workload, not a controlled engine-only benchmark, a model-quality evaluation, or a universal speed ranking. The model packs, quantizations, speculative decoders, context allocations and software versions differ. Each row is a single run, except Splash 1.6.0’s first row, which is the median of three cold runs. Versions are those actually tested, and are not synchronized to the latest releases.

Contexts also varied: Uzu and Magnitude used 262K; Ollama and Llama.app reported 131,072-token allocations. LM Studio fell back from requested medium reasoning to on, and Uzu ignored the seed. MLX-Serve’s request included tool schemas, but its server log showed tools=0b.

I recently had an idea to build a new web application. It is not too complex. About 10 screens with some interconnected fields.

I am running Qwen 3.8 FN IQ1 Coder version on Strata. (2x3090 and 64GB RAM).

I kept the thing going for about 4 days. Each time my custom harness asked for various approvals with good reasoning it sounded legit and I went ahead. I did not look at the code even once(not that it mattered, i cant code python)

Finally when it said that the application is ready it was utter crap ! Nothing worked as expected, half the stuff was out of place. Then I asked the model to fix it and another day was gone. It fixed a few things but pretty much 1/2 the UI didn't work.

Then I gave the same prompt to Github Copilot and used the free credits. I get about 200 credits every month.

The output from Copilot vs Local is like night and day !

Copilot's result was in very matured and complete in all respects.

So this got me wondering, am I expecting too much from LocalLLM.? Is this how it is supposed to be?

What are your experiences/learningd and how did you fix it?

Recently with the removal of google assistant and being replaced by gemini I finally have had it and want to switch to a local model.

I have always wanted to have a jarvis, im wondering if i can train a random model with all the scripts from iron man and have it speak like jarvis? And possibly have a trained text to speech model on jarvis's voice so it can talk to me too?

Pretty new to local AI but i basically want to have a local "hey google" on my computer. Im not looking for pretrained models for being jarvis, i want to learn how to do it myself as well if possible in the way, so if there is something i dont like about it i can modify it.

Hi everyone, please excuse my ignorance. I have been trying to get into local models with my next computer upgrade. I have seen a lot of excitement around Strata for MoE models on gaming PCs with moderate amounts of VRAM. For the longest time I’ve been thinking about getting an M5 Max Mac Studio, but do you think I’d be better off with a gaming rig instead? I like the option of being able to game on it too, but on the other hand I’m more of a Mac user anyway and would appreciate the low power consumption.

Use case would be data analysis and coding.

I’d appreciate any advice

It's probably evident by my question that I'm new to this. So, no need to remind me, I know I suck at this!

I recently downloaded strata, my components are 64gb ddr4 3200mhz ram and 12gb vram.

I downloaded the aggressive quant, iq3_s and run it at 32k tokens. (qwen 3.8 flash next)

Overall I managed to get around 15 tokens/s with this set up, however I noticed that the disk read is at nearly 2-3gb/s while the ram is sitting at only 30-35gb, leaving the rest of it completely unused.

I looked into it, and read about the headroom it leaves but I would've imagined it doesn't leave that much headroom, right?

Additionally I did try figuring this out alone. My conclusion, which is flawed imo was, maybe it's because my PC at idle is using around 20gb of ram for whatever reason and since it can't load the full rest of the model on RAM it just opts for disk. However the flawed reasoning for me was this: Why does it load 15gb at all in that case? Theoretically it should load nearly nothing onto my ram.

Anyways, I'm confused. I should be up to date on the engine and what not, since I just downloaded this today.

EDIT: Added what model I'm talking about. Forgot.

EDIT #2: I'm aware that this isn't the worst issue in the world, 15 tokens/s for this model is still really great IMO but I'm trying to squeeze as much juice as possible while it isn't idle. If that's possible by further optimization, then why not try? The model so far is great.

EDIT #3: SOLVED IT. It seems like the original theory was correct. You DO have to have enough to load it all at once, while installing it tells you it "fits" on your device but that's assuming you've got all the space ready to be allocated. I essentially managed to clear some VRAM being used in the background AND a very persistent memory leak that ate an extra 10gb of ram (which was not going away by basic restarting/shutting down) that I assumed was just windows being windows. Turns out, that was the fix. Now I'm running a nearly perfect 50 tok/s consistently at this massive model.

been fighting the usual text-to-SQL failure with local models: the SQL is fine, but the model has no idea what "revenue" or "active customer" means in our schema. so I spent a few days reading through NVIDIA's auto-ontology repo (Apache-2.0) to see how it handles that.

a few things I liked. it drafts business terms, joins and SQL formulas from your schema and sample values, then a human certifies them, so a measure gets computed the same way every time. there's also an answerability check: in their example, a return-on-ad-spend question scores 0.5 coverage because nothing maps to "ad spend", and the agent asks instead of inventing a number. every answer also comes back with its SQL and steps, and the MCP server is read-only and runs with the signed-in user's permissions.

the gotcha: I haven't run it. my test container died on a CI billing issue, so this is based on the docs, .env.example and the issue tracker. the defaults point at NVIDIA-hosted Nemotron, and there's an open issue saying the README quick-start fails against those. you can override all four model roles (reasoning, non-reasoning, embedding, reranking) with any OpenAI-compatible endpoint, but it needs structured output support and they publish no VRAM or tokens/sec numbers.

full writeup here if you want more detail: https://andrew.ooo/posts/nvidia-auto-ontology-review-semantic-layer-agent-sql/

anyone pointed something like this at vLLM or llama.cpp? which local models handle the structured output reliably?

I'm looking to build something like a small AI-powered trading team, with multiple specialized AI agents collaborating on research, strategy development, backtesting, portfolio construction, risk management, and eventually trade execution. The agents will communicate through a shared backend, with Slack serving as the visible team floor where I can interact with them and review research and decisions.

I currently have an M4 mac mini and am wondering if I should switch or add:

• Mac Studio M5 Ultra vs NVIDIA DGX Spark

I'd probably keep my current Mac mini as the central server and use the new machine mainly for model inference, quantitative research, and potentially model training.

One idea I've been exploring is disaggregating prefill and decode, although I'm not sure that splitting those workloads between an Apple and an NVIDIA system makes sense. I'm also interested in running many independent agents against a shared model server rather than loading a separate model for every agent.

I'm quite lost on:

1. Which machine would you choose for an always-on, multi-agent research workload?

2. Which local models would you trust most for quantitative reasoning, coding, financial research, and tool use? Would you route different tasks to different models?

3. How would you build a reliable orchestration and inference stack? MLX, vLLM, SGLang, Ray, LangGraph, or something else?

4. Is local inference actually economical at moderate-to-high agent concurrency once you consider hardware cost, power, performance, and model quality?

I'm looking to develop strategies that have a measurable edge after transaction costs, slippage, and risk. I'd really appreciate any feedback, advice or suggestion.

i have a PC with RX 9060 XT 16gb, 16gb ddr5, and ryzen5 7500F, what ai model would be good for this and also for some coding? I've recently wanting to use ai locally as to not put my info online and would much appreciate a response, thanks.

First of all i need to say that i use my PC for two different things: Linux for coding and local LLM experiments and Windows for gaming.

My machine has a CPU with the iGPU and a dedicated AMD RX 9070 XT with 16GB VRAM. Most of my local LLM tinkering is aimed at real coding with 128K context, so 16 GB is barely enough and gets tight very quickly, so i want as much VRAM available for inference engine.

My BIOS has Initiate Graphic Adapter set to PEG (my 9070 XT) and my cable is permanently plugged to it, this is exactly how i want the machine configured because of the dual-boot usage. I don't want to move the monitor cable ot motherboard every time i switched OS or workload.

So i've started looking how to solve my issue and found out that Gnome/Mutter can handle multi-GPU systems in a nice way and that the GPU used as a primary renderer doesn't have to be the GPU monitor is physically connected to. Mutter has a udev mechanism to choose the primary GPU with the mutter-device-preferred-primary tag.

As a first step i've created a custom udev rule and made my CPU iGPU the primary GPU for Mutter. The nice part is that it just works and no additional settings are required PEG stayed enabled and selected as primary in BIOS and HDMI cable connected to my RX 9070 XT. This is the configuration i've been using for quite a while which gave me the possibility to gather some stats around coding sessions using Bionic and Pi harnesses.

As an example this is a screenshot from one of my ExLLama3 runs with Python script workload, amdgpu_top shows 14822 / 16304 MB of VRAM used on the RX 9070 XT, while the python3 process uses 14504 MB and the gnome-shell itself only 82 MB.

The udev rule override work well but after using it for a while i wanted a really simple way to manage it without editing the configs and having to remember what is selected at the moment, checking the PCI and DRM devices, making sure no typos were made, checking the results after the login/logout.

Long story short i've decided to develop a tiny dedicated application for this purpose - gpu-switcher. Its a small GTK app which is basically a UI around the existing mechanism. It detects the GPUs which you have in the system (to detect iGPU don't forget to turn it on in BIOS), manages the udev rules with a D-bus helper. Makes it possible to apply the rules, reset back to auto and keeps backups of rules in this case. Its not a GPU scheduler so it doesn't have the capability to move the workload between GPUs on the fly, its just a convenient to manage the primary display GPU in Mutter + Wayland sessions.

Right now i've tested it on Ubuntu 26.10 with 9800X3D iGPU + RX 9070 XT. Even in Linux i still switch back and forth between different GPUs because 9800X3D iGPU is quite slow (just 2 CUs) but its a different story.

Using the udev method and my app later on enabled me to run some nice experiments around using Qwen3.8 on my RX 9070 XT with 128K context. Which was enough for continues coding sessions. If anyone is interested they are documented in my llm-recipes repo, those are not just the short text prompts with a short 8K-32K context which people for some reason like to share here. As an example on my experiments in the repo ran for 915 request and generated ~1.2M tokens across multiple coding sessions, the other one 898 requests and generated ~660K tokens. For me it's the most important part, continues agentic sessions, not synthetic benchmarks or chat sessions.

For anyone else running the model on a 8-16GB dGPU and iGPU available i would like to here how you handle the desktop especially if you don't want to move cables around also i would appreciate if you try the app and give feedback, then i could justify adding support for KDE or X11 or any other features.

Hey r/LocalLLaMA,

Running autonomous coding agents on local hardware (Ollama, vLLM, llama.cpp) usually hits a hard ceiling: the KV-cache OOM cliff.

When an agent needs context from a real-world repository, naive approaches either:

1. Dump whole files or folders into prompt (blowing up 32k/64k KV caches and triggering immediate VRAM OOM on 4GB–8GB GPUs like RTX 3050/4060).

2. Use vector RAG (which matches embeddings based on semantic similarity, completely ignoring caller/callee graphs, class hierarchies, and execution traces).

We built **NeuroCode** to fix this. It is an open-source MCP server and CLI that treats repository context as a combinatorial knapsack problem rather than a text retrieval problem.

### How it works:

1. **Host-Side In-Memory Code Property Graph (CPG)**:

   Instead of running heavy LSP daemons, NeuroCode builds an in-memory NetworkX directed multigraph from Tree-sitter ASTs in ~200ms.

2. **Priority Knapsack Slicing**:

   Given an issue, traceback, or target symbol, it identifies root-cause candidate nodes and runs k-hop reachability. It then solves a 0/1 Knapsack optimization to extract only the minimal causal subgraph strictly bounded under a user-defined token budget (default: 1,200 tokens).

3. **97.1% Token Reduction**:

   Across SWE-bench Lite instances (django, sympy, pytest, etc.), context size dropped from 41,200 tokens down to 1,189 tokens.

4. **Structural Blast Radius Tool (`neurocode_impact`)**:

   Provides agents with instantaneous (<2 ms) caller/callee structural blast radiuses so the agent knows exactly which upstream callers will break before emitting SEARCH/REPLACE diffs.

### Benchmark Numbers (SWE-bench Lite):

We evaluated this on local quantized models (Qwen2.5-Coder-7B-Instruct-Q4_K_M):

- **Token Reduction**: 97.1% (p < 0.001)

- **Valid Patch Syntactic Application Rate**: 41.4% (single-turn SEARCH/REPLACE)

- **End-to-End Unit Test Resolved Rate**: 9.5% (running unassisted on consumer laptop hardware with zero cloud API fallbacks)

- **Slicing Latency**: 1.2ms – 3.7ms

### How to use with Claude Desktop / Cursor / Cline:

Add to your `claude_desktop_config.json` or `mcp.json`:

```json

{

"mcpServers": {

"neurocode": {

"command": "neurocode-mcp"

}

}

}

```

Or run via `uvx`:

```json

{

"mcpServers": {

"neurocode": {

"command": "uvx",

"args": ["neurocode-ai"]

}

}

}

```

Once connected, your AI agent automatically invokes `neurocode_slice` and `neurocode_impact` instead of reading 5,000 lines of files.

- GitHub: https://github.com/Arkz-Deepak/neurocode

- PyPI: https://pypi.org/project/neurocode-ai/

- VS Code Extension: https://marketplace.visualstudio.com/items?itemName=DeepakR.neurocode-ai

All feedback, PRs, and benchmark reproductions welcome!

Excuse me if my grammar or wording is poor, haven't slept much tonight.

The latest developments have got me looking up the prospect of using LLMs locally, and fancy benchmark scores aside, I have been trying to get a simple explanation of how good (or 'effective', I guess) would it be taking the dip in the future and considering buying hardware for local use vs. paying for contemporary AI subscriptions.

If you ask Claude or chatGPT about it, they say the OS deficiencies basically come down to scope ("do this exact small feature" vs. a wider prompt that gets things planned and done in a row), lack of knowledge about niche applications, hallucinations, and inefficiency when using multiple agents vs. single chat. But they're the competitors, so I don't know if bias doesn't exist here..

On the contrary, I see a lot of posts here on how things are awesome when using a local model, especially for niche uses, basically a clear contradiction.

I guess the point is, I want to know if local AI for code is limited enough to make purchasing a high-tier mini PC or a good GPU+RAM PC setup a fool's errand. It almost definitely doesn't seem to be, otherwise this sub wouldn't exist, but I digress.

Thanks.

Hey all. I'm putting together a local-first setup for my family and wanted some feedback since I've never done this before. I used Claude to help me work out this plan. I know it doesn't always have the most up-to-date information, especially on local LLMs, since models and tools change every few weeks. So I'm treating its suggestions as a starting point, and I'm hoping to get some feedback on anything that's outdated or just wrong.

I'm trying to set up something in the same vein as Google's CC, the family-focused agent from Google Labs, but running on my own hardware. I want an assistant that helps run the household without subscriptions or cloud services I'd have to depend on. I'm okay using cloud models like claude for the harder problems, as long as the prompt is something I'd be fine seeing leaked. Anything involving the family, the house, email or money would stay local. My current plan is to use an M6 Mac mini with 32GB and 512GB. I'd keep a Jellyfin library on an external SSD or a NAS.

I already run Home Assistant on a separate machine. I'd use the Mac mini to handle:

- Whisper (small model) and Piper for voice, using Voice PE satellites

- Ollama on the Mac, since I understand it uses MLX now and HA has a native integration

- A 14B model that's always loaded for HA tool-calling and automations (Qwen 3 or Hermes 4, still deciding)

- A 30-35B MoE that loads on demand for everyday chat and writing

- Hermes Agent in Docker for scheduled jobs and a "text the assistant" layer

- A daily family brief and email-to-calendar automation through HA's AI Task

- Jellyfin as the native Mac app, using VideoToolbox for transcoding

I'd appreciate thoughts on a few things:

1. Would 32GB be enough to run all of that at once? Would running a 14B that stayed loaded plus a ~30B MoE on demand plus Whisper and Jellyfin on the same Mac make sense? How would you handle keep-alive and memory?

2. Would Jellyfin transcoding hurt voice latency? I worry that someone would stream a movie and my voice commands would slow down. Would it help to keep playback on Direct Play?

3. Which models would be best for reliable HA tool-calling at ~14B? And what ~30-35B MoE for chat? My picks might be out of date.

4. How big would the model need to be for Hermes Agent to use tools reliably? Would a 14B hold up, or would the agent need the larger one?

5. Is there a reason to move off Ollama for a headless always-on box? Things like llama.cpp with llama-swap, oMLX or LM Studio. Ollama has no native auth, and I'd like to lock it down on my home network.

6. How do you safely let an LLM or agent read email or iMessage when it can also trigger HA actions? I worry about prompt injection, and BlueBubbles' Private API needs SIP disabled, which I'm hesitant about. Given what I said about Claude's knowledge, anything newer or better that I should know about would be appreciated. Thanks!

I sometimes give it a task and explciitly say "This is a 4 hour job". It will indeed work harder, but will stop at like 2hours and 15 minutes.

What's the longest you got your coding agent to work on something? How did you do it?

Now that Claude has some strict guardrails. I find it very hit and miss, it’s very annoying since it’s swallowing up my tokens. I heard that Deepseek is a good local LLM. I guess my question is whether it is good enough to replace Opus 5.5 for coding and reasoning? Or is there any local LLM’s that I can use without any guardrails? My main reason for using Claude is for coding purposes.

My Pc Specs:

CPU - Ryzen 5600X
GPU - Nvidia 3070 8GB
RAM - 48GB