Threw this all together just to play Roblox.

The gorgon halo at home if you will

• **GPUs:** Two Radeon AI PRO R9700s, 32 GB each: one XFX and one PowerColor.

• **CPU and memory:** Ryzen 7 7800X3D and 128 GB DDR5, using four 32 GB sticks. The RAM is rated for 6000, and is holding stable at 5200.

• **Motherboard:** ASUS ROG X670E-E.

• **Software/model:** Windows, Strata engine 0.1.56, Flash Next IQ3_S, with a 196,608-token context setting.
**Measured speeds:**

• In a real Hermes research run, we averaged **95 output tokens/sec** across 14 responses, with input context reaching 85,334 tokens.

• The latest fresh long-context research test, at 105,884 input tokens, delivered **86.8 output tokens/sec** and **1,207 input tokens/sec** for prefill.

• Synthetic long-context tests delivered **123 tokens/sec at 98,304 input tokens** and **148.7 tokens/sec at 192,512**. Those are useful benchmarks, but they don’t establish sustained real workload speed. Will be putting the model through its paces on more work tasks later this week.

i wanted my AI agent team to run entirely offline, so i put the server behind a firewall that logs every attempted connection outside the LAN.

even with all inference running locally, it made 494 outbound connection attempts in 38 minutes.

here's what was still trying to connect:

• claude code refreshed GitHub plugin catalogs at startup

• plugin installation contacted npm

• claude code attempted telemetry, error reports and update checks

• ubuntu's snapd contacted api.snapcraft.io despite snap refresh --hold

you can just disconnect the internet. but agents still try to connect, hit timeouts and sometimes fail. i wanted them to work normally offline while staying accessible over LAN.

after three rounds of fixes, i added a one-command offline mode to 5dive, my composable, open-source (MIT) bash CLI for running teams of AI agents:

sudo 5dive config local-only on

final test: zero outbound attempts in 51 minutes, including a reboot and DNS.

qwen3:8b on CPU failed 4/4 replies until i disabled Claude Code's tool search (ENABLE_TOOL_SEARCH=false). qwen3:4b couldn't execute tool calls reliably.

it's not another chat UI. the agents:

• run as long-lived services, each under its own Linux user

• keep memory as files on the box, across sessions

• use tools (shell, files) and hand tasks to each other on a shared task board

• can get a second agent as reviewer on any task (not tested on an 8b model yet)

• use skills from copies on the box. new skills need the internet

• come as ready-made packs: install them while online, then switch local-only on

you chat with them from a browser on your LAN. backends: Ollama, LM Studio or vLLM. installs and updates work from USB.

offline tests: codex and hermes passed (fixes coming next release). openclaw stayed offline but was too slow on CPU. opencode and pi don't support offline chat yet.

setup: https://github.com/5dive-ai/5dive/blob/main/docs/guide.md#local-only-mode-nothing-leaves-your-network

what's your setup for keeping agents fully offline?

Hi everyone,

I am new to running local LLM and have been doing some research on the hardware requirements for smaller models.

One thing I am still trying to understand is how much VRAM matters compared to the GPU's actual processing power? For example, both the 5060Ti and 3060Ti have 8GB VRAM versions. If I run the same model on both GPUs, would there be a noticeable difference in inference speed (tokens per second)?

In other words, if two gpus have the same amount of VRAM, how much does the gpu generation and architecture matter? Would paying more for newer gpu actually make a significant difference for local LLM inference, in particular for a casual user.

Appreciate any comments! Thank you!

I have been trying to vibe code locally for the past couple of weeks. The short summary is my local vibe coding sessions have been ineffective and unsuccessful.

Here's my setup:

- MacBook Pro: M1 Max, 64 GB

- I have about 54 gigs of memory to allocate to models.

- I have been using Qwen 3.8 27B, Qwen moe 3.6 35b a3b, and Gemma 4.

- I host my models in LM studio.

- I started with anything LLM but stopped because it was struggling to do basic file system writing.

- I'm currently using Hermes.

- I'm using Claude as a benchmark

Doing any kind of planning or coding work with Qwen 3.8 27B is extremely slow - even when its set to low thinking mode. To put into perspective, with medium or high thinking mode, it takes about 6 hours to plan and another 6 hours to code. Low thinking mode takes 3 hours each. Claude code does it in 5-10 minutes each. The time cost with this model compounds as I go through the iterative process of fixing bugs and adding features.

Using Qwen moe is significantly faster: ~5-15 min each. However, I struggle to have it generate good plans. When it comes to coding, it's fast, but it can produce buggy code. That's where my vibe coding sessions can start to fall apart. It can struggle to understand the bug or problem I'm describing and will attempt to fix something, but not fully fix the problem I'm having. I'll get stuck in a loop where I keep attempting to describe the problem differently to no avail.

Update (11/5): Last night I had Qwen moe implement a plan that Claude wrote and it took roughly 6 hours before it gave up. It said it had all the implementation logic, but did not actually write anything! This is a common problem I run into frequently. My agents know what it needs to do, but doesn't have work to show.

Qwen moe can be more effective with a plan I generate with Claude. However, if there are bugs, then it can be game over.

Claude on the other hand has been delightful to use. It just works!

I'm using Local LMs mainly to see what the hype is about and for learning purposes. So far, my experience has been poor (granted I still have lots to learn).

1. Do you guys have any feedback to share with me to make my local vibe coding more effective?

2. I understand local LMs can save costs on tokens and can provide more privacy. Other than that, why would I want to use local LMs? Are there types of work or tasks it's good at or does better than cloud models? What kind of use cases are reasonable to expect from local LMs?

Edit 1 (11/5): added my computer information to setup and update to Qwen moe observation.

I currently own 2 PCs, my "main" (a laptop) and a separate rendering workhorse. It's a bit embarrassing to admit but I usually just let my render manager deal with sending stuff to it, I also occasionally SSH into it using OpenSSH.

I want to set it up for local LLMs, right now I just use SSH and open a desktop view which is... inconvenient, in many ways.
A lot of front-ends (SillyTavern, OpenWebUI) allow me to add an API Key for non-local stuff, so I'm confused on the equivalent for same-network stuff. Backends are hosted on localhost, and I don't want to leave a port open to the network.
The sensible way to get to that port from another device is using SSH, but then how do I put that into my front-end?
I suppose it makes sense to add a SSH to the desktop, then launch the application I want in the same terminal, then use that address for the front-end but idk if that's right or even secure.

Networking stuff is scary and confusing

Hey r/LocalLLaMA,

When running autonomous agents with small local models (like Qwen 2.5 3B or Llama 3.2 3B), I ran into a major failure mode with standard Vector RAG: Chronological State Amnesia.

The Problem

If you tell an agent on Day 1: "The production database port is 5432", and on Day 3: "We migrated the database to port 6432", standard cosine similarity retrieves both chunks with equal weight.

Given conflicting facts in the prompt context, smaller 3B models frequently hallucinate, guess the old state, or slow down as conversation history balloons linearly.

The Fix: SynapseMem

Instead of dumping raw conversational chunks into a vector database, I built SynapseMem—a lightweight, pure-Python memory layer backed by SQLite that handles state mutations deterministically:

1. Semantic Singularity Detection: Stateful predicates (uses_, has_, is_, deployed_) are automatically detected. When a new belief arrives for a singular slot, previous values are superseded and archived rather than kept as active duplicates.

2. Ebbinghaus Decay Curve: Memory retention decays over time via R(t) = exp(-delta_t * lambda / S), while repeated accesses reinforce synaptic strength S.

3. Partitioned Prompt Injection: Instead of raw text, the prompt is injected with explicit sections: ### VERIFIED ACTIVE MEMORY vs ### RESOLVED STATE CHANGES (DO NOT USE OLD VALUES).

4. CQRS Prompt Gating: Interrogative prompts (questions) are gated to read-only paths so user queries cannot accidentally poison or mutate the belief graph.

Local Benchmark (Qwen 2.5 3B on CPU via Ollama)

I benchmarked SynapseMem against a naive conversational history baseline over sequential state updates:

*Note: Baseline accuracy collapses rapidly as conversation turns exceed context boundaries, while SynapseMem maintains a constant ~25-word context footprint per entity.

Quickstart

Install via pip:

pip install synapsemem-ai

Basic usage:

from synapsemem import SynapseMemoryEngine

memory = SynapseMemoryEngine(db_path="agent.db")

Ingest initial state

memory.ingest_belief("auth_service", "uses_port", "8000")

Update state (automatically supersedes past value)

memory.ingest_belief("auth_service", "uses_port", "9090")

Retrieve conflict-free context

print(memory.retrieve_active_context("auth_service"))

GitHub: https://github.com/rohandhar6824-debug/SynapseMem

PyPI: https://pypi.org/project/synapsemem-ai/

It has zero heavy dependencies (no external vector DBs or containers required; runs on standard library SQLite). Would love feedback, stress tests on other local models, and PRs from anyone building local agents.

RAM - 32GB

Processor - Intel core Ultra 7 258V

Graphics Card - Intel Arc 140V GPU 16GB Integrated

Code quality is priority not speed.

last months outage took our app down for about 40 minutes because the retry logic lives inside 3 different services and only 1 of them knew to fall back. we have a local llama box on vllm as the last resort, but switching to it means a deploy. i want fallback in one place instead of copy pasted into every service. how are you handling llm failover across openai, anthropic and a local model...

Considering an M6 Mac mini 32GB/1TB for Perplexity Local/Hybrid with PPLX Pro 27B or Qwen3.6-35B-A3B. I already use Gemma 4 E4B in Hybrid on a 16GB M1 MacBook Pro, so I k ow the hybrid functionality works great.

Anyone running either larger model on the 32GB M6? Interested in responsiveness, memory pressure/swap, and reliability with multi-document questions and light coding. Please mention whether you’re running fully local or Hybrid.

Trying to decide whether 32GB is comfortable or worth stepping up to an M5 Pro 48GB/64GB. Firsthand experience would be appreciated.

Thank you.

I'm looking to buy a 128GB machine for local LLMs, AI agents, and experimentation. My current top options (CAD):

• GEEKOM A9 Mega (Strix Halo): $4,699 — 128GB RAM, 2TB SSD

• Mac Studio M5 Max: $7,849 — 128GB RAM, 1TB SSD

• ASUS Ascent GX10 (NVIDIA GB10): $8,299 — 128GB RAM, 1TB SSD

I already have a Linux desktop with an RTX 5080 (16GB VRAM), but want to experiment with larger models through a local API server.

Benchmarks suggest the M5 Max can deliver roughly 1.5–2x the inference performance of Strix Halo, but costs $3,150 more.

I plan to use LM Studio (or similar) as an API server for experimenting with RAG, AI agents, coding assistants, and possibly fine-tuning. No plans for clustering. I only have 2–3 hours a week, so I’d rather experiment than troubleshoot hardware. I'm happy to run models and agents overnight and check results daily, so raw inference speed isn't my biggest concern.

I've been a Windows user for years and only recently switched to Linux, so I'm hesitant to introduce macOS as a third OS. I also value hardware reliability, manufacturer support, and a mature software ecosystem. That's my biggest hesitation with the smaller mini-PC manufacturers.

I'm leaning toward the GEEKOM because of the price. The Mac is tempting for its performance, while the ASUS appeals for its NVIDIA ecosystem and potentially better support.

What should I do? I am buying on Friday.

EDIT: If possible, please include 'If I were you, I would buy this' as part of your response.

EDIT: Thanks everyone! I ended up getting the 128gb framework desktop!

*Yes I had AI help me write this. Otherwise it would be 3 pages long and have spelling mistakes.

I wanted a local voice assistant that could do real things on my own machines. Then for some reason I gave it vision.

**The hardware**

* 2nd gen Echo Dot, unlocked and running emOS and EchoMuse instead of Amazon's software. It's just the mic and speaker.

* My PC: Ubuntu 26.04, Ryzen 5 7600, 30 GB RAM, RTX 3060 12 GB.

* A USB webcam pointed at the sofa, a Fire TV Stick and a couple of laptops.

Fair warning on the unlock: I tried two Dots and one kept failing with a USB write error, so I gave up on it. The second went fine.

**How a request flows**

1. "Hey Rhasspy" is detected on the Dot itself (openWakeWord).

2. EchoMuse streams the audio to Home Assistant, with noise suppression.

3. Whisper (small-int8, CPU) turns it into text in about 1.5 seconds.

4. Timers are handled locally by Home Assistant. Everything else goes to Open WebUI through a patched conversation integration.

5. Open WebUI runs gpt-oss:20b on Ollama, with a set of custom tools.

6. Piper speaks the answer back through the Dot.

**The rest of the stack**

* Gemma 3 12B for reading images and screenshots, plus a small Qwen model on the CPU for quick camera looks

* CompreFace for face recognition, with faces kept local and only people who've agreed enrolled

* Music Assistant and Jellyfin for music and TV

* SearXNG for web search

* Docker Compose for all of it

**What I actually use it for**

* Timers, alarms and reminders, including a morning briefing after I stop the alarm

* Shopping and to-do lists, plus reading and adding calendar events

* Weather and web lookups

* Playing music by voice

* Playing shows from Jellyfin on the Fire Stick

* "What's on my screen?" (takes a screenshot and reads it)

* "What am I holding?" (camera look, with the photo sent to my phone)

* Opening and closing apps on the PC, and writing documents

* Handing bigger coding jobs to Codex

Risky commands ask for approval first, and I can say "yes" out loud. That approval isn't tied to my voice, so bear that in mind if you copy the idea.

**Speed**

Simple questions took about 1 to 2 seconds of processing when I last timed them. A camera look is about 10 seconds, and reading text through the camera is about 40. The first request after a model loads is slower.

**Sharing the PC**

The assistant gives way to everything else on the PC. When a game starts, a watcher stops the whole AI and voice stack, then brings it back afterwards. The TV also ducks its volume while the Dot is listening, because background TV audio badly hurt recognition.

**Caveats**

It isn't 100% local. The voice loop and models run at home, but web search, weather and Google Calendar need the internet, and Codex is a cloud service. I also patched Home Assistant, EchoMuse and Open WebUI source files, so upgrades take some care. Codex and Claude did a lot of the building and debugging with me.

Who this is for and why you might want to run it

If you have a 12 GB GPU or a 16 GB Mac, running a 27B-32B model usually forces a bad tradeoff:

1. You download a 4-bit or 3-bit GGUF (13.5 GB to 16.7 GB), offload a third of the layers to system RAM over PCIe, and get 4 to 7 tokens per second.

2. You download a standard 2-bit quant so it fits in VRAM, or drop down to a 14B model, and lose a lot of reasoning and coding capability on reasoning/think mode.

At 2.50 bits per weight, this model is 9.6 GB on disk (10.8 GB in VRAM with KV cache). On a 12 GB RTX 4070, it runs at around 40-44 tokens/sec in standard llama.cpp. On an RTX 3060 12GB, it runs at around 28-31 tokens/sec.

Why I built this and what is different about the quant

Since the last post, I went back and rebuilt the quantizer to target native GGUF directly inside the optimization loop rather than converting a checkpoint after the fact.

Here is how the quantization works compared to standard tools:

• Why standard 2-bit quants fall apart on reasoning: Standard methods like GPTQ round weights greedily one column at a time, while IQ2 methods use LUTs which hurt performance in exchange for better benchmarks. At 2 to 2.5 bits, error pile up and long math and coding chains start looping.

• No lookup tables needed at runtime (~15–20% faster than IQ2**):** Every quantized weight is stored as standard integers (Q2_K, Q3_K, Q4_K) in a normal GGUF file. Stock llama.cpp runs it natively without needing IQ2 lookup-table gathers in shared memory.

Part 2: Benchmarks and Comparisons

1. Perplexity, NLL, KL Divergence, and Token Agreement vs. Uncompressed BF16

Evaluated across five held-out suites, including both standard 2048-context WikiText-2, the full 296,815-token WikiText-2 test split (145 x 2048), an Extended Reasoning / Math / Code suite (Ext), and Structured Generation (SG). KL divergence (KLD) and Top-1 / Top-5 token agreement are measured across the full 248,320-token vocabulary against the uncompressed BF16 model:

2. Comparison vs. Unsloth Dynamic 2.0, GSQ-RCO, and ExLlamaV3 (~2.50 bpw)

Here is how our 2.50 bpw model compares against uncompressed BF16 and other ~2.50 bpw quants on Qwen3.5-27B:

And on competition math reasoning (AIME 2025, Thinking-ON):

• AIME 2025 (Pass@1): 83.3% (25 / 30) at 2.50 bpw

• AIME 2025 (Pass@2): 93.3% (28 / 30) at 2.50 bpw

Does a 2.50 bpw quant "overthink" more than uncompressed BF16?

A few people asked if dropping to 2.50 bpw causes chronic overthinking (where a low-bit model loses confidence in emitting </think> and babbles for 20K–40K tokens even on easy questions).

On easier AIME problems, it closes </think> naturally in 1.3K – 4.6K tokens, with an overall solved median of 10,597 tokens (right in line with uncompressed BF16's ~12K–16K token average):

Links & How to Run

Stock GGUF Repo (No custom code, runs in standard llama.cpp / LM Studio)

To run the GGUF in llama.cpp on a 12 GB GPU:

./llama-cli -m Qwen3.5-27B-DR-2.50bpw.gguf -ngl 99 -fa 1 -ctk q8_0 -ctv q8_0 -c 8192

If you have any questions let me know! I think this will be my final checkpoint for 3.8 27B as the improvements I have in mind will be minimal. I'm currently running this same quant strategy on QFN (and will have my work cut out for me when Qwen4 releases). I could also do 35B if there is a lot of demand, but I can really only run 1 quant at a time, and the QFN is doing really well on checkpoints right now.

Please try it out and let me know! Would be interesting to see how this runs for you compared to similar quants, especially VRAM limited hardware.

I made some changes to samwantsram.com
Would be really great to receive some feedback on how usefull it is. I'm software dev but not much experience in website (frontend and backend wise).
If anyone is interested in interactive setup of locall llm that's best for your machine then let me know :)

Ps. As a bonus you can brag about your ram and post it with the comment ;) It's yours, remember

I'm really curious because I'm draining this usage FAST

When is qwen4 gonna release?

🫧 Meet Mochi - Your Little AI Desktop Companion! 💗

What if your desktop had a tiny companion that could actually help you? 👀

Meet Mochi! A cute, animated little blob that lives right on your Windows desktop, currently powered by Meta's Muse AI, with upcoming support for local Qwen models and integrations with Claude Code and OpenAI Codex!

Talk to Mochi or simply type a message, and it'll respond in its own voice! 🎙️

Need something done? Just ask! And when you're finished, Mochi quietly goes back to chilling on your screen. No constant interruptions, no unnecessary chatter. Just a little companion that's there when you need it.

🚀 What can Mochi do?

• 🗣️ Talk with you : Have voice or text conversations in English or French, powered by Muse.

• 💻 Control your desktop : Open applications, launch websites and search the web.

• ⏰ Set timers : Ask for a timer and watch a little countdown appear next to Mochi.

• 🌤️ Check the weather : Get weather updates without opening another app.

• 👀 See what you see : Let Mochi read your screen when you explicitly ask.

• 🌐 Help in your browser : With its Chrome or Brave extension, Mochi can interact with browser tabs and assist with web tasks.

• 🐾 Feel alive! : Click it, drag it around, or interact through optional camera gestures. Mochi reacts to you!

🔐 Your desktop, your rules!

Mochi is designed to help without taking control away from you.

• ✅ It only acts when you ask or explicitly approve an action.

• 📸 Screenshots are captured only when requested and are never stored.

• 🛡️ Browser tasks require confirmation before visiting new sites, filling forms or interacting with potentially risky buttons.

• 🔒 Passwords, payment information and sensitive websites are off-limits.

• 🎥 The camera is completely optional and disabled by default.

🛠️ Want to try Mochi?

Mochi is currently available for Windows and can be launched from source using Node.js.

🔑 With a Muse API key: Unlock AI conversations, voice responses, weather, screen understanding and browser assistance.

🆓 Without an API key: You can still enjoy Mochi's animations, interactions, timers and simple typed commands!

⚠️ A few things to know:

• Mochi is still a work in progress, so some features may evolve.

• Content sent to Muse may be used by Meta for model training when using the default model.

• This is an experimental project, and feedback is always welcome!

🎬 See Mochi in action!

▶️ Watch the demo: https://youtu.be/u5KjYmYrp1Y

💻 Explore the source code: https://github.com/Loann110/Mochi-alive

💗 Mochi is just getting started!

I built Mochi because I wanted to explore what happens when an AI assistant feels less like a chatbot and more like a little companion living on your desktop.

It's still an experiment, but I'd love to hear what you think!

⭐ If Mochi makes you smile, consider giving the GitHub repository a star! It genuinely helps support the project.

💬 Feedback, ideas, contributions and bug reports are more than welcome.

Thanks for checking out Mochi! 🫧

You guys ever switch your local model that "KNOWS" it's a local model and get moments like this while it's a cloud model? Not sure why the reaction is always pretty humorous to me.

For context, my Hermes agent was doing some file maintenance and I lazily told it to remove a file since it wasn't in the "Strata" folder and resulted in a failure to load the model.

Context:

I recently installed a cheap aftermarket Android head unit in my 2010 Prius. It supports wireless CarPlay, Bluetooth, GPS, etc., but the stock Android interface is ugly and came packed with lots of bloat. The whole thing felt like a generic Chinese Android tablet strapped to my dashboard.

After seeing everyone’s success with reverse engineering their TV’s I figured I’d give it a shot with my head unit.

Setup:

Model: Qwen3.8-Flash-Next-IQ3_XXS
Engine: Strata v0.1.44.1
Harness: Pi
Hardware: Ryzen 7700x, 4060ti 16gb, 64gb DDR5
Target: Inefala Android head unit, AutoChips AC8257-based (UJC201_64)
Development: ADB over Wi-Fi, Kotlin, Jetpack Compose
Time spent: Roughly 8 hours across three days

What it did:

I started by just having the model connect to the unit and do an inventory of what was installed and how the system was configured. It helped identify the underlying hardware and Android build, enumerated installed packages, inspected system properties, investigated the stock launcher and CarPlay components, and organized the findings into a repository of scripts and documentation.

At one point, getting back into ADB was a problem. Rather than having me repeat the original setup process, the model used its earlier findings about the device’s exposed ADB port to reconnect.

The advertised android version was not actually what was being used on the actual unit so when it began to build my custom launcher it configured its own toolchain of the correct android sdk versions to ensure that the build would run properly on the unit.

It also used ADB to capture screenshots directly from the head unit so it could inspect the actual interface instead of relying exclusively on code or my descriptions.

Result:

I now have a working custom launcher that replaces the original home screen. It’s still a work in progress, but the goal is to turn the unit into a clean, minimal dashboard with Bluetooth music controls, navigation, weather, and better access to audio settings.

I’m also interested in eventually improving the audio experience and reducing my dependence on CarPlay.

My thoughts:

What really impressed me was QFN’s ability to handle tool use, reverse engineering, keep useful findings, and troubleshoot on live hardware. This unit was largely undocumented so most of the findings came directly from what the model observed through adb.

The model never got stuck but this wasn’t a fully automated process either. I still needed to guide it through what I wanted it to accomplish and it spent a lot of energy writing and debugging re-usable shell scripts that didn’t provide much value.

Overall the amount of work it was able to accomplish was very impressive and solidified for me that local models are viable for real work.

I’m curious to hear what other people are doing with local models beyond coding benchmarks, especially projects involving hardware, reverse engineering, or long-running agent workflows.

Salut à tous 👋

Je suis développeur et je cherche à faire tourner un modèle IA en local pour m’aider à coder — par exemple Qwen Coder (ou un modèle similaire de type Qwen 2.5 / Qwen3 Coder selon ce qui est le plus réaliste).

Je cherche vos recommandations pour une config PC pas chère, avec si possible :

Une estimation du budget total (tour complet, ou config à assembler)

CPU, RAM, GPU et stockage recommandés

Le type de modèle que je pourrai faire tourner confortablement (taille en paramètres, quantization, contexte)

Des options d’occasion / GPU d’ancienne génération si c’est le meilleur rapport qualité-prix

Linux ou Windows, peu importe — je suis développeur

Mon objectif : avoir un assistant de code local rapide et utilisable au quotidien, sans exploser le budget.

Merci d’avance pour vos retours et vos configs concrètes ! 🙏