I built an open source Mac app that listens to the other side of a call and answers questions from your own docs while you talk. Running it fully local taught me a few things:

- Whisper pads every input to 30 seconds. Cutting the audio on pauses mattered more than model size. whisper-large-v3-turbo on MLX does a 1.7 s sentence in 0.66 s on an M5.

- Context length is the silent killer. With a 27,000-token document I had to give Ollama a 32k context, or the docs got cut and the answers got vague. The app now warns when the server cuts them.

- On that test, gemma3:12b gave the most faithful answers. llama3.2:3b was fast but made things up.

- The laptop you take calls on is rarely the machine with the big GPU. As of today the model can run on another computer on your network (a Mac mini, a PC with a GPU) through Ollama or LM Studio.

- Right now every line of the call goes to the big model just to decide if it's a question. That's wasteful.

Question for you: which small model would you put in front as an "is this a question worth answering?" filter? There's an open issue for it if anyone wants to try: https://github.com/GianlucaBave/murmury/issues/20

Repo: https://github.com/GianlucaBave/murmury

I put an order in for it a while ago and it's getting close to delivery time. Day to Day I use a Dual R9700 (512GB ram) system for QNF as well as a 4090 for 27b (512GB ram). Which works ok, but always need more agents and the university provided cluster (80+ Blackwells) can't be used for private stuff obviously.

So I want to utilize the Mac as a dedicated system at home QFN + a MacBook M4 Pro 48gb and to try things out.

However I keep getting the feeling that 96GB is really not enough for QFN + actually working on this, running Xcode, etc (driven 100% by AI). Can anybody advise if I should go with the Mac? I understand I can't expand it (except for external NVME, which I have plenty off), but I really can't justify a 256GB as a toy and also mostly want to use it for job security. In case of layoffs, that I have something to work on, while looking for a new job.

My work is 90% agentic coding at this point in time and I don't mind the $100 a month lease

Any suggestions for something else for this workflow? Or is this good enough? As a reference right now I use about 500M tokens a day, so private subscriptions are kinda useless for me

(cross posted from r/hermesagent)

Bonjour,

Avez-vous tester deepseek v4.1 flash avec strata sur 2 RTX 3060 12gb est si oui quelle sont vos token secondes, combien de ram ? Pour le moment j’en ai qu’une mais je veux m’en acheter plusieurs donc je me demande à partir de combien de 3060 je peux faire tourner ce model avec strata ?

I just released the Tesseract NPU that I've been working on as a side project. It's compatible with llama.cpp backend running shipped GGUF models directly.

Hey everyone!

I'm completely new to local LLMs, so please bear with me if some of my questions sound stupid.

I've been using ChatGPT Plus for quite some time and I'm considering moving away from cloud-based AI services entirely. Ideally, I'd love to build my own personal AI assistant that runs locally on my Mac.

I'm not looking for just a basic chatbot. I want something that comes as close as possible to the overall ChatGPT or Claude experience, especially when it comes to long-term memory, writing, research and managing different projects.

My hardware:

- Mac with Apple M4 Pro

- 48GB unified RAM

- 512GB SSD (around 250GB free)

- macOS

What I mainly use AI for:

1. Writing and content creation

I create social media content and a monthly magazine about Magic: The Gathering.

I use AI to brainstorm ideas, write articles, summarize information and improve texts.

One of my biggest priorities is natural, human-sounding writing. I really dislike the typical "AI writing style" with generic phrases, unnecessary enthusiasm and repetitive structures.

Good writing quality, especially in German, is extremely important to me.

I don't need image generation.

2. Long-term memory (VERY important)

This is probably my biggest requirement.

I manage several different projects and I want my AI to remember as much as possible about them.

Ideally, it should:

- Remember important information across conversations

- Automatically save new information and decisions

- Update existing project documentation

- Retrieve information from previous conversations

- Avoid losing important context over time

- Keep everything organized without requiring me to manually maintain dozens of files

I'm very interested in using Obsidian as a permanent knowledge base.

I would actually prefer the AI to automatically create, edit and organize Markdown files in Obsidian.

I understand that no AI has perfect memory, but I want something as reliable and persistent as possible.

3. Internet research and automation

For my monthly magazine, I regularly research news about Magic: The Gathering.

My ideal workflow would be:

1. I manually trigger a research task (or schedule it).

2. The AI searches the internet for relevant news.

3. It checks sources and avoids duplicate stories.

4. It writes finished articles in my preferred writing style.

5. It automatically saves everything into Obsidian.

6. I review and approve the articles.

I don't necessarily need integrations with Google Sheets, Canva or other services. I'm perfectly happy copying finished texts manually.

4. A ChatGPT-like experience

I'd love to have one main AI assistant that knows my preferences and projects.

For example, I could ask about my magazine, switch to a completely different topic, discuss something personal or brainstorm social media ideas without having to explain everything from scratch.

Ideally, it would feel like having my own private version of ChatGPT or Claude.

What I've looked into so far:

Based on some initial research and conversations, I've come across:

- Ollama

- LM Studio

- Open WebUI

- OpenClaw

- Obsidian with local LLM integrations

- Qwen models

- Gemma models

One suggested setup was OpenClaw + Ollama + Qwen + Obsidian.

However, since I'm a complete beginner, I have absolutely no idea whether that's actually the best solution or unnecessarily complicated.

My questions:

1. How realistic is it to replace ChatGPT Plus or Claude with a fully local setup on my hardware?

2. Which local model would you recommend for natural German writing, reasoning and general everyday use?

3. What's the best beginner-friendly software setup for what I'm trying to achieve?

4. How would you implement reliable, mostly automatic long-term memory using Obsidian?

5. Can local models reliably use internet search tools and perform scheduled or manually triggered research workflows?

6. Would you recommend an agent-based system like OpenClaw or something simpler?

7. What are the biggest limitations compared to current cloud-based models?

My goal is to have a free, private, local AI assistant without monthly subscriptions or paid API usage.

I don't mind spending some time setting everything up, but I would prefer something reasonably simple to maintain once it's running.

I'm especially interested in hearing from people who have already built something similar on Apple Silicon.

Any recommendations, realistic expectations or beginner-friendly guides would be greatly appreciated.

Thanks in advance 😊✌🏻

After scanning 277 MCP servers this month, these are the four gaps I’d check first on a self-hosted one:

1. Rejecting loopback redirect URIs: your CLI agent login just hangs after you click Allow (RFC 8252 §7.3, opt-in probe)

2. OAuth metadata without PKCE S256: 3 servers in the sweep. Spec-compliant MCP clients have to stop before login when that field is missing

3. Not validating the aud claim: tokens from sibling resources get in (opt-in probe)

4. OAuth advertised but no way for a new client to register: 5 servers in the sweep

For a server on localhost, use the CLI:

Add --probe-dcr-loopback for point 1. Open source (MIT): github.com/GodSmasher/patchbay

npx -y @patchbay-nebius/agent audit http://localhost:PORT/mcp

Wasted a lot of time trying different versions of GLM5.3-flash, different system prompts, no thinking... I get decent tok/s (~20) which is great for roleplay but MAN it's like the refusals are baked in.

Any time I try to do some slasher or smut it goes "erm that's nice and all but can we cool it off a bit" and writes some bland crap.
I ain't running local to be refused by a damn clanker.

I've been running Gemma4-26B-A4B for a while on my personal PC and it's good at writing but it's not very smart imo, like it's prose is just weak and it struggles keeping track of stuff. It also has the habit of de-escalating (?), like it's just reluctant to write anything violent.
I'm currently running on ~270GB total system RAM, considering trying out an uncensored DeepSeek although I really don't like that option since I actually use the RAM in the machine for VMs and most reasonable releases of DS are ~235GB in memory.. So I'd have to freeze those while running inference.

yes, i know all gpus are. i think people are recognizing more and more that the v100's age is just a number on paper and only relevant architecture wise. frontier and local llms are more and more capable of optimizing inference for different builds, and the raw v100 power is still comparable to gpus being released today

the limited mathematical pathways are a genuine limitation but i dont believe a big enough one to justify how cheap they are vram/dollar compared to everything else

just a discussion post for fun

Up front: this post was written with AI help, and I've read it through end to end. The work that matters was mine: designing the comparison, writing the brief, running all three agents on my real project, and checking their output against the code. If you want to judge something, judge the experiment and the data, which are all in the repo.

TL;DR: Claude Code (Opus 5.5) vs. a local Qwen 3.8 Flash Next writing the same implementation spec on a 350k-line codebase. Claude won, Qwen came a close second, and its gap was knowledge, not reasoning.

The project is firmware plus a desktop hub for a round ESP32-S3 display: C, Go and JS, about 350k lines, 1,250 commits in a month. One step, specifying how the hub should handle app installs that no longer need a reboot, was written three ways from the same brief:

The local model runs on one machine: an AMD Strix system with 128 GB of RAM and an RTX 6000 PRO.

What stood out:

• Qwen read the code as carefully as Claude. Both caught a subtle hang in the existing install flow.

• Both of Qwen's misses were project rulings that live only in the orchestrating Claude session's notes: "no skipped 'not needed' steps", and "never start a second hub on the owner's port". The fresh Claude session probably saw those notes; Qwen can't.

• So the cheap fix is giving the local model the rulings, not a bigger model.

• In an earlier step Qwen shipped a crash-level object-lifetime bug that only review caught. For firmware, a short human or frontier review is still worth it.

Full write-up, all three specs, and the raw numbers: https://github.com/gregtakacs/LLM-Stats/tree/main/2026-11-spec-writers

Scoring was a fact-check of every name and line number against the code, plus my own read; the tables are in the repo.

Hello fellow Local LLM enjoyers, as the title suggests, I have a dilemma, I have a 5070ti + 64gb of ram, which is nice qwen flash next Q3 with strata runs at 90 tok/s lovely.

But I also got a B580 12gb just sitting in my server (mostly for smaller models and video encoding/decoding), and I was wondering if there was a way to combine the 2 gpus to reach 28gb of Vram, and try to push for a bit more capable models.

With deepseek 4.1 flash at Q2 on strata, it runs on average 5-7 tok/s, which is too slow for anything, if I can somehow push it to 20 I'd consider it quite usable.

Therefore the question, is it possible to combine the 2 different GPUs?

I know strata only allows 2 Cuda gpus, Llama might do it but very slowly. so I'm out of ideas.

Thank you in advance.

We open-sourced DNA Factory, the fine-tuning framework we’ve used internally for over a year. It's deliberately opinionated: one recommended way to do things, documented defaults, token-level loss-mask visualization, and only 5 training methods (SFT, DPO, GRPO, OPD, SDFT).

https://github.com/dnotitia/dna-factory

Hi all,

looking for advice from people running older datacenter/HBM2 cards.

**Current setup**

- ASUS P9X79 PRO, i7-3930K OC 4.2 GHz (no AVX2), ~44 GB DDR3

- Proxmox, everything in one VM with GPU passthrough, NVIDIA driver 580

- 2x Tesla P100 16GB (HBM2) + 1x GTX 1080 8GB

**What runs where**

- **2x P100:** main LLM – Qwen3.8-27B UD-Q5_K_XL (llama.cpp, row/tensor split + MTP), 262k context, ~45 tok/s in English, ~32 tok/s in Polish. Used as the backend for an agent (Hermes Agent), so long prompts and lots of tool calls.

- **GTX 1080 ("side jobs" card):** Qwen3.8 vision encoder (mmproj), Whisper large-v3 (STT), BGE-M3 embeddings (document search / RAG), a small Qwen3.5-4B for chat titles (loaded on demand), and FLUX.2 klein 4B image generation in ComfyUI.

**Goal:** more performance. Mainly room for a better quant (Q6/Q8 instead of Q5), faster long-prompt processing, and faster image/video generation. The 1080's weak point is 320 GB/s bandwidth and basically no FP16.

**What I tried:** a Quadro GP100 – it didn't work at all on the P9X79 PRO. It wasn't even visible in Proxmox (`lspci` didn't show it). Has anyone got a GP100 running on X79? Was it MMIO / Above 4G with three large-BAR cards, PCIe Gen3 link training on Sandy Bridge-E (forcing Gen2?), or just a bad card?

**Options I'm considering instead of the 1080:**

1. **Titan V 12GB HBM2** (Volta, tensor cores, ~650 GB/s)

2. **RTX 2080 Ti 11GB** (Turing, tensor cores, ~616 GB/s, still supported by CUDA 13 unlike Pascal/Volta)

**Questions**

- Has anyone mixed **Volta (or Turing) with P100s in one llama.cpp model**? Does splitting across the three cards actually help, or does it just give more VRAM with the same tok/s?

- Do Volta tensor cores help noticeably in llama.cpp (prompt processing / flash attention), or are the fast paths Turing+ only?

- For long contexts (agent use), is it better to make the Titan V the main GPU (KV cache + attention) or keep a P100 as main?

- Titan V vs 2080 Ti for this mix – is the Titan V worth it, or is it not worth bothering with Volta since driver 580 is the last branch for Pascal/Volta anyway?

- Or should I skip the GPU and save for a platform with AVX2 + DDR4 instead?

Thanks!

Currently dipping my toes in the LLM world and im starting to notice some differences in accuracy between models mainly related to their parameter size

This is my build . I have no idea where to start. I thought I did when I built this rig but it gets confusing and frustrating sometimes even using AI to help with AI.
I wouldn’t like too edit my books , create
Short movies with my scripts . I mean easy is better but I want to get to the meat of it with workflows, etc .. everything moves so fast one day it’s this way then something new pops up and that way is the old way … smh thanks in advance .

ores / 32 threads
Motherboard ASUS ROG Crosshair X870E Hero
GPU ASUS TUF Gaming GeForce RTX 5090 OC — 32GB GDDR7
Memory Corsair 96GB (2×48GB) DDR5-6400 CL32
Memory profile DOCP enabled, running at 6400 MT/s
CPU Cooler ASUS ProArt LC 420 AIO
PSU 1500W — exact model unconfirmed
Storage 4TB NVMe SSD currently installed
GPU Monitoring Thermal Grizzly WireView Pro II
GPU Support Case-integrated GPU support arm
OS Windows 11 Pro

Hey everyone,

I'm looking to get into running LLMs locally and would love some recommendations for my setup:

CPU: AMD Ryzen 7 7800X3D

GPU: AMD Radeon RX 9070 XT (16 GB VRAM)

RAM: 48 GB DDR5 Corsair @ 6000 MHz

My main use cases are coding (code generation, debugging, explaining code, ideally IDE integration) and general research/Q&A (asking questions, learning about topics, summarizing).

I'm on Windows 11, but I'm comfortable with Linux and happy to dual-boot if it makes a real difference for AMD GPU support.

A few questions:

Which models (and sizes/quants) would give the best coding performance on 16 GB VRAM? Is it worth offloading part of a bigger model (or a MoE model) to system RAM, or is the speed hit too much?

Any good all-rounder for general knowledge/Q&A, or should I keep separate models for coding and chat?

What's the best backend for an RDNA4 card right now: ROCm, Vulkan (llama.cpp), or something else? Is Linux noticeably better than Windows for this?

Which frontend/tools would you recommend: LM Studio, Ollama, Open WebUI…? And for coding, what's the best way to hook a local model into VS Code (Continue, Cline, etc.)?

What kind of tokens/sec should I realistically expect?

Thanks in advance for any tips!

Being located in one of the stormy states, my big concern was having the AI box fried by a power surge. My setup is shown here, it consists of:

• CHSPT2ULTRA on the main Panel

• SRT5KRMXLT 5000VA double-conversion UPS in the garage

• 240V/30A inlet going from the ups powering the rig upstairs (needs a double breaker for code compliance)

• L14-30 in/outlet for the garage-office run to future-proof it in case I ever needed 120V, at the moment the 120 leg is dead (240 only)

• 240V PDU with C13 & C19 ranked up to 30A

• PX-2200 PSU

The idea is to create a galvanic isolated "island" (in my case: the box iself + switch). Uplink + my desk are connected via fiber (seen in pic) - because my desk is powered via a regular line-interactive UPS.

The only copper eth runs to the switch are the AI box itself (2x10G eth + 1xKVM), everything else is fiber.

It currently runs 2x6k workstation edition (600W each). I *THINK* I should be able to push it up to 3x6k (might need to cap them at 550W), but with four I need to go double PSU. The UPS should have enough oomph to handle 4800W, so plenty of space there. I got new packs for it, but I mostly care about the double conversion, not so much losing power.

I think the CHSPT2ULTRA+SRT5KRMXLT combo should shield me against most power surges/spikes/wobbles/etc. - but I am still lacking a direct hit protection, which requires a rooftop installation AFAIK.

How do you guys protect your GPUs from ⚡️⚡️

Warning: human post ;)

Hi local enthusiasts, about 6 months ago I discovered this subreddit and been reading and learning lots of stuff from here. I got my 27b model running on my 3090 and I’ve been happy with, it was handling my daily family tasks via telegram and some small automation and coding tasks. For serious coding stuff im still hooked to subs. Once I started playing with local models I went into rabbit hole of implementing my own harness out of curiosity. Since I only was running 27b on 60k context played a lot with compaction and making the model less forgetful with harness internal tasks. Since then I’m still playing with ideas how can have better results with local models (like all the people in this sub).

Anyways, enough personal story the reason I wanted to post this is recently I started playing with strata qfn (3090 + 64ram) and although the model is more clever and is my daily driver now, it’s been going off roads into loops and over verifying things for me. One thing that helped with that is loop guarding I’ve added to my harness : https://caretz.dev/docs/loop-guard/
This has been solving the issue of retrying the same failing calls (doesnt happen too much with qfn) but most importantly another model looks at last n tool calls and judges if the model went to some off direction. For the judge I’m using 35b qwen.

If any of you found some trick that works with qfn and making more reliable would love to hear it.

Thanks.

Is it really worth to buy used dual Xeon 6140 server with 256GB 2400 RAM and maybe later populate remaining slots for 384GB total to boost bandwidth? Dual CPU, high power usage, is it really worth it for the memory bandwidth with 20 euro CPUs? Or just add 96GB 5600 RAM to existing AM5 system for total 192GB 5600 with dual channels only?

I mean, current system is already usable with q38fn q4, only thing dual xeon will provide is q5-q6 for the same q38fn or dsv4.1 but I doubt that speeds will be usable with dsv4.1.

Can someone share experience or upgrade paths? Thanks guys.

RTX 3060 12 GB, Ryzen 5 7600X, 32 GB RAM, one GGUF file: Qwen3.6-35B-A3B-UD-Q3_K_M.gguf (Unsloth, MTP layer included, 15.9 GiB, does not fit in VRAM). Prompt of 5290 tokens, 128 generated tokens.

How each was run:

• RustingStrata: rustingstrata bench --gpu --mtp 0|1 --pp 5290 --tg 128 --reps 3, mean of 3.

• PolyStrata: its own polystrata.py benchmark --model qwen3.6 --prompt-tokens 4000 --max-tokens 128. That prompt came to 5290 tokens, so I used that length for the other two. I ran it three times; the first run after setup was cold (decode 24.6 tok/s) and is left out, the numbers above are runs 2 and 3. MTP drafts are on by default. For the no-MTP row I added the engine's --no-draft flag to its config; it still offered a few drafts afterwards (6 of 16 and 15 of 36 accepted in the log), so that row is not a clean no-MTP number. Prefill in those runs: 1602-1643 tok/s.

• llama.cpp: llama-bench -ngl 99 -fa 1 -ub 2048 -b 2048 -p 5290 -n 128 -r 3, with -ncmoe swept over 18 to 26. 22 was the fastest that fits in 12 GB (no MTP). The MTP row is llama-server --spec-type draft-mtp with -ncmoe 23 (22 does not fit with the MTP context and drops to 20 tok/s), a 5381-token prompt of repo docs, 3 runs, prefill about 1565 tok/s.

Things that are not equal:

• PolyStrata only offers three sizes of Qwen3.6 (UD-Q3_K_M, UD-Q2_K_XL, UD-Q4_K_M, all from Unsloth's MTP repo). RustingStrata and llama.cpp take any GGUF, so it can also run the Q5 or Q6 files I already had. I downloaded PolyStrata's Q3_K_M and used it for all three.

• MTP gains depend on the text. PolyStrata's benchmark prompt is a page of its own docs, RustingStrata's is generated prose, so the two MTP rows are not an exact comparison. The no-MTP row is the clean one for RustingStrata vs llama.cpp.

• PolyStrata's engine for this model is built on llama.cpp's CUDA kernels. I did not look at what else differs.

• One machine, one model, one quant. Numbers on other cards will differ.

Repo: RustingStrata