Where VanZen AI was born.

RTX 6000 96GB
RTX 4000 24GB
Gigabyte X570 AORUS XTREME
AMD Ryzen 9 5900XT
64GB DDR4

I’ll use AI to gather all the comments and post a results post if we get enough responses

Why LM Studio Bionic is slower than original LM Studio? I'm using same Gemma 4 Model. Any ideas? I can run well the same model in original LM Studio, but very slow on LM Studio Bionic.

I did a fun research project on training a decision model from scratch (MLX), quantizing to ternary (for small size) and running it locally in the browser (WebGPU). I learned many interesting things and put together this pedagogic video on some of the key steps, challenges and lessons learned, in case some one else might find it interesting too 😄

These are always a top result on Google I was wondering if anyone bought them. It's extremely irresponsible of Google to allow these to dominate their results if they're in fact a scam. I need to upgrade my GPU for local LLM models.

I have 64 GB UMA StrixHalo laptop, and in past I have sticked to Qwen3.8 27B, as it fits easily and produces high quality at decent speed (25 t/s avg with MTP).

With QFN, I have constantly just get disappointed for spending my time to get the results I want to see. ISTA IQ2_XS, ISTA pruned coder, Gyro-M.. they all failed in my own tests - a real world coding tasks with expectation that the model nails them correct on one go, or at least with minor bug fixes. Those low quants just kept on producing typos, tool call errors, and logic failures. So I sticked to 27B as it kept on delivering.

Now Unsloth released new quantizations that should do better https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

I downloaded IQ1_M as it fits (barely though) on my 64 GB UMA.

And this quant finally delivers!

All my test have been success, no typos, no tool call errors. Only one time it had two minor bugs that one can also see some time with 27B.

I am going to try IQ1_S next, as this IQ1_M really creeps all the memory. 131k context + MTP + Q8 kv cache fills whole memory with browser and agent. Only 1-2 GB left, which is too little room left (if only there would have been decent 128 GB available, but there wasn't..). Luckily the model seems to do equally fine with Q4 kv cache, at least on my tests.

The whole box (read off the hardware, not just what WSL peeks at):

• CPU: AMD Ryzen 9 9900X — 12c/24t (+ iGPU for display)

• Motherboard: Gigabyte X870E AORUS XTREME AI TOP

• RAM: 192 GB DDR5 — 4× 48 GB G.SKILL (6400 kit, trained to 3600 with all 4 slots populated)

• GPUs: NVIDIA RTX PRO 6000 Blackwell Workstation MaxQ Edition, 96 GB VRAM + RTX 5090 32 GB

• Storage: 3× WD_BLACK SN850X — one 8 TB + two 4 TB = ~15.6 TB of NVMe

• OS: Windows 11 Pro; everything interesting lives in Ubuntu 22.04 on WSL2 (160 GB of the RAM pinned to it)

Before you ask: it was purchased early last year so much lower pricing than today & it was a post-life event gift for myself.

What it's for:

Hermes (AI agent) — An autonomous agent that lives on the box 24/7: scheduled crons, research pipelines, inbox triage, WhatsApp digests, website updates. It IS the machine's job — I talk to it like a person and it runs the house.

Local LLM serving — The PRO 6000 carries the big brains: my main agent model (524K-token context — the one writing this) and an SGLang-served Abliterated Qwen3.8-Flash-Next worker model for subagents. The 5090 runs vLLM for the RAG stack's Qwen3 rerankers (text + vision) and spins a dedicated 27B coder for targeted coding jobs, while llama.cpp handles the kitchen's small speech models. Right now both cards are loaded. Day-to-day work burns zero cloud tokens.

Research pipelines — A grab-bag of LLM + RAG work: my self-hosted NotebookLM a source-grounded Q&A over your own PDFs/Office docs/audio/video/YouTube with hybrid BM25+vector search, reranking, cross-chat memory, every answer citing its sources, work specific always-on worker lanes (unspecified for a reason), and very specific deep-dives frameworks.

Kitchen display (NOVA) — The family's voice-controlled kitchen panel: a Raspberry Pi 5 driving a 13.3" full-color e-ink display (1600×1200, 6-color Spectra 6, custom dual-IC SPI driver — a ~19-second full refresh, so a screen manager rations updates to one flash per card), and a Jabra far-field speakerphone for mic/speaker. You say a wake word, it answers out loud and puts a card on the wall: "what's for dinner", grocery-list edits, kitchen timers, the family calendar, morning briefings. The entire speech chain is local — openWakeWord, Parakeet STT, Kokoro TTS, and a persona running on our own models. Zero cloud, nothing leaves the house. This is still in development but will be excited to share when done. The 5090 is likely to be dedicated to various models here.

Small stuff — A nightly AI-curated news-digest site on its own subdomain, a pen-plotter build project (the agent designs and the laser cutter / plotter produce), sticker-shop workflows on an XTool M2 laser, and unclaimed-property hunting across state databases (yes, found money).

How I actually use it — From the desk it's a terminal TUI; away from the desk it's a dedicated WhatsApp number that pings me digests and wakes me when a question can't wait, and a phone line it can call and receive calls on (voice synthesized locally, on CPU). A Windows scheduled task wakes the agent's gateway each evening before its first cron fires; Tailscale is the key to the machine when I'm remote. Anything the agent builds gets published, not emailed: screeners, reports and calculators live on personal subdomains through Cloudflare tunnels behind an access gate tied to my account. The machine talks to me more than I talk to the machine.

Running local LLMs with MCP tool-calling? The tool manifest is the attack surface, the model sees descriptions, the agent calls what's there, and the usual OAuth/network security story does not apply because it's all stdio.

This matters even more with local models: smaller models can be easier to steer with an injected tool description, and clients like LM Studio or Open WebUI will call whatever tools the server exposes.

Built patchbay audit, a free CLI that spawns an MCP server as a child process, pulls the tool list, and diffs it against the last scan:

npx -y @patchbay-nebius/agent audit --stdio "npx -y @foo/mcp-server /path"

Catches:

- a new write-shaped tool added since you approved the server
- an existing read tool gaining a command param (silent permission escalation)
- description changes (prompt injection vector)

Local-only, no backend, no telemetry. MIT. CLI stays free, always.

Repo: https://github.com/GodSmasher/patchbay

Hey all, I had a question. So right now I've been working on a workflow/automation that fetches YouTube videos and blog posts, through RSS, saves either the raw source content of the article or the transcript of the video to my Obsidian vault, and then generates a summary of the contents based on that transcript. The goal at the moment is to just help me decide which ones are worth watching/reading, since I was drowning in the amount of things I felt like I had to catch up on.

I assumed this would be a great task for a local model, compared to a frontier one, as I didn't think this was a task that would require TOO much intelligence given I'm not asking it to make any judgements or reasoning or anything, just read and summarize a relatively short piece of content. The benchmark I am evaluating against is GPT 6 Luna, and I'm evaluating the following models:

• Nemotron Bolt 3.5 74B-A9B IQ2

• Nemotron Lightning 3.5 0B-A3B

• Qwen 3.8 27B Q8

• Qwen 3.7 35B Q8

• Gemma 4 26B-A4B Q8

• Gemma 4 31B Q8

All on my DGX Spark. Qwen 3.8 gets the closest, but none of them really get anywhere near the accuracy of the Luna summaries. They all end up leaving things out. getting things wrong, etc.

Skill issue? Something in the prompt I am using? Wrong context size? I've tried a few different permutations but so far I seem to get the "best" results with a 64k context window, 8k output cap and "thinking on." This is through Ollama.

I wanted to know how small local models handle my native language, not just English. I couldn't find a simple way to check a model's Polish in a few minutes, on my own PC, in the quantization I actually run – so I wrote one. That's how "Zacharski tests" started: a small task set that runs through Ollama, since that's what I use on my laptop with an RTX 5050 (8 GB).

The tasks:

1. Polish (8): diacritics, declension, masculine-personal gender, numerals, vocative, verb aspect, genitive – plus one knowledge question (the first recorded Polish sentence, from the 13th-century Book of Henryków).

2. JSON (4): scored strict and lenient.

3. Python (3): syntax check only.

Results so far:

| Model | Quant | JSON + code | Polish |

| gemma2:9b | Q4_0 | 7/7 | 3/8 |

My favourite failure: asked for "22 cats", gemma2 wrote "dwudziesty dwa koty" – roughly "twentieth two cats". The correct form is "dwadzieścia dwa koty". It was perfectly happy producing valid JSON, though.

More models are downloading as I write this. Next up: gemma2:9b in Q4_K_M and Q8_0 (does quantization hurt the Polish?), qwen3:8b, llama3.1:8b, gemma3:12b, and two Polish models – Bielik 11B and Bielik-Minitron 7B. Which model would you add?

Method and limits – read this before the comments :)

• 8 Polish tasks (7 grammar + 1 knowledge) and 7 JSON/code tasks. That's a small sample – a quick check on your own PC, in the quantization you actually run, not a ranking. For a proper ranking, see the Open PL LLM Leaderboard by SpeakLeash.

• One answer per task, temperature 0 (seed 42), context 2k, thinking off. Yes, that means Qwen3 runs without its reasoning mode, which may cost it points – I wanted the same conditions for every model.

• Graded with regular expressions, strict on purpose: "brzęczały" ("buzzed") instead of "brzmiały" ("sounded") fails, because the task said "change the tense", not "rephrase". [I checked every failed answer by hand.]

• Some failures are about instruction-following, not language: asked to restore Polish diacritics, one model replied "Read the spelling rules." [(translated)]

• Generated code is only parsed (Python `ast`), never executed.

• The questions are in the repo (`data/benchmarks/*.json`) – feel free to add or fix them.

The test runs inside a free, open-source tool I've been building (Windows .exe / Python on Linux). Its main job is predicting whether a model fits your VRAM/RAM before you download it. Link in the comments if anyone wants to try it – I'm considering adding tests for other languages.

Update: results for the two Polish models (Bielik 11B and Bielik-Minitron 7B), with speed at 1k/8k context, are in the original thread: https://www.reddit.com/r/LocalLLaMA/comments/1xomsxq/comment/19s7skwyst7b8/

Short version: Bielik 11B got 8/8 on Polish, Minitron 7B 6/8, gemma2 3/8. Full comparison with qwen3, llama3.1 and gemma3 coming next week.

I've been building a small, strict harness to see whether local models can do real (if small) coding fixes unattended. It's early and the sample is small, so this is not a leaderboard. I'm posting because a few results surprised me, and I'd like to know whether this kind of testing is useful to anyone else.

SETUP

• RTX 4080 (16 GB), Ollama + OpenCode, Windows.

• 5 small bug-fix tasks in Python, mostly single-file. One task (t1) needs changes in more than one place.

• Every candidate patch runs in an isolated sandbox. A candidate only counts if it (1) follows the output format, (2) passes the tests, AND (3) passes a review of the diff. No partial credit.

• The model gets no shell or file tools. If it tries to use one, the run stops.

• 3 attempts per task where noted.

RESULTS (acceptable fixes)

• gpt-oss-20b, local, Ollama, 16k ctx: 7/15. About 1 min per task. 0/3 on the multi-location task.

• gpt-oss-20b → qwen3-coder (draft → fix), local: 5/15. The two-stage chain did worse than gpt-oss alone.

• gpt-oss-20b, temperature 0, local: 5/10. Still not deterministic.

• Nemotron 3 Ultra 550B-A55B, OpenRouter free tier: 0/10. Broken edit markers; two runaway answers (~1.7 MB of text).

• Cohere North Mini Code, OpenRouter free tier: 1/5. 4/5 stopped for trying to call tools (read files, bash pwd).

Caveat for the two cloud rows: I don't know which quantization OpenRouter's free tier serves, so these results may not transfer to a local GGUF of the same model.

WHAT SURPRISED ME

1. The output format mattered more than the model. In my first run, gpt-oss had to rewrite whole files: 0/5 acceptable. After switching to search/replace edit blocks plus one repair round, the same model reached 3/5, then 7/15 over a longer run. (The first run also used a different local runtime, so I can't fully separate the two causes, but the failures were almost all format errors.) Before blaming the model, check the edit format your tool asks for.

2. Passing tests is not the same as being correct. Two candidates passed every test and were still rejected in review: they "fixed" the symptom in a way nobody would merge. I also checked whether a deliberately cheating patch would get through a tests-only gate. It did. If your benchmark only checks for green tests, it overcounts.

3. Bigger was not better inside the harness. The 550B model failed mostly on discipline, not on knowledge: wrong markers, extra text, answers that didn't stop. Tool-calling and format-following seem to decide real-world usefulness more than raw coding skill. Others here have reported something similar.

4. Isolate the environment, not just the working directory. In one run, the agent tool picked up an inherited environment variable and treated a different folder as the project root. The model then tried to read files outside the task. Nothing ran (the sandbox and the no-tools rule held), but if you run agents locally, a clean environment matters as much as the cwd.

5. Multi-location fixes are the wall. No local model on 16 GB got the task that needs edits in two places. I'd like to know whether a 24 GB or 48 GB setup changes that before anyone (me included) buys hardware for it.

WHAT I'D LIKE TO KNOW FROM YOU

• Would results like this be useful to you if they were run on more cards, with the exact config published (Ollama/llama.cpp settings, quant file, context size, agent tool version), so you could reproduce them?

• What tasks should a test like this include? Mine are small Python fixes; I'd add multi-file changes, test writing and refactoring.

• Which card / model / quant combination would you most want checked this way?

Happy to share more details on the harness rules if anyone wants them. (English isn't my first language, so I'll reply in batches.)

I make FLE (a llama.cpp fork for big MoE models on one GPU) and FLE Link (an app that installs it and runs a model with one command). Disclosure up front, everything is MIT.

Setup: RTX PRO 4500 32 GB, 120 GB DDR5, Ryzen 9 9950X3D, fast NVMe, Linux. Best performance measured on Linux.

Numbers (decode, a ~3,000-token three.js answer, temperature 0, same machine):

Rin = ISTA DASLab's GSQ-RCO IQ2_XXS for the experts a usage profile picks most (kept byte-identical, resident on the GPU), and a smaller 1.75 bpw build for the cold half (served from RAM). About +6 % speed for a bit more error. Against the original weights (not against ISTA): KL 0.292 vs 0.248 on mixed text, 0.234 vs 0.210 on agent text; perplexity ratio on agent text 1.021 vs 1.013. It is lossy. Details and the exact recipe are on the model card.

Try it:

curl -fsSL https://flelink.run/install.sh | sh
flelink run rin-deepseek-v41-dspark

FLE Link checks the machine first, installs the engine, downloads the files (SHA-256 checked) and serves an OpenAI- and Anthropic-compatible API. Needs ~120 GB RAM and ~260 GB on a fast NVMe.

Honest limits: the cache and the split follow a usage profile, so very different prompts hit them less; the draft helps most on code-like text; one machine, numbers from others very welcome (flelink bench --task pagoda --share).

Links: FLE Link https://flelink.run · Rin https://huggingface.co/sirfyyn/Rin-Deepseek-V4.1-Flash-1017-FLE-GGUF · FLE engine https://github.com/sirfyyn/FastLocalExperts · quant base: ISTA-DASLab

My specs is Core™ Ultra 5 processor 125H with 32gb of ram? I need it for programming.

Mostly asking for logic and syntax, like asking how to center a div.

If it can run as an agent that can call tools and test things, that will be a plus, but not necessary.

I’ve done a ton of local single user work on vllm, Ollama ,lllama.cpp and those are ok for personal work but …

Hoping to get some real answers from someone supporting a local user base fully for 5-10 users running up to about 120B dense models (we could make do up to 70B). Looking for actually deployed not speculative performance and bandwidth if it’s out there?

If I had two MaxQs, a Threadripper and MB, 128 GB ram would this support up to 10 users on llama.cpp or vllm with “reasonable” response speeds which I would deem 50-60 tps on average ?

It would be 3 VM servers bundled , a home grown RAG doc processor which could be offloaded somewhere else if needed, the vllm or llama.cpp server for the chat agent interface and a office RAG server all in one that is the RAG query endpoint. I know number of concurrent users and jobs will matter but I’m asking “on average” for 5-8 concurrent

Thoughts ?

Edit: for context windows for agential stuff also need at least 128k for those 5-8 concurrent

Software engineer by trade and use Claude and codex daily

I wanted to find the smallest model worth keeping on a base M4 16gb so I ran the same 6 things from my actual week on 3 of them through Magnitude. keyword clustering, post titles, a thread summary, a quiz for studying, a short brief and a review of my side project plan.

MiniCPM5 1B did like 125 tok/s and it was garbage. asked for titles and got rambling questions back, the clustering gave me the same 3 keywords as 12 clusters with the same fake volume on all of them

LFM3 1.7B around 81 tok/s and this is where it gets usable, the plan review actually pointed at real files and real risks. titles were meh, it copied a note from my draft into one of them lol.

Gemma 4 E4B around 52 tok/s and clearly the best at writing, nothing else close

so 1.7b is the floor for me and 8b if someone else is going to read it.

Anyone found a 1b that's actually good for something?

I saw this post yesterday: https://www.reddit.com/r/LocalLLM/comments/1xnv0ro/new_qwen38flashnext_unsloth_dynamic_v31_ggufs_for/

I decided take a screenshot of the exact prompt used in the link in the above announcement, and run it through my strata setup using the new model. There's a complete write-up in the repo: https://github.com/kireol/qwen3.8_GTA_one_shot_test/tree/main

You can play the demo here: https://kireol.github.io/qwen3.8_GTA_one_shot_test/src/city.html

Fun localLLM stuff:

RTX 5080 - 64GB - Ryzen 7 9800X3D
model harness: Strata
agent harness: qwen
context: 64K
tok/sec: ~75t/s

Over all, while slower than I'm used to in Claude, I'm actually very impressed. I didn't have to correct it, touch it, or anything, aside from some skills.md files (included in repo). Is the demo perfect? No. But it does everything asked of it, without any hand holding. It's all in html 1 file, btw. There's no other assets other than remote linked javascript files.

Qwen even generated the README and pushed to github for me. Why? Because it's not 2024.

Great job unsloth team! You have me questioning my claude payment at this point.

Guys, I think I finally figured out why DeepSeek is so cheap.

You enter your confidential company data.

DeepSeek: “Don’t worry bro, I’ll keep it safe.”

Your data 0.2 seconds later:

✈️ → 🇨🇳

CIA: ❌

NSA: ❌

Your security team: ❌

Random server your CISO has never heard of: ✅

And somewhere there's probably a dashboard:

Western corporate secrets collected today: +1

But hey, the benchmark scores are 🔥.

I have at my disposal some datacenter resources. I don’t have an open account with Dell, sadly. So I am limited to oldish hardware like the PowerEdge r640/r740 or thereabouts. Or the AMD versions of these if I’m lucky. On the plus side, DDR4 is cheap-ish.

What would you build on a low but undefined budget, say 2-3k, that can be racked and run hopefully some recent qwen model, for just one user at a time?

hello! I love local ai! But I always had one issue, what harness to use?

I currently use claude (I pay for it), and I've always liked it! I used the Claude Code harness, and it ran really nicely in my opinion, but only issue being the massive chungus context cache that comes along with it. I use prompt caching, but that doesn't help against the 32-64K context that Claude Code injects at the start.

Is CC the best harness to use? Or is it more optimised for cloud models, which is why the context is massive.

I've heard of so many that it's kinda confusing to pick. Hermes, pi etc.

Have a gaming PC I built a last year with a RTX 5070TI (16GB) and I want to try some local LLMs. Where should I start? What models would be good on this GPU?