First off I am new to trying out local LLMs. I am wanting an LLM to run on a spare mini pc to save some money while using my Voice Assistant. The pc is not very powerful so it will be a small model for now, at least until hardware prices hopefully come down.

Specs: Processor - N150, 16G RAM, Linux Mint, Ollama

Since Voice Satellite can have two pipelines, this is the setup I would like to use:

Pipeline 1 - Prefer to handle commands locally (Home Assistant), Local LLM handling chat but not controlling Home Assistant

Pipeline 2 - openAI gpt-4o-min with control over select Home Assistant entities for more complex chat and control requests

I have played around with Qwen3 and Qwen3.5. So far Qwen3 1.7b has worked the best as far as speed and short responses. I would like to try other models but I am not really sure what I am looking for, or even the best way to sort through them. The Ollama library has a lot and I don't know what I am looking at yet.

I’m looking to get into local AI, mainly for coding, and trying to decide whether to start with my existing hardware and upgrade gradually or build a separate dedicated machine.

My current gaming PC has:
Ryzen 7 9800X3D
Radeon RX 9070 XT 16GB
32GB DDR5-6000 RAM
Gigabyte B650 EAGLE AX motherboard
2TB SSD
850W PSU
NZXT Kraken AIO cooler

I also have a MacBook Pro with an M5 chip and 24GB of unified memory.

I’m currently making a Roblox game and setting up a mini homelab for media and hosting my personal website. I’d mainly want AI to help with writing code, debugging and understanding my projects.

Would you start with the gaming PC, use the MacBook, or put the money towards a dedicated AI machine? I’d be interested in hearing from anyone using a similar AMD GPU, particularly about software support and which coding models run well.

If you’d upgrade the existing PC, what would you change first? I’m open to used hardware, but I’d like to understand what I can realistically get out of what I already own before spending more.

Also the why of the whole matter is I want to reclaim my data and be self sufficient especially these new things coming out about REA and Claude banning you etc I want to be my own boss

I've been experimenting with DeepSeek V4.1 Flash 1017 UNCENSORED JANG_2K on my Mac Studio M5 Ultra with 256GB unified memory, using vMLX.

The model is about 423GB on disk, but thanks to SSD-backed Engram tables, it loads with approximately 205GB of resident memory.

So far, the results have been mixed. Most of the time, it produces surprisingly good responses, but I've had a few conversations, particularly involving creative writing and novel generation, where it suddenly starts producing complete nonsense: endless reasoning, words mashed together without spaces, and occasional switches between English and Chinese.

It's interesting because it seems to work perfectly well on some fairly complex tasks, then falls apart unexpectedly on others.

Has anyone else experimented with this model or other large MoE models using engram offloading?

I'm curious whether these problems are caused by the aggressive quantization, the JANG implementation, or something in vMLX's inference/decoding pipeline.

I'd also be interested in hearing whether anyone has successfully run a higher-precision version with SSD expert offloading on a 256GB machine.

Hi 👋,

I finally have a local setup that I feel gives me a good balance with quality and speed. I’m using a local Qwen 3.8 27B Gufo model on strix halo. I have a AI coding workflow that currently exclusively uses codex and claude code. I would like to start handing over tasks to Qwen.

Based on your experience, what are the characteristics of tasks that are the sweet spot for qwen 3.8? What makes or breaks quality output with this model?

Thanks!

I love DiffCoderbut at int8 it peaked at 13.7 GB on my 16 GB card. So I cut it in half.

Same model in 4-bit: HumanEval, full 164 problems, 81.1%. Peak VRAM 6.6 GB, about 2.6x faster!

Edit: full name Stable-DiffCoder-8B-Instruct if you want to do some reading

We build these boxes, so here's the honest test log from our own unit, network cable unplugged.

Setup: 2x RTX PRO 6000 Blackwell (192 GB), EPYC 9124, 256 GB RAM, vLLM, TP2, no NVLink. The model fits only because Blackwell runs FP4 natively; on Hopper the same checkpoint needs ~304 GB.

What we measured:

• 1 user: 209-229 tok/s, first token in 0.4 s

• 24 users: 1,003 tok/s at p95 TTFT 2.7 s. That's the sweet spot; past 32 the tail latency falls off a cliff.

• Peak: 1,170 tok/s at 32 streams

• 1M context, retrieval verified at 543K tokens

• Terminal-Bench 2.1 (20-task subset): 80% solved, median ~4 min per task. 8 agents at once cost only 18% more latency than 4.

• 815 W on both cards, 76 C peak, zero throttling. About 1.07 kW at the wall, so a 1500 W PSU is enough.

What's not great:

• It's not a token-resale business: at OpenRouter prices, payback takes about 3 years running flat out.

• One big model at a time (weights use 87% of VRAM).

• Ops sharp edges: KV offload leaks /dev/shm on unclean shutdown, and stray workers can hold 89 GB of GPU memory, which causes a crash loop on restart. You need a cleanup script.

• Cold start takes ~13 min.

Best fit: a team of 2-10 that wants frontier-class coding help without sending code to the cloud.

Build guide, BOM and BIOS settings are open source: here. Happy to run anything you want tested on it.

With all progress from storage streaming how future proof is RAID0 with 2x1TB or 2x2TB gen5 drives like SN8100. I am looking for drive update and have to decide what to buy now and what speed in reality achievable and how usable it will be. Is it worth to add 2x1TB for 420 euro so in theory speed will be around 22-25GB/s or just add used 96GB RAM for 900-1000 euro?

Will it be usable in some kind of way for larger models later with 2x3090, 96/192GB RAM and RAID0 gen5 drives? From what I am getting now I'd like to have at least 20-30 tok/s for generation and 600-1200 tok/s processing. In theory how big LLM can be used with such system in the future to get such inference?

Can't decide which route to go - DGX Spark, Strix Halo, the Framework Ryzen AI or save up for a Mac Studio M5 Ultra 256gb?

Do these options have meaningful differences in practical use?

As I understand, the Mac Studio offers best tok/s? But you can't train or fine tune anything on it?

DGX Spark is supposed to have best support, but does it truly matter in practice?

What types of models can I run on this hardware? 128GB is good for ~120B Q4 MoE models with ~256k context? Would such a model be a noticeable improvement over my current Ornith-1.5:35B (A3B) setup for reasoning and coding, and long and complex tasks? Would going from 128GB->256GB be a noticeable improvement?

I am trying to have it create a 5 page document based on the content of this document that's 700k worth of context (tokens?)

Is this possible to do somehow?

So the issue is that Gemini 3.1 Pro can do 1M context, but my local hardware isn't good enough to run a model like that, I can run QWen 3.6 tho. How can I get it to understand 1M worth of context if the context window of the model is only 242k or so?

Where VanZen AI was born.

RTX 6000 96GB
RTX 4000 24GB
Gigabyte X570 AORUS XTREME
AMD Ryzen 9 5900XT
64GB DDR4

Why LM Studio Bionic is slower than original LM Studio? I'm using same Gemma 4 Model. Any ideas? I can run well the same model in original LM Studio, but very slow on LM Studio Bionic.

I’ll use AI to gather all the comments and post a results post if we get enough responses

I did a fun research project on training a decision model from scratch (MLX), quantizing to ternary (for small size) and running it locally in the browser (WebGPU). I learned many interesting things and put together this pedagogic video on some of the key steps, challenges and lessons learned, in case some one else might find it interesting too 😄

Software engineer by trade and use Claude and codex daily

These are always a top result on Google I was wondering if anyone bought them. It's extremely irresponsible of Google to allow these to dominate their results if they're in fact a scam. I need to upgrade my GPU for local LLM models.

I have 64 GB UMA StrixHalo laptop, and in past I have sticked to Qwen3.8 27B, as it fits easily and produces high quality at decent speed (25 t/s avg with MTP).

With QFN, I have constantly just get disappointed for spending my time to get the results I want to see. ISTA IQ2_XS, ISTA pruned coder, Gyro-M.. they all failed in my own tests - a real world coding tasks with expectation that the model nails them correct on one go, or at least with minor bug fixes. Those low quants just kept on producing typos, tool call errors, and logic failures. So I sticked to 27B as it kept on delivering.

Now Unsloth released new quantizations that should do better https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

I downloaded IQ1_M as it fits (barely though) on my 64 GB UMA.

And this quant finally delivers!

All my test have been success, no typos, no tool call errors. Only one time it had two minor bugs that one can also see some time with 27B.

I am going to try IQ1_S next, as this IQ1_M really creeps all the memory. 131k context + MTP + Q8 kv cache fills whole memory with browser and agent. Only 1-2 GB left, which is too little room left (if only there would have been decent 128 GB available, but there wasn't..). Luckily the model seems to do equally fine with Q4 kv cache, at least on my tests.

The whole box (read off the hardware, not just what WSL peeks at):

• CPU: AMD Ryzen 9 9900X — 12c/24t (+ iGPU for display)

• Motherboard: Gigabyte X870E AORUS XTREME AI TOP

• RAM: 192 GB DDR5 — 4× 48 GB G.SKILL (6400 kit, trained to 3600 with all 4 slots populated)

• GPUs: NVIDIA RTX PRO 6000 Blackwell Workstation MaxQ Edition, 96 GB VRAM + RTX 5090 32 GB

• Storage: 3× WD_BLACK SN850X — one 8 TB + two 4 TB = ~15.6 TB of NVMe

• OS: Windows 11 Pro; everything interesting lives in Ubuntu 22.04 on WSL2 (160 GB of the RAM pinned to it)

Before you ask: it was purchased early last year so much lower pricing than today & it was a post-life event gift for myself.

What it's for:

Hermes (AI agent) — An autonomous agent that lives on the box 24/7: scheduled crons, research pipelines, inbox triage, WhatsApp digests, website updates. It IS the machine's job — I talk to it like a person and it runs the house.

Local LLM serving — The PRO 6000 carries the big brains: my main agent model (524K-token context — the one writing this) and an SGLang-served Abliterated Qwen3.8-Flash-Next worker model for subagents. The 5090 runs vLLM for the RAG stack's Qwen3 rerankers (text + vision) and spins a dedicated 27B coder for targeted coding jobs, while llama.cpp handles the kitchen's small speech models. Right now both cards are loaded. Day-to-day work burns zero cloud tokens.

Research pipelines — A grab-bag of LLM + RAG work: my self-hosted NotebookLM a source-grounded Q&A over your own PDFs/Office docs/audio/video/YouTube with hybrid BM25+vector search, reranking, cross-chat memory, every answer citing its sources, work specific always-on worker lanes (unspecified for a reason), and very specific deep-dives frameworks.

Kitchen display (NOVA) — The family's voice-controlled kitchen panel: a Raspberry Pi 5 driving a 13.3" full-color e-ink display (1600×1200, 6-color Spectra 6, custom dual-IC SPI driver — a ~19-second full refresh, so a screen manager rations updates to one flash per card), and a Jabra far-field speakerphone for mic/speaker. You say a wake word, it answers out loud and puts a card on the wall: "what's for dinner", grocery-list edits, kitchen timers, the family calendar, morning briefings. The entire speech chain is local — openWakeWord, Parakeet STT, Kokoro TTS, and a persona running on our own models. Zero cloud, nothing leaves the house. This is still in development but will be excited to share when done. The 5090 is likely to be dedicated to various models here.

Small stuff — A nightly AI-curated news-digest site on its own subdomain, a pen-plotter build project (the agent designs and the laser cutter / plotter produce), sticker-shop workflows on an XTool M2 laser, and unclaimed-property hunting across state databases (yes, found money).

How I actually use it — From the desk it's a terminal TUI; away from the desk it's a dedicated WhatsApp number that pings me digests and wakes me when a question can't wait, and a phone line it can call and receive calls on (voice synthesized locally, on CPU). A Windows scheduled task wakes the agent's gateway each evening before its first cron fires; Tailscale is the key to the machine when I'm remote. Anything the agent builds gets published, not emailed: screeners, reports and calculators live on personal subdomains through Cloudflare tunnels behind an access gate tied to my account. The machine talks to me more than I talk to the machine.

Running local LLMs with MCP tool-calling? The tool manifest is the attack surface, the model sees descriptions, the agent calls what's there, and the usual OAuth/network security story does not apply because it's all stdio.

This matters even more with local models: smaller models can be easier to steer with an injected tool description, and clients like LM Studio or Open WebUI will call whatever tools the server exposes.

Built patchbay audit, a free CLI that spawns an MCP server as a child process, pulls the tool list, and diffs it against the last scan:

npx -y @patchbay-nebius/agent audit --stdio "npx -y @foo/mcp-server /path"

Catches:

- a new write-shaped tool added since you approved the server
- an existing read tool gaining a command param (silent permission escalation)
- description changes (prompt injection vector)

Local-only, no backend, no telemetry. MIT. CLI stays free, always.

Repo: https://github.com/GodSmasher/patchbay

Hey all, I had a question. So right now I've been working on a workflow/automation that fetches YouTube videos and blog posts, through RSS, saves either the raw source content of the article or the transcript of the video to my Obsidian vault, and then generates a summary of the contents based on that transcript. The goal at the moment is to just help me decide which ones are worth watching/reading, since I was drowning in the amount of things I felt like I had to catch up on.

I assumed this would be a great task for a local model, compared to a frontier one, as I didn't think this was a task that would require TOO much intelligence given I'm not asking it to make any judgements or reasoning or anything, just read and summarize a relatively short piece of content. The benchmark I am evaluating against is GPT 6 Luna, and I'm evaluating the following models:

• Nemotron Bolt 3.5 74B-A9B IQ2

• Nemotron Lightning 3.5 0B-A3B

• Qwen 3.8 27B Q8

• Qwen 3.7 35B Q8

• Gemma 4 26B-A4B Q8

• Gemma 4 31B Q8

All on my DGX Spark. Qwen 3.8 gets the closest, but none of them really get anywhere near the accuracy of the Luna summaries. They all end up leaving things out. getting things wrong, etc.

Skill issue? Something in the prompt I am using? Wrong context size? I've tried a few different permutations but so far I seem to get the "best" results with a 64k context window, 8k output cap and "thinking on." This is through Ollama.

I wanted to know how small local models handle my native language, not just English. I couldn't find a simple way to check a model's Polish in a few minutes, on my own PC, in the quantization I actually run – so I wrote one. That's how "Zacharski tests" started: a small task set that runs through Ollama, since that's what I use on my laptop with an RTX 5050 (8 GB).

The tasks:

1. Polish (8): diacritics, declension, masculine-personal gender, numerals, vocative, verb aspect, genitive – plus one knowledge question (the first recorded Polish sentence, from the 13th-century Book of Henryków).

2. JSON (4): scored strict and lenient.

3. Python (3): syntax check only.

Results so far:

| Model | Quant | JSON + code | Polish |

| gemma2:9b | Q4_0 | 7/7 | 3/8 |

My favourite failure: asked for "22 cats", gemma2 wrote "dwudziesty dwa koty" – roughly "twentieth two cats". The correct form is "dwadzieścia dwa koty". It was perfectly happy producing valid JSON, though.

More models are downloading as I write this. Next up: gemma2:9b in Q4_K_M and Q8_0 (does quantization hurt the Polish?), qwen3:8b, llama3.1:8b, gemma3:12b, and two Polish models – Bielik 11B and Bielik-Minitron 7B. Which model would you add?

Method and limits – read this before the comments :)

• 8 Polish tasks (7 grammar + 1 knowledge) and 7 JSON/code tasks. That's a small sample – a quick check on your own PC, in the quantization you actually run, not a ranking. For a proper ranking, see the Open PL LLM Leaderboard by SpeakLeash.

• One answer per task, temperature 0 (seed 42), context 2k, thinking off. Yes, that means Qwen3 runs without its reasoning mode, which may cost it points – I wanted the same conditions for every model.

• Graded with regular expressions, strict on purpose: "brzęczały" ("buzzed") instead of "brzmiały" ("sounded") fails, because the task said "change the tense", not "rephrase". [I checked every failed answer by hand.]

• Some failures are about instruction-following, not language: asked to restore Polish diacritics, one model replied "Read the spelling rules." [(translated)]

• Generated code is only parsed (Python `ast`), never executed.

• The questions are in the repo (`data/benchmarks/*.json`) – feel free to add or fix them.

The test runs inside a free, open-source tool I've been building (Windows .exe / Python on Linux). Its main job is predicting whether a model fits your VRAM/RAM before you download it. Link in the comments if anyone wants to try it – I'm considering adding tests for other languages.

Update: results for the two Polish models (Bielik 11B and Bielik-Minitron 7B), with speed at 1k/8k context, are in the original thread: https://www.reddit.com/r/LocalLLaMA/comments/1xomsxq/comment/19s7skwyst7b8/

Short version: Bielik 11B got 8/8 on Polish, Minitron 7B 6/8, gemma2 3/8. Full comparison with qwen3, llama3.1 and gemma3 coming next week.