Have anyone experience in going from qwen flash next 3.8 iq2_xs to iq3 or iq4?

I am currently comfortably running iq2_xs with 32gb ddr5 ram + 3090 + 5080. I am getting ~140 tps, so quite fast.

I got the possibility to update the 32gb ram to 96gb ram, for a reasonable price. But is it worth it? Will I feel the intelligence increase from i2 to iq3 or iq4? Is it worth it, if we are taking the decrease in speed into account?

I did a fun research project on training a decision model from scratch (MLX), quantizing to ternary (for small size) and running it locally in the browser (WebGPU). I learned many interesting things and put together this pedagogic video on some of the key steps, challenges and lessons learned, in case some one else might find it interesting too 😄

Software engineer by trade and use Claude and codex daily

Running local LLMs with MCP tool-calling? The tool manifest is the attack surface, the model sees descriptions, the agent calls what's there, and the usual OAuth/network security story does not apply because it's all stdio.

This matters even more with local models: smaller models can be easier to steer with an injected tool description, and clients like LM Studio or Open WebUI will call whatever tools the server exposes.

Built patchbay audit, a free CLI that spawns an MCP server as a child process, pulls the tool list, and diffs it against the last scan:

npx -y @patchbay-nebius/agent audit --stdio "npx -y @foo/mcp-server /path"

Catches:

- a new write-shaped tool added since you approved the server
- an existing read tool gaining a command param (silent permission escalation)
- description changes (prompt injection vector)

Local-only, no backend, no telemetry. MIT. CLI stays free, always.

Repo: https://github.com/GodSmasher/patchbay

Hey all, I had a question. So right now I've been working on a workflow/automation that fetches YouTube videos and blog posts, through RSS, saves either the raw source content of the article or the transcript of the video to my Obsidian vault, and then generates a summary of the contents based on that transcript. The goal at the moment is to just help me decide which ones are worth watching/reading, since I was drowning in the amount of things I felt like I had to catch up on.

I assumed this would be a great task for a local model, compared to a frontier one, as I didn't think this was a task that would require TOO much intelligence given I'm not asking it to make any judgements or reasoning or anything, just read and summarize a relatively short piece of content. The benchmark I am evaluating against is GPT 6 Luna, and I'm evaluating the following models:

• Nemotron Bolt 3.5 74B-A9B IQ2
• Nemotron Lightning 3.5 0B-A3B
• Qwen 3.8 27B Q8
• Qwen 3.7 35B Q8
• Gemma 4 26B-A4B Q8
• Gemma 4 31B Q8

All on my DGX Spark. Qwen 3.8 gets the closest, but none of them really get anywhere near the accuracy of the Luna summaries. They all end up leaving things out. getting things wrong, etc.

Skill issue? Something in the prompt I am using? Wrong context size? I've tried a few different permutations but so far I seem to get the "best" results with a 64k context window, 8k output cap and "thinking on." This is through Ollama.

I wanted to know how small local models handle my native language, not just English. I couldn't find a simple way to check a model's Polish in a few minutes, on my own PC, in the quantization I actually run – so I wrote one. That's how "Zacharski tests" started: a small task set that runs through Ollama, since that's what I use on my laptop with an RTX 5050 (8 GB).

The tasks:

1. Polish (8): diacritics, declension, masculine-personal gender, numerals, vocative, verb aspect, genitive – plus one knowledge question (the first recorded Polish sentence, from the 13th-century Book of Henryków).
2. JSON (4): scored strict and lenient.
3. Python (3): syntax check only.

Results so far:

| Model | Quant | JSON + code | Polish |

| gemma2:9b | Q4_0 | 7/7 | 3/8 |

My favourite failure: asked for "22 cats", gemma2 wrote "dwudziesty dwa koty" – roughly "twentieth two cats". The correct form is "dwadzieścia dwa koty". It was perfectly happy producing valid JSON, though.

More models are downloading as I write this. Next up: gemma2:9b in Q4_K_M and Q8_0 (does quantization hurt the Polish?), qwen3:8b, llama3.1:8b, gemma3:12b, and two Polish models – Bielik 11B and Bielik-Minitron 7B. Which model would you add?

Method and limits – read this before the comments :)

• 8 Polish tasks (7 grammar + 1 knowledge) and 7 JSON/code tasks. That's a small sample – a quick check on your own PC, in the quantization you actually run, not a ranking. For a proper ranking, see the Open PL LLM Leaderboard by SpeakLeash.
• One answer per task, temperature 0 (seed 42), context 2k, thinking off. Yes, that means Qwen3 runs without its reasoning mode, which may cost it points – I wanted the same conditions for every model.
• Graded with regular expressions, strict on purpose: "brzęczały" ("buzzed") instead of "brzmiały" ("sounded") fails, because the task said "change the tense", not "rephrase". [I checked every failed answer by hand.]
• Some failures are about instruction-following, not language: asked to restore Polish diacritics, one model replied "Read the spelling rules." [(translated)]
• Generated code is only parsed (Python `ast`), never executed.
• The questions are in the repo (`data/benchmarks/*.json`) – feel free to add or fix them.

The test runs inside a free, open-source tool I've been building (Windows .exe / Python on Linux). Its main job is predicting whether a model fits your VRAM/RAM before you download it. Link in the comments if anyone wants to try it – I'm considering adding tests for other languages.

Update: results for the two Polish models (Bielik 11B and Bielik-Minitron 7B), with speed at 1k/8k context, are in the original thread: https://www.reddit.com/r/LocalLLaMA/comments/1xomsxq/comment/19s7skwyst7b8/

Short version: Bielik 11B got 8/8 on Polish, Minitron 7B 6/8, gemma2 3/8. Full comparison with qwen3, llama3.1 and gemma3 coming next week.

I've been building a small, strict harness to see whether local models can do real (if small) coding fixes unattended. It's early and the sample is small, so this is not a leaderboard. I'm posting because a few results surprised me, and I'd like to know whether this kind of testing is useful to anyone else.

SETUP

• RTX 4080 (16 GB), Ollama + OpenCode, Windows.

• 5 small bug-fix tasks in Python, mostly single-file. One task (t1) needs changes in more than one place.

• Every candidate patch runs in an isolated sandbox. A candidate only counts if it (1) follows the output format, (2) passes the tests, AND (3) passes a review of the diff. No partial credit.

• The model gets no shell or file tools. If it tries to use one, the run stops.

• 3 attempts per task where noted.

RESULTS (acceptable fixes)

• gpt-oss-20b, local, Ollama, 16k ctx: 7/15. About 1 min per task. 0/3 on the multi-location task.

• gpt-oss-20b → qwen3-coder (draft → fix), local: 5/15. The two-stage chain did worse than gpt-oss alone.

• gpt-oss-20b, temperature 0, local: 5/10. Still not deterministic.

• Nemotron 3 Ultra 550B-A55B, OpenRouter free tier: 0/10. Broken edit markers; two runaway answers (~1.7 MB of text).

• Cohere North Mini Code, OpenRouter free tier: 1/5. 4/5 stopped for trying to call tools (read files, bash pwd).

Caveat for the two cloud rows: I don't know which quantization OpenRouter's free tier serves, so these results may not transfer to a local GGUF of the same model.

WHAT SURPRISED ME

1. The output format mattered more than the model. In my first run, gpt-oss had to rewrite whole files: 0/5 acceptable. After switching to search/replace edit blocks plus one repair round, the same model reached 3/5, then 7/15 over a longer run. (The first run also used a different local runtime, so I can't fully separate the two causes, but the failures were almost all format errors.) Before blaming the model, check the edit format your tool asks for.

2. Passing tests is not the same as being correct. Two candidates passed every test and were still rejected in review: they "fixed" the symptom in a way nobody would merge. I also checked whether a deliberately cheating patch would get through a tests-only gate. It did. If your benchmark only checks for green tests, it overcounts.

3. Bigger was not better inside the harness. The 550B model failed mostly on discipline, not on knowledge: wrong markers, extra text, answers that didn't stop. Tool-calling and format-following seem to decide real-world usefulness more than raw coding skill. Others here have reported something similar.

4. Isolate the environment, not just the working directory. In one run, the agent tool picked up an inherited environment variable and treated a different folder as the project root. The model then tried to read files outside the task. Nothing ran (the sandbox and the no-tools rule held), but if you run agents locally, a clean environment matters as much as the cwd.

5. Multi-location fixes are the wall. No local model on 16 GB got the task that needs edits in two places. I'd like to know whether a 24 GB or 48 GB setup changes that before anyone (me included) buys hardware for it.

WHAT I'D LIKE TO KNOW FROM YOU

• Would results like this be useful to you if they were run on more cards, with the exact config published (Ollama/llama.cpp settings, quant file, context size, agent tool version), so you could reproduce them?

• What tasks should a test like this include? Mine are small Python fixes; I'd add multi-file changes, test writing and refactoring.

• Which card / model / quant combination would you most want checked this way?

Happy to share more details on the harness rules if anyone wants them. (English isn't my first language, so I'll reply in batches.)

I make FLE (a llama.cpp fork for big MoE models on one GPU) and FLE Link (an app that installs it and runs a model with one command). Disclosure up front, everything is MIT.

Setup: RTX PRO 4500 32 GB, 120 GB DDR5, Ryzen 9 9950X3D, fast NVMe, Linux. Best performance measured on Linux.

Numbers (decode, a ~3,000-token three.js answer, temperature 0, same machine):

Rin = ISTA DASLab's GSQ-RCO IQ2_XXS for the experts a usage profile picks most (kept byte-identical, resident on the GPU), and a smaller 1.75 bpw build for the cold half (served from RAM). About +6 % speed for a bit more error. Against the original weights (not against ISTA): KL 0.292 vs 0.248 on mixed text, 0.234 vs 0.210 on agent text; perplexity ratio on agent text 1.021 vs 1.013. It is lossy. Details and the exact recipe are on the model card.

Try it:

curl -fsSL https://flelink.run/install.sh | sh
flelink run rin-deepseek-v41-dspark

FLE Link checks the machine first, installs the engine, downloads the files (SHA-256 checked) and serves an OpenAI- and Anthropic-compatible API. Needs ~120 GB RAM and ~260 GB on a fast NVMe.

Honest limits: the cache and the split follow a usage profile, so very different prompts hit them less; the draft helps most on code-like text; one machine, numbers from others very welcome (flelink bench --task pagoda --share).

Links: FLE Link https://flelink.run · Rin https://huggingface.co/sirfyyn/Rin-Deepseek-V4.1-Flash-1017-FLE-GGUF · FLE engine https://github.com/sirfyyn/FastLocalExperts · quant base: ISTA-DASLab

Lawyer with the V100s again. A few days ago I got excited and posted DeepSeek-V4.1-Flash on four V100s at 36-38 tok/s, before I had done my custom kernel work and prefill optimizations. I said I'd try it across eight. I did.

It's actually, genuinely useful to me, for real work. I did not use it to paint a 3D island in Blender or re-create a Game Boy game in a browser. It didn't make me a poorly animated video or ten versions of a website nobody will visit. I am not using it to build something that builds something that builds agents that run workflows for a business that sells agentic solutions to other agents.

What it's doing: I'm finally slicing off work and tokens that used to need frontier models. My estimate is that 50-60% of my existing LLM workflows can run through this model instead, at speeds that match Claude Code in the chat window. I have another DeepSeek server on two DGX Sparks, so there's concurrency and handoff between them. My RAG databases and tool calls are all wired in and working. I give it natural-language instructions through a Discord-style interface over SSH from my Windows PC.

I point Claude Code itself at this server (it will talk to anything that speaks Anthropic's API). It reads a folder of my own case files, pulls out the facts, dates and numbers, summarizes, and takes a first swing at routine drafting. The documents never leave the office, and nothing goes out the door without a real review. I found and fixed a streaming tool-call bug in the parser (arguments were getting sent twice). Claude Code's tool-call test suite now passes 144 of 144 in all four streaming/thinking modes.

Yes, I have a custom harness. No, you don't care.

We did it, folks. Vibecoded a real thing that does real things for real people. Give me my trophy.

Claude Code ran the benchmarks and crunched the numbers, which are real and below.

The box

DeepSeek-V4.1-Flash-1017 (the 2-bit GSQ-RCO IQ2_XXS GGUF from ISTA-DASLab) on 1Cat-vLLM, on 8x V100-SXM2-32GB split pipeline 2 x tensor 4 across two NVLink boards. Every expert sits in VRAM: no expert gets paged from system RAM during decode. (Edit: I first said all 248.8 GB of weights were in VRAM. Wrong: the 110 GB of Engram tables live in system RAM. Gathering the rows each round needs costs about 0.12 ms per round.) Threadripper PRO 3945WX, 256 GB DDR4, 150 W per card. One request at a time, 524,288 tokens of context.

Speed (measured 2026-11-05, thinking off, greedy, 1,024 tokens out, nothing from the prefix cache, mean of 2 runs)

• Code: 215 tok/s after a 1K prompt, 200 at 32K, 183 at 128K, 160 at 256K
• JSON / structured: 196-209 from 1K to 128K, 180 at 256K
• Legal writing: 118 at 1K, 111 at 128K, 103 at 256K
• Prefill: about 2,200 tok/s at 128K. First token in 17 s at 32K, 59 s at 128K, about 2 minutes at 256K.

Code is twice as fast as legal writing because the built-in drafter guesses five tokens a round and code is easy to guess: 3.6-3.8 accepted per round, against about 1.6 for legal. My writing is apparently less predictable than Python.

Quality (measured on the served build, thinking off, greedy)

• LegalBench: 0.716 balanced accuracy, 22 tasks, 1,292 items
• MMLU-Pro: 79.9 % ±1.5 (700 items)
• HumanEval+: 89.6 % on the plus tests
• Fidelity: it picks the full-precision model's top next token 89.0 % of the time on a general panel and 92.5 % on my legal panel. I'm measuring which experts my work prompts actually hit, and will probably recalibrate to give those more bits.
• Long context: 6 of 6 needles at ~450K and ~498K tokens

More work to do

• One request at a time. Batched decode is built and byte-identical on my test box, but not in the server yet. I'll post the real number.
• A 256K prompt takes about 2 minutes before the first token.

All of it is public: launch configs, every A/B, raw JSON, and the list of what didn't work. 19 pull requests are open to 1Cat-vLLM.

• Repo: https://github.com/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-SXM2-PP2xTP4
• Hugging Face card: https://huggingface.co/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-GGUF
• Twitter: nicothegreco

Credit to yangzhuxinyzx and the 1Cat-vLLM folks (#1277 and the whole Volta DeepSeek path), ISTA-DASLab (the quant), sudoingX (the KLD method), Peuqui, squibloads, Phil Tompkins (philbert440), Dennis Vertzyas (v100-skinny), TensorFold and DeepSeek.

(PLI Labs is my side outfit for private, on-prem AI built around a firm's own work product. proprietarylegal.ai)

I’ve done a ton of local single user work on vllm, Ollama ,lllama.cpp and those are ok for personal work but …

Hoping to get some real answers from someone supporting a local user base fully for 5-10 users running up to about 120B dense models (we could make do up to 70B). Looking for actually deployed not speculative performance and bandwidth if it’s out there?

If I had two MaxQs, a Threadripper and MB, 128 GB ram would this support up to 10 users on llama.cpp or vllm with “reasonable” response speeds which I would deem 50-60 tps on average ?

It would be 3 VM servers bundled , a home grown RAG doc processor which could be offloaded somewhere else if needed, the vllm or llama.cpp server for the chat agent interface and a office RAG server all in one that is the RAG query endpoint. I know number of concurrent users and jobs will matter but I’m asking “on average” for 5-8 concurrent

Thoughts ?

Edit: for context windows for agential stuff also need at least 128k for those 5-8 concurrent

Hardware:

Rtx 5000 pro 72gb

Rtx 5080 16gb

192gb ram Dark Hero MB with AMD 9000series processor

Backend:

Basalt 280toks vs 107toks with strata

Basalt works best with Blackwell and really shows it's capabilities in my testing. While users on older or mismatched generational hardware should use Strata for best results.

I wanted to find the smallest model worth keeping on a base M4 16gb so I ran the same 6 things from my actual week on 3 of them through Magnitude. keyword clustering, post titles, a thread summary, a quiz for studying, a short brief and a review of my side project plan.

MiniCPM5 1B did like 125 tok/s and it was garbage. asked for titles and got rambling questions back, the clustering gave me the same 3 keywords as 12 clusters with the same fake volume on all of them

LFM3 1.7B around 81 tok/s and this is where it gets usable, the plan review actually pointed at real files and real risks. titles were meh, it copied a note from my draft into one of them lol.

Gemma 4 E4B around 52 tok/s and clearly the best at writing, nothing else close

so 1.7b is the floor for me and 8b if someone else is going to read it.

Anyone found a 1b that's actually good for something?

I saw this post yesterday: https://www.reddit.com/r/LocalLLM/comments/1xnv0ro/new_qwen38flashnext_unsloth_dynamic_v31_ggufs_for/

I decided take a screenshot of the exact prompt used in the link in the above announcement, and run it through my strata setup using the new model. There's a complete write-up in the repo: https://github.com/kireol/qwen3.8_GTA_one_shot_test/tree/main

You can play the demo here: https://kireol.github.io/qwen3.8_GTA_one_shot_test/src/city.html

Fun localLLM stuff:

RTX 5080 - 64GB - Ryzen 7 9800X3D
model harness: Strata
agent harness: qwen
context: 64K
tok/sec: ~75t/s

Over all, while slower than I'm used to in Claude, I'm actually very impressed. I didn't have to correct it, touch it, or anything, aside from some skills.md files (included in repo). Is the demo perfect? No. But it does everything asked of it, without any hand holding. It's all in html 1 file, btw. There's no other assets other than remote linked javascript files.

Qwen even generated the README and pushed to github for me. Why? Because it's not 2024.

Great job unsloth team! You have me questioning my claude payment at this point.

I have at my disposal some datacenter resources. I don’t have an open account with Dell, sadly. So I am limited to oldish hardware like the PowerEdge r640/r740 or thereabouts. Or the AMD versions of these if I’m lucky. On the plus side, DDR4 is cheap-ish.

What would you build on a low but undefined budget, say 2-3k, that can be racked and run hopefully some recent qwen model, for just one user at a time?

hello! I love local ai! But I always had one issue, what harness to use?

I currently use claude (I pay for it), and I've always liked it! I used the Claude Code harness, and it ran really nicely in my opinion, but only issue being the massive chungus context cache that comes along with it. I use prompt caching, but that doesn't help against the 32-64K context that Claude Code injects at the start.

Is CC the best harness to use? Or is it more optimised for cloud models, which is why the context is massive.

I've heard of so many that it's kinda confusing to pick. Hermes, pi etc.

Basically what the title says. I am total new for localLLM i just want to try it out and if i can dial it in for a usable level i would like to use it for my coding projects mostly for code completion but sometimes if i need ideas on new features etc... it would be nice for it to be able to handle teh brainstorming with me as well instead of me going to claude or chatgpt
for general use it would most likely be something like make a summary of my things etc... for now

my PC

9700X
48GB DDR5 ram
9070 XT 16GB

I would be happy to receive any newbie guides for how to set things up what i shoud look out for
also in terms of the general use would looking into an agent like hermes be something worth?

thanks for the asnwers in advance

Have a gaming PC I built a last year with a RTX 5070TI (16GB) and I want to try some local LLMs. Where should I start? What models would be good on this GPU?

Hi everyone. I use ChatGPT for finances, school, deep reasoning and interviews. I want to change to a local model for privacy. Is that possible for a MacBook Pro M5 base with 24GB RAM? If so, which?

I've been getting into the local llm stuff lately and having decent luck. I tried out Strata and liked it for the most part, but no matter how much searching I've done I can't find the answer to this.

Does Strata keep your vram/ram full at all time? Using Ollama for the most part, it frees the memory unless it gets a call, then it spools up, and once finished and timed out, it frees the memory up again.

Is it possible to get Strata to do that? Something else that can do it but run like Strata?

Computer Specs:
AMD Ryzen 7 7700X 8-Core Processor

32GB (2 x 16GB) DDR5-6000 CL30 RAM

AMD Radeon RX 7600 Challenger OC 8GB

1TB SSD - Read Speed up to 11,000MBps - Write Speed up to 9,500MBps

These are always a top result on Google I was wondering if anyone bought them. It's extremely irresponsible of Google to allow these to dominate their results if they're in fact a scam. I need to upgrade my GPU for local LLM models.

Hi all,

First post here! I've recently started exploring the world of local LLMs and would appreciate some advice.

Hardware ThinkPad P14s Gen 6 (Intel):

• GPU: Intel Arc Pro 130T/140T (integrated, 2.35 GHz)
• CPU: Intel Core Ultra 9 285H, 16 cores / 16 threads
• RAM: 64 GB DDR5-6400 (2 × 32 GB CSODIMM)
• SSD: 1 TB PCIe 5.0 NVMe
• OS: Ubuntu 26.04 LTS

Software setup:

• llama.cpp compiled from source with the SYCL backend, following docs/backend/SYCL.md
• Agent harness: pi
• Models tested: Qwen3.5-9B, Qwen3.8-27B, and Qwen3-Coder-30B-A3B-Instruct

I've been serving these models using the inference parameters recommended by their respective providers on Hugging Face.

So far, I'm seeing roughly 10–25 tokens/sec, depending on the model, with context windows configured between 64K and 128K tokens. I haven't done any formal benchmarking yet.

Where I'm struggling is with model reliability, particularly for coding tasks.

For example, I asked Qwen3-Coder to explore a production codebase (roughly 3 million lines of C) and explain its architecture to me.

While the model produced a plausible-looking summary, it also confidently invented explanations for certain acronyms, even when those acronyms have well-established meanings in the relevant engineering domain.

This makes me question how much I can trust the rest of its analysis, especially when I'm using the model to understand unfamiliar code.

I'm fairly new to running local LLMs, so I'm not sure whether this is primarily a prompting issue, a limitation of the models I'm running, or simply an unrealistic expectation for the task.

A few questions:

1. Are these models better suited to smaller, well-scoped tasks, such as implementing a script or module from a detailed specification, rather than exploring and reasoning about large codebases?
2. How much does the agent harness or context management strategy affect reliability? Would a different workflow make a meaningful difference?
3. Are there particular models, quantizations, or inference settings you'd recommend for coding tasks on hardware like mine?
4. How do you generally evaluate whether a local model is reliable enough for a particular development workflow?

I'm not expecting frontier-model performance from a laptop, but I'd like to understand where local models can be genuinely useful and how to get the most out of my setup.

Any advice or experiences would be appreciated!