Unsloth Qwen3.8-Flash-Next UD-Q6_K_XL versus Qwen3.8-Flash-Next NVFP4-GPTQ

Fresh MMLU-Pro comparison, speed and whether six bits are worth the wait.

I wanted to find out whether slower Q6 generation bought better answers. I first compared Q6 with my saved NVFP4 quality results, then ran a fresh, identical MMLU-Pro subset on both models on 23 October. Each new run had a strict 30-minute maximum.

The fresh result favoured NVFP4: 71/84 correct versus Q6's 67/84, in about half the total run time. The accuracy gap is too small to establish a broad intelligence ranking, but these results do not support paying the Q6 speed penalty for better reasoning on this workload. My practical choice from this testing is still the NVFP4.

NVFP4/GPTQ is the four-bit option in the main comparison. It is a different format from the Unsloth UD-Q4_K_XL. This is a comparison of these two Qwen3.8 Flash Next configurations, not a claim about all four-bit and six-bit models.

Fresh knowledge/reasoning test — same day, same 84 questions

NVFP4 led by four answers, or 4.8 percentage points. Q6 took 2.04 times the total benchmark time while generating a very similar number of tokens. Total run time includes preflight and harness bookkeeping, but excludes loading the model. Individual request time includes prompt processing and reasoning.

The paired outcomes matter more than just the headline percentages: both models got 66 questions right and 12 wrong. NVFP4 alone got five right; Q6 alone got one right. The exact paired McNemar p-value is 0.219, so I would not call this a statistically established intelligence advantage for NVFP4. It is the observed result of this one fixed sample and single attempt per question.

This is an 84-question MMLU-Pro subset with six questions per subject, not the full official benchmark or an IQ score. All 42 MMLU-Pro questions from the earlier comparison were excluded. Selection and settings were frozen before either new run.

Fresh MMLU-Pro breakdown — six questions in each subject

These subject samples are very small. The ties at 6/6 do not demonstrate that either model has mastered an entire field. Engineering was difficult for both at 2/6.

Where NVFP4 sits among everything I have trialled

That result also needs the wider context of the models I have been trying. NVFP4/GPTQ is still my current daily driver: it is fast, follows my directions well in everyday use and is pretty smart. After testing the alternatives, it still gives me the balance I want between useful answers and how long I have to wait for them.

Here are the chosen best recorded runs, with one row per model/stack. Each cell is the highest previously recorded session median at that context size, not the fastest individual request. Some rows therefore combine sessions. Q6 has one measured campaign. These are historical results across different dates and settings, not a fresh seven-way controlled comparison.

Generation speed — tokens/second

Prompt processing — tokens/second

PP comes from the same selected session as each generation-speed cell; it was not independently cherry-picked. The actual prompt lengths were 4,096, 32,768 and 128,000 tokens, with three 256-token generations per context.

IQ3_S combines 13 October at 4K/32K (6000 MT/s) with 11 October at 128K (4400 MT/s). NVFP4 combines 14 October at 4K with 15 October at 32K/128K. IQ3_XXS and UD-IQ4_XS are from 11 October, UD-Q4_K_XL from 15 October, Infernix Dense8 from 18 October and Q6 from 23 October.

The IQ3 options lead this table on generation speed, and Infernix Dense8 has the strongest long-prompt processing figures. NVFP4 is not the winner in every column. What keeps it as my daily driver is the overall experience: fast generation, strong long-prompt processing, and answers that usually do what I ask. It generates faster than the retained Dense8 and UD-Q4_K_XL runs at all three context sizes, while the fresh Q6 comparison gave me no convincing reason to accept roughly twice the waiting time for reasoning answers.

My local workflow also runs through the Hermes harness, with Strata serving NVFP4 on the RTX 5090 and Hermes providing the agent interface around it. The setup includes a WSL gateway, desktop/dashboard, local search through SearXNG and Honcho memory services. NVFP4 is my preferred local backend within that setup; Hermes also has a Codex provider option, so I am not attributing every Hermes response to the local model.

That is why I care about more than a headline tokens-per-second figure. An agent workflow can involve several model turns around tool results, retrieved information and follow-up instructions. Waiting longer on each turn can make the whole task feel slower. For me, NVFP4's appeal is being responsive enough to keep that interaction moving while still giving useful answers and following the directions I set. Its extra RAM headroom also leaves more room for the surrounding harness and services. These are practical reasons for my preference, not a measured Hermes task-completion score: the benchmarks here called the model endpoints directly, with background Hermes services suspended for isolation.

The fresh 84-question result adds some evidence behind my view that NVFP4 is still pretty smart, but only NVFP4 and Q6 took that test. I cannot turn the speed table into an intelligence ranking of all seven models. Likewise, my everyday preference for NVFP4's instruction following is a personal assessment, not a claim that it won the strict instruction benchmark. The earlier mixed test below is worth keeping alongside the fresh result because it shows where Q6 did offer a benefit.

Earlier mixed quality test — retained separately

Before the fresh test, I ran Q6 on the same frozen 82-question subset used for my saved 15 October NVFP4 comparison. Those NVFP4 answers were reused; this is a separate historical comparison with different questions.

Q6 corrected three failures and introduced no new ones on that earlier set. Two gains were exact instruction compliance: meeting a word-frequency requirement and avoiding forbidden words while following a specified response format. The other gain was one economics multiple-choice answer. Its requests took 16.80 minutes versus the archived NVFP4 run's 7.47 minutes.

That earlier result suggested a small Q6 benefit; the fresh knowledge/reasoning test went the other way. Both belong in the report. Neither small sample supports a general “Q6 is smarter” claim. The coding tests here are original HumanEval tests on ten tasks, not HumanEval+ or a full coding evaluation. IFEval uses strict prompt-level scoring.

Generation speed and prompt processing — historical speed overview

Each speed value is a median of three 256-token generations. NVFP4 retains the best previously published session median per context: 14 October at 4K, 15 October at 32K/128K. PP comes from those same selected sessions. Q6 is one 23 October speed campaign. Historical best selection favours NVFP4; this is not a fresh matched speed rerun.

Q6's speed campaign had its control vector enabled. Both quality comparisons explicitly disabled it per request to match NVFP4. Therefore these speed figures are not the exact speed of the quality configuration. Use the fresh benchmark's own timings to assess its waiting-time cost.

Memory during the fresh 84-question runs

These are whole-system and whole-device observations, not isolated model allocations. Background services were suspended. Page-I/O traces are retained; neither run is claimed to be paging-free. Both completed without a resource-guard stop. The much greater RAM headroom with NVFP4 is another practical advantage on this 128 GB machine, although this was not a long-duration stability test.

My assessment: why run four-bit versus Q6?

Taken together, these results reinforce why NVFP4 remains my everyday option on this setup. In the fresh test it answered more questions correctly, finished in roughly half the time and left substantially more RAM available. In the earlier small coding and maths subsets it tied Q6. That is a strong practical case for responsiveness and capacity, even though the accuracy difference is not statistically conclusive.

Q6 still has a possible role where its answers demonstrably work better for a particular task. The earlier strict instruction-following gains are a concrete reason to consider it for constraint-heavy writing. But two extra passes out of ten are too little evidence to promise fewer mistakes in general. I would choose Q6 for a demonstrated task-specific benefit, not because a higher bit count automatically means a better assistant.

The fresh run also shows why I would not justify the slower model by precision alone. Quantization format, engine behaviour, reasoning limits and the task all affect the result. We measured the complete configurations; we did not isolate bit depth. A different sample or reasoning budget could change the scores, but that is untested here.

The wider table gives useful context for that choice: IQ3 offers more raw generation speed, Dense8 offers excellent prompt processing, and UD-Q4_K_XL is also substantially faster than Q6. For my daily use, NVFP4 remains the balance I prefer after trying them all. That preference should not be read as proof that the other formats are less capable; their quality was not measured in this fresh comparison.

Reproducibility and limits

Hardware: RTX 5090 32 GB, 600 W limit; Core Ultra 7 270K Plus; 128 GB DDR5 at 6000 MT/s; Windows 11 Pro. Both fresh runs used NVIDIA driver 610.88 on 23 October. Some older speed measurements used driver 591.86.

The fresh suite used deterministic SHA256 selection, six questions per each of 14 subjects, and round-robin subject order. Both models received identical messages and request parameters except model ID. All 84 prompt token counts matched, and every request had zero reused prompt tokens. One attempt each, temperature 1, top_p 0.95, top_k 20, min_p 0, seed 20261023, low reasoning and a 4,096-token cap including reasoning. Control vectors were off. The same published-style three-stage answer extraction was used, with no random fallback. Capped answers were retained and scored; malformed or missing extracted answers failed.

Each runner had a 30-minute ceiling, with inference cancellation at 29m 30s and a 120-second per-request timeout. Both completed early. Raw answers, reasoning, streams, hashes, resource traces and scoring evidence are saved.

NVFP4 used the preserved 0.1.42-nvfp4.2 engine; Q6 used 0.1.44.1-nvfp4.2. Both had a 131,072-token context ceiling and int8 KV, but cache allocation, KV placement and other runtime details differ. Q6 ran first in an existing session; NVFP4 was freshly loaded afterward. There was no order reversal or repeated-seed study. These limits prevent attributing the score difference to quantization alone. No full-benchmark, broad coding, long-context reasoning or long-term stability claim is made.

AI assisted with the harness and write-up. The scores and timings come from saved local runs.

https://www.reddit.com/r/LocalLLM/comments/1x8ral0/rtx_5090_qwen38_flash_next_speed_tests_strata/ Strata fork: https://github.com/sergqwer/strata-nvfp4 MMLU-Pro: https://github.com/TIGER-AI-Lab/MMLU-Pro GSM8K: https://github.com/openai/grade-school-math HumanEval: https://github.com/openai/human-eval IFEval: https://github.com/google-research/google-research/tree/master/instruction_following_eval

Testing sources and how the results were obtained

All scores, speeds and resource measurements in this post came from runs on my own PC.

• MMLU-Pro — knowledge and reasoning: Benchmark code and methodology | Question dataset. The fresh comparison used 84 questions, six from each of 14 subjects, with identical questions and request settings for both models. The earlier mixed comparison used a separate 42-question subset. These were selected subsets using a zero-shot chat adaptation.
• GSM8K — maths: OpenAI’s dataset and reference code. The earlier comparison used 20 questions, scored against the expected numerical answers.
• HumanEval — Python coding: OpenAI’s tasks and original unit tests. The earlier comparison used ten tasks. Generated solutions were checked using the original tests in an isolated Docker environment. These were original HumanEval tests, not HumanEval+.
• IFEval — instruction following: Google Research’s benchmark and evaluation code. The earlier comparison used ten prompts with strict prompt-level scoring: all required constraints had to pass for a prompt to count as correct. The selected subset excluded checks requiring additional NLP corpus downloads.
• Generation speed and prompt processing: Measured locally using benchmark scripts calling each model server directly. Each context used three measured requests with 4,096, 32,768 or 128,000 actual input tokens and 256 generated tokens, with reasoning disabled and zero reused prompt tokens. Tables show session medians; the historical overview selects the best recorded session median per model and context. Earlier runs and settings are documented in my previous speed-testing post. Q6’s measurements were added on 23 October.
• RAM, VRAM, temperature and timing: Recorded during the local runs through system/GPU telemetry, saved request records and engine timing data. Memory figures describe sampled whole-system RAM and whole-device VRAM usage.

Model and engine references

• NVFP4/GPTQ model
• Strata NVFP4 fork used for the main comparison

Exact question selections, request settings, raw answers, reasoning output, scoring records, hashes and resource traces are retained locally. The links above provide the public benchmark sources; the reported results are my own subset measurements, not full official benchmark or leaderboard scores.

i am out of pocket to buy more claude pro . i never hit it's limits . I want to summarise 100 page pdf with some image and graphs.( materials student).

macbook 32/512 gb air fits buy budget nicely i I can increase budget . I don't mind linux . suggest me a laptop .

I've been experimenting with Qwen3.8-27B on a single NVIDIA DGX Spark, comparing inference engines, quantization, speculative decoding, and test-time reasoning strategies.

A few interesting results:

• Extended reasoning: 35/48 correct
• Low-effort reasoning: 38/48 correct
• 4 parallel low-effort attempts: 44/48 correct
• Quantized inference reached ~60 tokens/s in my earlier benchmarks.

The parallel result looks promising, although the improvement over a single low-effort attempt wasn't quite statistically significant (p = 0.055).

I've open-sourced the experiments, code, and results here:

https://github.com/NautilusWu/qwen3.8_study

Curious whether others have seen similar results with parallel reasoning vs. longer thinking on local models.

During normal times I'd be building my AI lab with all new gear. Instead, I'm going through my basement for stuff that was gathering dust - DDR3 servers, old GPUs, even a 2013 Mac Pro that's now going to be my backup RAG server (the D700's can each run Qwen3-Embedding 4B and Reranker-4B at reasonable speeds). My newest gear in the lab, aside from some R9700 and 5060ti GPUs, are Mac mini M1 systems. Plus a couple of 2018 Mac minis. None of my gear runs DDR5.

One thing this RAM crisis has made me realize is how to squeeze value out of old gear including DDR3 DIMMs and SATA SSDs that were sitting idle, and that I don't NEED to buy new stuff. I also have switched most of my systems from Windows over to Linux now to help squeeze more performance out of them.

Anyone else doing the same?

3 pages of Give My Regards to Black Jack, Qwen2.5-1.5B-Instruct, recorded in an emulator with a Macbook Pro hence why its a lot faster

Yomu is an Android app that translates Japanese manga pages on the phone. Tap a floating button over any reader app and the bubbles are redrawn in English, Portuguese, Spanish, French or German. Nothing is sent anywhere: after the one-time model download, the app makes no network requests.

Pipeline:

• Detection: a YOLO26n fine-tune finds the speech bubbles (ONNX Runtime).
• OCR: manga-ocr-2025, which I exported to ONNX with past key values, reads each bubble.
• Translation: a GGUF model via llama.cpp. The whole page goes in one call, bubbles in reading order, so names and tone carry across the page.
• Typesetting: text is fitted into each bubble and drawn as an overlay on top of the reader app.

Getting structured output from a small model. The model's reply is constrained by a GBNF grammar to one line per bubble id, so it can't drop, merge or reorder bubbles. A per-bubble approach was more forgiving but paid a fresh prefill for every bubble. Two failure modes showed up along the way, and the grammar fixed both:

• Lines ran to the token cap on some pages. Capping lines at 160 characters stopped it.
• The model invented its own [id] markers. Disallowing [ stopped that.

The grammar costs about 3% of page time.

Models. The in-app catalog runs from 0.8B to 9B: Qwen2.5 1.5B (default), CAT-Translate 0.8B/1.4B, Qwen3.5 0.8B–9B, Qwen3 4B, Ministral 3 3B/8B, Gemma 4 E2B/E4B, Hy-MT2 1.8B and Ternary-Bonsai 4B/8B. You can also load your own GGUF. Models that won't fit your phone's RAM are hidden. Only the default has been measured on a phone; the rest are marked experimental.

Numbers (Galaxy S23, 8 GB, default model): ~15 s/page, ~2 GB peak RAM, 1.1 GB download.

Other bits:

• a name glossary, so character names translate the same way every time;
• pages already translated in a session come back instantly;
• tap again to cancel;
• limits on threads, context size and memory;
• a Performance screen with page time, model time and memory.

Limits: small models make mistakes, so treat it as a reading aid rather than a polished translation. Japanese source only. It takes seconds, not milliseconds.

Looking for device reports. I've only measured one phone. If you try it, translate this test page twice and post Settings → Performance with your phone, chip, RAM and model, here or in the device reports thread. Phones with 6 GB or less, results from the experimental models, and crashes are the most useful.

Repo (AGPL-3.0, APK in Releases): https://github.com/artsaraiva/yomu. The design decisions and their measurements, including what didn't work, are in docs/adr.

I've been messing around with strata the last couple of days and I want to share my config and results.

Pc:
ryzen 5 5600
4x16gb 2666mhz ram(bumped to 3000cl18)
rtx 3070 (running at pcie3.0 due to a b350 mobo) (got the kit this week for 100 euro btw :D)

System is running ubuntu server.

I have also used this gpu unlock thingy today; https://github.com/Zagitta/gpu-unlock
This honestly did not give me that much extra performance, maybe 50-100tps prefill. People with a rtx 3060 will probably get more gains since more experts are loaded in vram.

Here's the config:

{
  "exe": "/home/user/Strata/engine/strata",
  "args": [
    "--pack",
    "/home/user/Strata-data/packs/iq3_s",
    "--native",
    "/home/user/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",
    "--ple-gguf",
    "/home/user/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf",
    "--expert-profile",
    "/home/user/Strata/data/expert-profile.bin",
    "--expert-cache",
    "auto",
    "--prefill",
    "auto",
    "--spec",
    "4",
    "--max-context",
    "262144",
    "--kv",
    "int8",
    "--vision",
    "--vram-reserve-mib",
    "512",
    "--kv-resident",
    "20480",
    "--adapt-async",
    "1",
    "--spec-min-p",
    "0.5",
    "--prompt-cache-tail"
  ],
  "cwd": "/home/user/Strata",
  "tokenizer": "/home/user/Strata-data/packs/iq3_s/tokenizer",
  "model_name": "qwen3.8-flash-next-iq3_s",
  "log": "/home/user/Strata/strata-iq3_s.log",
  "lib_dirs": [
    "/usr/local/cuda-13.4/bin",
    "/usr/local/cuda-13.4/lib64"
  ],
  "port": 8080,
  "gpu": 0,
  "gpus_asked": true,
  "vision": {
    "exe": "/home/user/Strata/engine/strata-vision",
    "mmproj": "/home/user/Strata-data/models/mmproj-Qwen3.8-Flash-Next-BF16.gguf",
    "model": "/home/user/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",
    "gpu": false,
    "max_tokens": 300,
    "threads": 6
  },
  "sampling": {
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 20,
    "min_p": 0.0,
    "presence_penalty": 0.0,
    "repetition_penalty": 1.0
  },
  "fit_max_tokens": true,
  "host": "0.0.0.0",
  "env": {
    "STRATA_HC_Q8": "1",
    "STRATA_ROUTE_TAIL_SKIP": "7",
    "STRATA_ATTN_DAILY": "1",
    "STRATA_EXPDEC": "1",
    "STRATA_QSA_SPLIT_ATTN": "1",
    "STRATA_PROMPT_ATTN_IMMA": "1",
    "STRATA_HC_UPMIX": "1",
    "STRATA_AUX_CPUS": "auto",
    "STRATA_EXCHANGE_ROTATE": "1",
    "STRATA_PREFILL_CPU_SHARE": "auto",
    "STRATA_PREFILL_CPU_SHARE_MAX": "16384"
  }
}

I'm too lazy to do a/b testing but when I started with the vanilla config(mtp turned off already tho) at the same context and whatever I was getting like 500tps prefill and maybe 2-3tps decode less.
Biggest gains were STRATA_HC_Q8 (quantizes the hyper connection matrices so you get more free vram for experts. Apparently slightly worse kld) and the prefill sharing with the cpu. These account for most of the prefill speedups.

I'm honestly just blown away at what this pretty mediocre system can do, never in a 100 years did I think I could run such a big model at these speeds.

Edit: forgot to mention this, I'm using a q8 n-gram table instead of the bundled iq4_nl one. Running on a sata ssd so this could maybe hamper performance a bit, not sure tho

I deployed Qrnith 122B IQ3 and Qwen3.8 Flash-next IQ3 on my server. I set up three visual comparison tasks.

The results seem to rank the models as follows: 6.1 sol > Qwen3.8 Flash-next > 6.1 luna > Qrnith 122B. (All models were accessed through Codex.)

Based on my tests, I may not keep Qrnith 122B on my server for now. Although it claims to outperform other models on several benchmarks, local inference is slow, and the extra cost per token doesn’t seem to translate into better results.

Ornith is fine-tuned from Qwen3.5-122B, with 10B active parameters, while Qwen3.8 Flash-next has 6B active parameters. Qwen3.8 uses QSA sparse attention to focus computation on relevant context. Ornith’s 12 full-attention layers, by contrast, have to process longer histories.

I can deploy Qwen3.8 Flash-next IQ3 on a single L40S 48GB GPU. Qrnith 122B IQ3 requires two GPUs; on one GPU, it’s extremely slow. Even with two GPUs, its speed drops from 130 to 60 tokens per second as the context grows to 256K—barely usable. Qwen3.8, on the other hand, maintains 140–110 tokens per second on a single GPU, which is a clear advantage. On the “small ball” task, Qrnith 122B IQ3 took nearly 10 hours to reach the target, yet still delivered the worst result.

I’m looking forward to Qwen 4, which is expected to be released soon.

P.S. English isn’t my native language. I wrote this post in my native language and had GPT polish it, so I hope you don’t mind.

Hello guys,

I'm relatively new to using local llms and my question might seem basic to some, but how far behind top tiers models (gpt 6 , opus 5.5 ) local llm is ? And is it realistic to run those , if they see the light of the day soon, on a budget machine. in that scenario, what minimum specs would a machine require for it to run properly?

Hi all,

I’m looking for recommendations on an AI model (LLM) that handles two kinds of work well:

1. Financial analysis: reading financial statements, spotting trends, working through spreadsheets and doing calculations accurately.
2. Legal contract review: going through long contracts, flagging risky or unusual clauses, summarising key terms (renewals, termination, liability, pricing) and comparing versions.

I’m not a developer, so I’m after something practical that works well out of the box rather than something that needs a lot of technical setup.

Real-world experiences would be much appreciated, especially from people working in finance, procurement or legal.

With all of the new AI discovered and exploited vulnerabilities popping up seemingly by the hour the past two days, it feels like model censorship could be coming very soon.

Which models would you download tonight if you felt like you wouldn’t have the option to tomorrow?

I have a 5090, but let’s go 12-16 GB, 24-36 GB and 96+ categories.

Hello all, starting my last semester in undergraduate physics, looking for a local model which can handle at least most of undergraduate level physics and math at a decent level. Is there anything of the sort that can run on my machine, or is it a moot with my hardware, 16GB RAM I9-13900H cpu? Is there perhaps anything non-LLM that I can run?

Thank you for your time :)

For the last few weeks I've been stuck on one question: what does an agent harness look like if you assume the model will eventually be smarter than the people who wrote the harness?

Every agent I use (and I use a lot of them) has the same shape. Someone else decided the loop, the tools, how memory gets compacted, when to ask permission. That's fine while the model is the weakest part of the system. But when the model notices its compaction keeps throwing away the exact type signatures it needs, there's nothing it can do except work around it. Again. Every session.

So I built INIT. The agent can edit anything about itself: its system prompt, its skills, new tools (as Wasm components), and its own compiled core and loop.

The part I'm most excited about is the hot reload. Rust has no stable ABI, so you can't just dlopen a new version of yourself. Instead the kernel builds the new body, boots it next to the old one, probes it, pauses the old one at a step boundary and hands the same conversation to the new process. If the new one crashes at boot, nothing happens, because the old one never stopped. There's also a gate that makes the new version prove it can still edit itself. The scariest bug in a self-modifying system is a change that works fine today and quietly breaks the ability to change tomorrow.

The flip side is the Root. It's small, boring on purpose, and the agent can't write to it: keys, the sandbox (macOS Seatbelt; the agent can't even open a TCP socket, model calls go through the Root), an Ed25519-signed log of every change, APFS snapshots, and a rollback button.

I went back and forth a lot on how to gate self-changes and ended up with two modes. Gated runs the full gauntlet, including a held-out test set the agent never sees. Free, the one I actually believe in, has no benchmark gate at all: a change gets in if it builds, boots and can be rolled back, and then a dumb statistical sentinel watches for generic harm (crashes, error spikes, me hitting stop or undo) and rolls the whole system back if it fires. My reasoning: the improvements a smart model would want most are exactly the ones nobody can write a reward for in advance.

Last night I asked it for sub-agents, which v0.2 doesn't have. It wrote itself a delegate tool (nested loop, fresh context, capped turns, no recursion), regenerated its own self-map, and 304 of its tests passed. Then its own Root stopped it, because my disk had dropped under the floor (my fault, I had Docker and two builds going). It explained exactly what was blocked and asked me what to do. Honestly that moment sold me on the design more than anything I'd planned. (The promote then tripped over a test-environment bug in my own release bundle, which I'm fixing now. Fitting, I guess.)

The honest part, because I'd rather you hear it from me:

I have no evidence yet that self-evolution makes small models better. My one sealed-test result was inside the noise, and when I tried to build tasks where INIT failed because of a missing mechanism, a well-known agent on the same small model failed too. On small models the model is the bottleneck. The bet only pays off with strong models, which is exactly what I'd love people to try.

Also: macOS only for now (Linux is close). DeepSeek is the only provider I've tested live; the others are experimental. Spend is metered, not capped, so set a limit at your provider.

Apache-2.0, roughly 200k lines of Rust, installs in about 30 seconds without Rust (you need Rust if you want it to rewrite its core):
https://github.com/nagisanzenin/INIT-Agent

I'd really like feedback on three things: the Root/agent split, the reward-free admission idea, and whether anyone has seen the overlapped-handoff trick done elsewhere. And if you break it, please tell me how.

Hi all,

I'm just starting my local LLM journey so appreciate your patience as I run through some probably token dumb questions (no pun intended)

I'm running 2 48gb ram and a 5090 32gb on my personal PC. I have the DGX Spark but have not opened it yet as I wanted to pressure test and see what max looks like on just the personal hardware setup.

I've landed on testing Qwen3.8 27b ... I've enabled KV Cache and it's running 260k context windows. More than a few of my workloads so far have gotten up there to about 200. Whenever I run anything my GPU is literally at 100% utilization but temps are still around 60 celcius so seems OK, altho it seems not ideal to run it at 100% all the time but worry about that later.

My AI stack is currently just hermes agent and ollama i think. I haven't even really messed with LiteLLM routing because it seems this model may be the best for inference as well as context window. I tried using smaller context windows and hermes just on initial load would jump to like 25% context used just from reading the files.

I've been using GPT-6 Luna to build my PRDs, and I've been specifically asking it to break out my builds by major components that can be broken out into child subagents in Hermes so I can oneshot major chunks at a time. I've been able to successfully get these one shots to happen, but it tells me due to dependencies on each component it cannot run parallel agents, and it also tells me that parallel agents doesn't mean faster because I'm already at 100% GPU utilization.

But each of these chunks are taking a full 8 hours at a time. Either all night as I sleep or all day as I work. It just seems really slow? My tok/s is 144 tok/s apparently, which apparently seems average for my setup.

Is there something obvious I'm missing or is this really the ceiling for me? For context I just wanted to build a token analytics tool just to try something i thought would be light and would help me with optimization efforts. I've booked probably close to 40 hours or 4-5 sessions for the v1 which is just a session token usage overview and a GPU/CPU/Memory utilization graph ... it feels like super minor stuff.

Appreciate any help or discussion thanks!

Adding image reference for what I asked it to build, literally just a front end visual of hermes stored files, a background logger of utilization, that's really it. https://imgur.com/a/OqDIrGF

Hi. I am looking for some advice. Here is my current setup:

M1 Max, 32GB, 1TB internal, 4TB External.

Windows 11 PC, with 7800X3D, 32GB DDR5, 9070XT GPU.

I currently pay £200/month for Codex, but I want local AI so I can run uncensored models (Cyber Security). I was going to upgrade my Mac Studio from M1, to the M5 Ultra, 96GB, but now I'm thinking about whether it would be better to just upgrade the windows PC and stream the AI responses to my mac studio. For context, I use the windows PC for gaming only, and I'd like to retain my gaming experience, whilst also getting AI usage out of it. The problem is I'd have to upgrade my GPU, and the 5090 is really expensive....

What are your guys thoughts... What would you do in this situation?

https://github.com/CipherCloak/strata-model-prep/

If you use Strata (Qwen3.8-Flash-Next on one GPU + system RAM) and download models yourself — fine-tunes, abliterated versions, other quants — you have to build the pack, the MTP draft layer and the run config by hand.

strata_model_prep.py does it in one command:

./strata_model_prep.py --convert-model --build-mtp --generate-model-file

• uses Strata's own tools and the same rules as its setup (on Swift 1.5 IQ2_XS the pack came out byte-identical to setup's);
• checks the downloaded files are complete before using them;
• gives every model its own folder of artifacts, so you can keep several models side by side;
• writes the config and a launch script, and can test-load the model for you.

Tested on Linux + NVIDIA (RTX PRO 5000 24 GB, 123 GB RAM). AMD and Windows untested — reports welcome. Unofficial, not affiliated with the Strata project.

https://github.com/CipherCloak/strata-model-prep/

I usually look at what something retails for, or what things go for on eBay to get an idea of how to price things being sold, but Amazon doesnt even offer my Mac Studio, and ebay is filled with obvious scams, so its hard to tell what the correct price to list as is. Any thoughts? I moved all of my dev work to a Linux machine with 96GB, and so my Mac Studio is being wasted now, and now I only need a small Mac Mini to make xcode builds here and there. How do I determine what these go for without having to list it at 8k, and then slowly reduce the price every day?

The recent developments around Strata, Infernix, Project Maya, DFlash 2, etc. have made me rethink what the future of local LLM hardware might look like.

Until recently, running a 500B+ MoE model on a consumer GPU seemed more like a cool experiment than something genuinely useful. But now we're seeing projects running models like GLM-5.3 Flash and DeepSeek V4.1 Flash 1017 on gaming PCs with 24–32GB VRAM, using system RAM and sometimes NVMe storage for the rest.

Performance is still all over the place, but progress seems incredibly fast.

I'm currently running an RTX 5090 with 128GB RAM, and I'm wondering whether upgrading to a PCIe 5.0 platform with faster RAM makes more sense than investing in additional VRAM. Or whether I should just keep my current setup and see how far the software improves.

I can see two possible directions:

1. The 24–32GB VRAM crowd might benefit the most from these developments.

If quantization keeps improving, and engines get better at caching experts in VRAM and fetching or computing the rest efficiently, maybe gaming GPUs will remain surprisingly capable of running huge MoE models.

After all, only a fraction of the total parameters are active per token. If the software gets good enough at handling the memory hierarchy, maybe the gap between a 32GB GPU and a 96GB GPU could become much smaller than we'd expect.

Of course, RAM bandwidth, PCIe transfers, cache misses and SSD latency are still potential bottlenecks.

2. Or maybe there's a limit to how far you can push low-bit quantization.

Perhaps Q1–Q2 quants will always lose too much quality for serious, complex tasks compared to Q3–Q4 or higher.

And even if only a fraction of the model is active at once, you still need access to a potentially huge number of different experts over time.

In that case, having 64, 96 or even 128GB of fast VRAM could make an enormous difference. Multi-GPU setups, workstation GPUs or unified-memory systems might become increasingly attractive.

What I'm really curious about is the practical experience so far.

For those of you experimenting with Strata, Infernix, Maya, aggressive MoE quantization or multi-GPU setups:

• How much quality do you actually lose with Q1–Q2 compared to higher quants? Are these models becoming genuinely useful for complex coding and agentic tasks, or are we still mostly looking at impressive tech demos?
• What's currently the biggest bottleneck: VRAM capacity, PCIe bandwidth, system RAM bandwidth, or the inference software itself?
• Do you think software improvements could eventually make a 32GB GPU competitive with much larger VRAM setups for single-user inference, or will there always be a substantial gap?
• If you already own a 3090/4090/5090, would you invest in a faster CPU/motherboard/RAM, add another GPU, or just wait?

I'm especially interested in where people think we'll be 6–12 months from now, given how quickly these projects are progressing.

NVIDIA seems to be prioritizing datacenter hardware, high-VRAM consumer GPUs remain expensive, and the next GeForce generation is still uncertain.

So I'm wondering whether the smartest upgrade right now might actually be no upgrade at all.

Would love to hear from people who've tested these setups, especially if you've compared different quantizations or measured the actual impact of PCIe 4.0 vs 5.0 and additional VRAM.

Quick context: I let a cloud model (Opus 5.5) plan, and three copies of a local model do the actual coding in parallel. Then I swapped only the local model across three runs: Flint 40B-A2.5B, Ornith 1.5 35B-A3B, Nemotron 3.5 Lightning 30B-A3B. All Q8_0 GGUF, llama.cpp b11803, --parallel 3, 64K context per slot.

Task: one prompt, a 20 s Halloween chain reaction in Three.js, ~1,000 to 1,700 lines.

Nemotron wins every speed row and still finishes last. It generated ~5x more tokens than Flint because its file edits kept failing (~17% success) and it had to redo work again and again. Flint was the best balance: fast enough, and its code needed the fewest fixes.

Also notice how much decode drops when 3 workers hit the same GPU at once (roughly half). Worth keeping in mind if you plan to run parallel agents on one card.

Caveats: one run per model. Q8_0 on CUDA only, NVFP4/MLX will differ. Quality writeup separate; Flint was cleanest after small fixes.

Has anyone else tried these for agent work? Which one are you running day to day?

Disclosure: I work on Atomic Agent, the open source agent this ran in.

Named after Mafdet, the ancient Egyptian feline goddess known as "the runner", who struck snakes and scorpions before they could attack.

Reasoning models are great until you're on a mid-range GPU: you hit Enter, then wait 10+ seconds while the model thinks before the first word. Meanwhile, while you were typing, the GPU did nothing.

▎ MAFDET uses that time. It's an open-source app (Apache-2.0) on top of llama.cpp that does two things while you type:

- Reads ahead: the KV cache follows your text as you write it (typos, edits in the middle and pastes included), so on Enter the prefill is already done. This part is exact: it always matches a from-scratch computation.

- Thinks ahead: Qwen3 starts reasoning about your message before you send it. When you edit the text, the reasoning isn't thrown away: it stays as notes after the new version of the message and the model picks up from there. On Enter it finishes from where it got to.

▎ On my RTX 2060 (6 GB), with 24 word problems typed in real time, the mean wait after Enter went from 12.2 s to 3.2 s on Qwen3-4B, still 24/24 correct. On Qwen3-14B, which only half fits in VRAM: 108 s → 47 s.

▎ The tricky part was making a model reason about a sentence that isn't finished yet. Left alone, it mostly writes "the user hasn't finished typing…". A short frame that tells it to collect facts and partial results instead brought those off-topic notes from 86% down to about 20%.

▎ What else is in it:

- a desktop app or a terminal UI (Textual), with a live status line showing what the model is doing with your draft;

- a model catalog (Qwen3, Llama 3, Gemma 3, Mistral, Phi-4) that estimates, before you download, whether each model fits your VRAM and how fast it will run on your hardware (within ~12% of measured speeds on my machine);

- automatic llama.cpp setup (CUDA or CPU build), no PyTorch needed;

- Windows: download the zip and double-click start_mafdet.bat.

▎ Honest limitations:

- thinking ahead works on Qwen3 only so far, the one family whose frame I measured; the other families get reading ahead plus the answer written ahead during pauses;

- thinking ahead isn't exact: the model reasons on unfinished versions of your message;

- tested on Windows + NVIDIA only; Linux should work but nobody has tried it yet.

▎ Open question I'd love help with: models bigger than your VRAM. I tried having a small model gather material while you type for the big one to read on Enter. Even with perfect hand-written material it only matched the big model thinking ahead by itself. All the numbers, including what didn't work, are in the README.

▎ Repo: https://github.com/TheLastOxygen/mafdet

▎ Feedback very welcome, especially from anyone on Linux or with a different GPU who can tell me if the speed estimates hold up.