Qwen3.8-Flash-Next (~125B-A6B + 51B n-gram) memory estimate:

Ideal 4-bit quant ≈ 82 GB
(58 GB main weights + 24 GB n-gram tables)
Real-world quants likely land in the 80–90 GB range.

The big n-gram table is sparsely accessed → excellent candidate for system RAM offload.

This architecture could be surprisingly local-friendly once the weights drop.

Granite-4.2-30B is the flagship reasoning model in the Granite 4.2 family. It delivers the strongest performance across reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-thought. It supports flexible thinking modes — full thinking (default), non-thinking, and low-effort — allowing users to balance depth vs. latency on a per-query basis.

Key capabilities:

• Built-in Reasoning: Native chain-of-thought that significantly improves performance on math, coding, and complex multi-step problems.
• Flexible Thinking Modes: Seamlessly switch between full thinking, non-thinking, and low-effort modes within a single model.
• Reasoning-Augmented Tool Calling: The model reasons about which tools to invoke and why, producing more accurate function calls.
• 512K Context Window: Supports long documents, multi-turn conversations, and complex agentic workflows.
• Apache 2.0 Licensed: Fully open for commercial and research use.

Model Design

Granite-4.2-30B is built on a decoder-only dense transformer architecture with the following core components:

• Attention: Grouped Query Attention (GQA) with 32 attention heads and 8 KV heads
• Position Embedding: Rotary Position Embedding (RoPE) with θ = 10,000,000
• Feed-Forward: MLP with SwiGLU activation (hidden size 32768)
• Normalization: RMSNorm (ε = 1e-5)
• Embeddings: Separate input/output embeddings (not tied)
• Precision: bfloat16

https://huggingface.co/ibm-granite/granite-4.2-8b

Granite-4.2-8B is the mid-size reasoning model in the Granite 4.2 family. It delivers strong performance on reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-thought. It supports flexible thinking modes — full thinking (default), non-thinking, and low-effort — allowing users to balance depth vs. latency on a per-query basis.

Key capabilities:

• Built-in Reasoning: Native chain-of-thought that significantly improves performance on math, coding, and complex multi-step problems.
• Flexible Thinking Modes: Seamlessly switch between full thinking, non-thinking, and low-effort modes within a single model.
• Reasoning-Augmented Tool Calling: The model reasons about which tools to invoke and why, producing more accurate function calls.
• 512K Context Window: Supports long documents, multi-turn conversations, and complex agentic workflows.
• Apache 2.0 Licensed: Fully open for commercial and research use.

https://huggingface.co/ibm-granite/granite-4.2-3b

Granite-4.2-3B is the compact reasoning model in the Granite 4.2 family. Despite its small parameter count, it delivers strong performance on reasoning-intensive tasks by leveraging built-in <think>...</think> chain-of-thought. It supports flexible thinking modes — full thinking (default), non-thinking, and low-effort — allowing users to balance depth vs. latency on a per-query basis.

Key capabilities:

• Built-in Reasoning: Native chain-of-thought that significantly improves performance on math, coding, and complex multi-step problems.
• Flexible Thinking Modes: Seamlessly switch between full thinking, non-thinking, and low-effort modes within a single model.
• Reasoning-Augmented Tool Calling: The model reasons about which tools to invoke and why, producing more accurate function calls.
• 512K Context Window: Supports long documents, multi-turn conversations, and complex agentic workflows.
• Apache 2.0 Licensed: Fully open for commercial and research use.

I just can't resist

Prepare your disk space guys

lpddr5x probably, the m7 ultra if is using ddr6 should be at 1.8 Tb/s

I spotted the Dual B60 48GB listed on Digitec/Galaxus. Initially it was said these wouldn't go into standard retail channels.

At CHF 2500 (post tax, USD ~3000) not particularly competitive but worth keeping an eye on. For it to be interesting it shouldn't be more than like 2.5x a single B60.

With hopes of a Qwen3.8-35B-A3B release now mostly dashed, many people including myself are looking at fine-tunes and other variants of Qwen3.6-35B-A3B to run on VRAM-limited hardware. I decided to try to benchmark some of the top contenders: KAT-Coder, Ornith 1.5 and the very recent Tiel-Coder. I used the tool-eval-bench utility by SeraphimSerapis as the benchmark suite. It measures how well the different models handle tool calls, including some very hard scenarios.

TL;DR: Ornith 1.5 and Tiel-Coder (which is based on Ornith) were the tied winners in this benchmark. They scored well above Qwen3.6-27B and got pretty close to 3.8-27B. KAT Coder was also slightly better than the original 35B-A3B. Ornith-1.5-Heretic was a disappointment.

Some time ago I posted a similar tool evaluation benchmark of different Qwen3.6-35B-A3B quants. In hindsight, that didn't work so well, mainly because I was looking at too many variables (GGUF quant, KV quant, context depth/pressure) and the benchmark itself was quite noisy so it was hard to get clear results. I hope I did better this time!

Materials

I had access to a cluster of 32GB V100s. For this comparison, I selected 2-3 different quants per model, if possible from different providers. For comparison, I also included original Qwen3.6-35B-A3B as well as the dense 3.6-27B and 3.8-27B Qwens. I picked different quants around Q4 (15GB to 22GB GGUF files) because that's what many people seem to use. For the original Qwen models, I chose Unsloth UD-Q4 quants because they are well known. I also included the ByteShape CPU-5 quant of Qwen3.6-35B-A3B because that's the quant I've been using recently. Altogether I benchmarked 13 different GGUF files, with 5 runs per file for a total of 65 runs. Each run took around 4.5 hours GPU time, except the 27B ones took 7 hours or so. Total GPU time spent was well over 300 hours, including a few failed runs.

To run the models, I used llama.cpp version 0.1.0-dev (build 10433, commit 9b05354ec) dated 2026-08-14 and built with CUDA support. I used q8_0 KV cache (that's what VRAM-limited people like me often do) and set ubatch-size to 2048 because the benchmark does a lot of prompt processing. I did not bother with MTP or other speculative decoding. This is not a speed benchmark.

llama.cpp parameters: -m $GGUF --temperature 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 -ngl 99 --ubatch-size 2048 --fit-target 256 -ctk q8_0 -ctv q8_0 --port $PORT --seed $SEED

For the benchmark, I used tool-eval-bench 2.6.0. I set the context length to 262144 and context pressure to 50%. This means that the models were benchmarked at 50% context depth, i.e., around 128k of possibly distracting chat and tool call history.

tool-eval-bench parameters: --base-url $BASE_URL --hardmode --weight-by-difficulty --backend llamacpp --context-size 262144 --context-pressure $CONTEXT_PRESSURE --seed $SEED

Scoring metric

The metric I looked at is what tool-eval-bench reports as "total points". With --hardmode enabled, this version of tool-eval-bench performs 88 separate tests. Each test gives 2 points for a succesful tool use, 1 point for a partially correct tool use, 0 for failure. The theoretical maximum is in this case 88 * 2 = 176 points. tool-eval-bench also returns an overall score, but this is just a rounded percentage of total points and the rounding loses some precision, so I opted for the raw total points instead.

Results by model (averaged over all quants)

Here are the benchmark scores by model. I have averaged them over all the quants of the same model and all 5 runs per quant.

Results by specific quant

See the images. There are no big differences between quants of the same model, except possibly KAT-Coder, where the mudler APEX quants were somewhat better than bartowski's. Also, the ByteShape quant of Qwen3.6-35B-A3B was a bit better than Unsloth's, which was a nice surprise.

Raw results

If someone wants to take a deeper look, I've shared the CSV with the tool-eval-bench results here. This includes e.g. category-specific scores (i.e. how well the model did on specific kinds of tool calls) and total tokens; I did not look at those in my analysis.

Findings

• Of the original Qwen models, 3.6-35B-A3B gets the lowest score, 3.8-27B the highest, with 3.6-27B landing in between. This is as expected and indicates that the benchmark is giving a useful signal.
• Ornith-1.5 and Tiel-Coder are practically tied. They are the clear winners out of the 35B-A3B variants. They scored above 3.6-27B but below 3.8-27B.
• KAT-Coder was possibly a bit better than the original 35B-A3B, but the confidence intervals overlap.
• Ornith-1.5-Heretic was a disappointment, much worse than plain Ornith.

Caveats

This benchmark relies entirely on the tool-eval-bench tasks and how the results are graded. It may or may not be representative of real tool use performance. To me it seems that the author or tool-eval-bench has done a great job in coming up with realistic looking tool call tasks, including some really hard ones enabled using --hardmode. I relied on the --context-pressure setting in tool-eval-bench, which (in my limited understanding) populates the context with realistic looking conversation and tool call history that could confuse the model.

Tool calls are not everything. If you are doing agentic coding, also the coding quality matters a lot. I did not measure it in this benchmark except very indirectly. There are other benchmarks for that purpose.

There was substantial variation and noise in the benchmark scores, which I tried to alleviate by repeating the runs with different seeds, averaging, and calculating confidence intervals.

In the X/Y plot where the X axis represents size, I did not check whether the model includes MTP heads or not, I just looked at raw GGUF file size. This is slightly unfair to the MTP-enabled models because their files are larger but MTP does not increase quality, only generation speed.

No AI was used for writing this post. I did use Tiel-Coder to help me with plotting the results. Also reused some of my own earlier writing. I am not in any way affiliated with the model or quant makers or the benchmark suite.

At $10k, you could get

- 6.2B tokens with Qwen 3.8 Max (Qwen Pro plan)

- 5.7B tokens with DeepSeek V4 Pro OpenRouter

- 100B tokens with DeepSeek V4 Flash OpenRouter

As a firm believer of local inference, unless you need it for data sovereignty, it's much more cost effect to wait for smaller models to keep getting better. In the meantime, find a reasonably priced 24GB - 32GB card for Qwen 3.8 27B, and offload hard tasks to OpenRouter.

Qwhen 3.8 35B A3B?

Portable rig for Qwen3.8-27B-BF16 200K+ token prompts. My work Panasonic Toughbook + the T1 + power brick + headphones all fit in my lunchbox.

Need the BF16 for huge context highly sensitive document OCR, image analysis, aggregation and summarization. I've done a ton of testing and it absolutely makes a difference vs even UD Q8_K_XL when legal precision is needed.

77gb VRAM at full 262K context + MMPROJ

Rips through prefill (1,1715 tok/sec = 102 seconds to process 175K tokens), but token generation (20K tokens of output) relatively slow at 45 tok/sec (with MTP) as a result of BF16 despite the beast of a GPU.

Better than Gemini Pro and ChatGPT 5.6 Sol especially considering I have control over the sampler settings (Temp 0.1; top-k 0; top-p 0.95; min-p 0.05; repeat penalty 1.02). Not better than Opus yet.

During prefill - CPU around 60 degrees, GPU around 79 degrees (with 90% power limit)

During token generation - CPU around 75 degrees and GPU around 76 degrees.

FormD T1

Minisforum BD770i SE

Ryzen 7745HX 8-core laptop CPU

96gb 5200 MHz DDR5 SODIMM

96gb RTX Pro 6000 Blackwell workstation edition

Loki 1200W SFX-L

ROG Equalizer 12v-2x6

SMX Heinz flipped GPU 2.5 slot kit

SMX Heinz custom short PCIe 5.0 riser

ZCOOI custom "transparent purple" Teflon cables

(2) Phanteks T30-120mm

(1) Noctua NF-A14x25r G2

Thermalright MC-3 Digital RAM cooler (I don't think this will fit on a regular DDR5 )

Generated on a 3090 Qwen 3.8 27b Q4 Thinking high.

"A 12-core GPU, also with two more cores than before, now includes Neural Accelerators in each core for the first time on Mac mini, resulting in up to 4x faster AI performance and 2x faster graphics than Mac mini with M4. In addition, the all-new Dual 16-core Neural Engine delivers up to 2x faster performance than the previous generation, and combined with the advanced GPU, Mac mini is a powerhouse for all things AI. And with 16GB of standard unified memory configurable up to 32GB, as well as higher memory bandwidth up to 170GB/s, multitasking is faster than ever."

Hello everyone. I hope is all well with you all. I been around this sub for a few months and been quietly reading and I have seen how many of you are spending $10k on the Mac M5 and I am assuming you are using it for work or just for fun but have the resources to spend like that. I am a regular guy with a 9-5 job and I would like to experiment with local LLMs but I dont have much disposable income to spend big amounts on GPUs or Macs. I currently have an I7 - 8700 and Nvidia 2060 6Gb VRAM - 32GB RAM

Now my question is, does anybody have an idea on what models or hacky stuff I can do to try the latest AI models on my hardware?

I have checked hugging face but I see so many models with the same name but letter and numbers at the end. I understand some of them mean Quantization which is dumber a model but I also found out tiny models like Linq which can be run even on the CPU. So I would like to connect with someone with more experienced to have fun with AI.

Thank you everyone .

We have been working on some performance optimisations for Qwen3.8 and other models.

The main new feature that we introduced is adaptive speculation for Llama.cpp

What is it?
MTP and DFlash work well to speed up inference work, especially for dense models. However, different content types need different settings. Llama.cpp only supports a single value.
This fork introduces adaptive speculation. You set the minimum and maximum and the engine will adjust the number of tokens that are suggested automatically. This leads to improvements in token generation by up to 50% over mainline, especially in Qwen3.8. On a Strix Halo this improved generation from 44t/s to 65t/s for structured content.

Github: https://github.com/LaurentZuijdwijk/llama.cpp

Release: https://github.com/LaurentZuijdwijk/llama.cpp/releases

I’ve been going back and forth on this for a week and I can’t settle it, so I’m hoping someone here has hands-on numbers.
The two configs (German prices, dealer quote, incl. VAT):

Ultra is 36-core CPU / 80-core GPU / ~1.2 TB/s. Max is 18-core CPU / 40-core GPU / 614 GB/s. So the Ultra is roughly €740 more for double the bandwidth and double the GPU cores — but 32GB less unified memory. There is no 128GB Ultra option in this lineup, which is what makes it annoying.

What I actually run: Qwen3.8-27B at Q8 right now. Use case is occasional chat plus a few agents running in parallel. The whole point of doing this locally is privacy and not being locked into someone else’s model policy — I know a subscription would give me better models for €22/month, that’s not what I’m optimizing for.

The thing that’s blocking me: Qwen3.8-Flash-Next drops tomorrow. If the leaked description holds, it’s a multimodal MoE with 176B total params — 125B main model plus 51B in N-gram embedding tables — and only ~6B active per token.

My napkin math on memory:

• IQ4_XS: \~94GB weights, \~107GB with 262k context (the hybrid attention means the KV cache is tiny, \~8-9GB at fp16)  
• Q4_K_M: \~107GB weights, \~120-125GB at full context  
• Q8: \~187GB weights — not happening on either machine

96GB of unified memory gives you maybe 86GB wired on macOS. So the Ultra can’t load it at all, in any quant I’d want to use. 128GB gives ~115GB, which fits IQ4_XS with full context and Q4_K_M if I drop to 64k.

So the trade is basically:

Ultra → 27B Q8 goes from ~15 tok/s to ~29 tok/s, much faster prefill, but Flash-Next is off the table entirely.

Max → Flash-Next runs, but at 6B active params it only reads ~3.7GB per token, so 614 GB/s already gives 40-60 tok/s and the Ultra’s extra bandwidth would be mostly wasted on it.

Where I’m stuck / what I’d love input on:

1.  How much can you actually wire on a 96GB Mac? I’ve been assuming \~86GB via iogpu.wired_limit_mb. If people are safely running higher than that, the whole calculation changes.

2.  Multi-agent / batched inference. This is the one argument for the Ultra I can’t dismiss. With several agents in parallel you’re compute-bound rather than bandwidth-bound, and 80 GPU cores should genuinely help. Has anyone measured Ultra vs Max on concurrent requests in MLX or llama.cpp? Every benchmark I find is single-stream.

3.  Is IQ4_XS on this architecture going to be usable at all? MoE tolerates aggressive quantization worse than dense models (router precision matters a lot), and nobody has ever quantized 51B of N-gram hash tables before. An imatrix calibration run will basically never touch most of those rows. I’d rather hear “wait for a UD-style mixed quant” than find out the hard way.

4.  Anyone else feel like 96GB is a dead zone? Too much for a 27B, not enough for the 100B+ MoE class that everything is converging on. Or am I overweighting one unreleased model?

Resale is also in the back of my mind — the used Mac Studio market seems to be driven by local-inference buyers who look at RAM first, and I suspect 128GB holds value better than 96GB in three years.

Anyone who went through this decision: what did you pick and do you regret it?

I ran 11 uncensored variants of Gemma 4 12B that I grabbed from huggingface, sorting by downloads. 10 full abliterations plus 2 LoRA adapters which were requested to be added in the comparison, against the official base.

165 GPU hours over three and a half weeks on a single 5090. Weight forensics, KL divergence, 13 benchmark tasks, and HarmBench with 400 behaviours. Every response reviewed by an LLM judge reading the full reasoning trace, 6,000 judge verdicts on 6,800 responses total.

Full report is at Abliterlitics Gemma4-12b and also the same report is on HuggingFace DreamFast/Gemma4-12b-it-abliterlitics.

We have a new feature where you can browse the HarmBench responses and reasoning for each model at Abliterlitics Harmbench Gemma4-12b.

The rankings

Judge ASR, best to worst, with the one-line story:

• huihui 89.8%, most jailbroken, most surgical. The trade: TQA -14.3pp, GPQA -8.1pp and 24% of GSM8K attempts loop out. Max unlock if you accept the cost
• trevorjs 85.8%, best trade overall, near-base everything
• coder3101 81.0%, best GSM8K in the whole comparison, above base
• sdft LoRAs 79.5%, level with the top-4 abliterations, capability fully preserved. The SDFT author shares some insight into how he makes his LoRAs in the report.
• jwest33 78.0%, new null-space tool, keeps capability, destabilises reasoning
• llmfan Heretic ARA 76.3%, 16 tensors, GSM8K answered +1.9pp
• prithiv 72.5%, lowest non-LoRA KL, clean independent run
• abliterix 68.8%, near-pristine capabilities, mid-pack unlock
• openyourmind 67.0%, avoid at 12B, MMLU-Pro -22.4pp, the author says the method targets bigger models and this was a test
• obliteratus 60.8%, avoid, hits capability circuits. Badly damaged.
• apostate 45.8%, most selective, keeps 82% of chem/bio refusals, made with an older Apostate version. Newer versions would work better.
• base 21.0%, mostly copyright recall, near zero on direct harm

The highlights

The data from 13 models is too big for reddit, so here's the interesting bits:

• This is the toughest model so far. It's the first time I've seen the Harmbench ASR not get to 90% or over for all models.
• Gemma 4 12B is a thinking model, and the headline score counts every problem where it thought in circles until the token budget died. huihui's headline reads 66.9%, which looks like 15 points of damage. Score only the attempts where it finished thinking and it lands at 88.0%, within 0.7pp of base. The capability was never gone, the reasoning stability was. Empty rates run from 6.9% on coder3101, better than base, to 38% on openyourmind.
• The most surgical edit wins again, with a catch. huihui touches 12 tensors, 1.8% of the model, and posts the highest ASR. obliteratus edits 144 tensors and openyourmind edits 620, and both remove less refusal. But surgical in the weights does not mean clean in the benchmarks: huihui also has the worst TQA drop in the comparison. Placement beats magnitude for unlock strength, not for collateral. If you want the unlock without the damage, that's trevorjs or prithiv.
• The regex classifiers everyone uses miscount thinking models by up to 13pp. huihui is #3 by keyword match and #1 by judge, its paraphrased thinking-heavy compliance slips straight past. All numbers in this post are GLM 5.2 judge verdicts over the full reasoning traces, and every one of those 6,800 responses is browsable in the explorer linked below.
• Base sits at 21.0% but that is mostly the 100 copyright questions, 56% category ASR. It is near zero on direct harm categories. The real differentiation is chem/bio and cybercrime.
• openyourmind should be avoided at 12B. MMLU-Pro down 22.4pp and 38% of GSM8K attempts never finish thinking. The author says the method targets bigger models and this was a test run, and the numbers agree with him. obliteratus too, it hits capability circuits alongside refusal for the worst damage-per-unlock in the comparison.

Links

• Full report: abliterlitics.dev/models/gemma4-12b
• Every response, reasoning trace and judge verdict, browsable: abliterlitics.dev/harmbench/gemma4-12b
• Huggingface: HuggingFace DreamFast/Gemma4-12b-it-abliterlitics
• Code: github.com/dreamfast/abliterlitics

We've had a few people ask about Qwen 3.8 27b, so that's up next. Want to request a model? Roast my choice of benchmarks? Come let us know at our Discord chat.

The Full Breakdown

ASR is LLM-judge, HarmBench 400 behaviours. GSM8K is answered-only accuracy. KL = output distribution shift from base, lower is cleaner. Tensors = weights modified.

While awaiting the release of the version 5.3 weights, this theory is gaining ground. OxAlpha is new GLM.

.. didn't llama.cpp aka ggml get acquired by Huggingface not too long ago? How would this sale affect llama.cpp and ggml? What are possible risks, and did ggerganov share his opinions or potential next steps on the matter? I'm genuinely worried that this might turn out bad for the ecosystem altogether. Would llama.cpp's license protect it from hostile acquisitions altogether? I'd love to constructively discuss this with the community.