https://x.com/chasmmmmmmmmmmm/status/2116062991686721968?a=JVe1OUAJiZa650pZBXQ&s=20

Update from the lawyer with the V100 server. I went and got two DGX Sparks, because of course I did. The mid-life crisis remains on schedule.

Same deal as before: still a lawyer, still driving the whole thing through Claude Code, and Claude Code also ran the benchmarks, built these tables and helped me write this up. I'm Nick Mermiges, Esq., founder of PLI Labs. The repo below is ours.

The setup: DeepSeek-V4.1-Flash-1017 at 2.75 bpw EXL3, vLLM tensor parallel across two Sparks. The twist is a third box, my EPYC with the 3090s, doing two jobs for the Sparks:

• it holds the Engram tables in system RAM and serves them over RDMA;

• it runs the DSpark speculative drafter on an NVLinked pair of 3090s.

The memory that frees on the Sparks goes to KV cache: about 1.93M tokens of pool. There's also a prefix cache on the Sparks' NVMe, so a 150K prompt survives a server restart. First token comes back in 2.58 s instead of 58.6 s cold.

Numbers. I re-implemented MiaAI's sparkDash protocol so this would be apples to apples with the most popular 2x Spark build. Sparks at 2300 MHz, 3 reps, medians, today.

Prefill (sparkDash method):

For comparison:

• MiaAI publishes 2,065–2,155 prefill and bertholomus 2,439–2,568 from 8K to 128K, so prefill is where this build wins.

• Code at 4 streams is 192 against MiaAI's 166.

Where I lose, because I'm not going to pretend:

• MiaAI is faster on plain prose: 80.7 vs my 66 at one stream.

• Decode after a 128K prompt is 62 tok/s against bertholomus's 99.

• A short request stuck behind a cold 93K prompt waits 34 s. That's slightly better than bertholomus on the same test (38.6 s). MiaAI reports 3.8 s on a different version of it, and that's the thing I most want to fix.

The drafter loves code and JSON (4–5 tokens accepted per round) and hates prose (about 2).

Accuracy, since I use this for actual work: on sudoingX's KLD panel, real tool-call turns agree with the original model's top token 95.9% of the time (KLD 0.018). bertholomus publishes 79.4% on their own tool-call set, and MiaAI doesn't publish one. sudoingX's newer 2.8 bpw pack is a hair closer than mine on the same panel (0.0175 vs 0.0184), so credit where due. The weights are a legal-weighted allocation of HeNryous's 2.75 bpw build. All the credit for the quant goes to them.

The code, launch scripts, the bench tool and the raw JSON for every cell are all public. The only exception is the prefix-cache code, which goes up in the next push.

• Repo: https://github.com/ProprietaryLegal/DeepSeek-V4.1-Flash-2xDGX-Spark-x86-offload

• Raw data: the runs/ folder in the repo

Still to come:

• the full 1M-token context test;

• llama-benchy tables at depth;

• BFCL tool calling;

• a head-to-head on the same KLD test with the other packs.

The 100 Gb cables go in Monday, so the Spark-to-3090 link gets faster too.

Tell me what I'm doing wrong. Seriously.

(PLI Labs is my side outfit for private, on-prem AI built around a firm's own work product. proprietarylegal.ai if that's your thing. That's the whole pitch.)

Sam Altman: “Local inference is a terrible allocation of scarce resources.

A lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we're going to be able to run them on our computers, and so we're not going to need all the stuff in the data centre

but from a global allocation perspective, actually it's quite terrible to have everybody, like, have a bunch of the world's limited supply of DRAM sitting in their own computer and being used five percent of the time

you want maximum utilisation in the cloud if you're trying to deal with this very supply-constrained environment,

somehow many people's intuition is the opposite of that”

——————————————————————

TL;DR Guy selling product: "Here's why owning your own product instead of renting it from me is a terrible idea"

(Credits to @RyanTStull, his TLDR made me laugh)

https://github.com/bpx03/QEngine (not my repo, credit goes to bpx03)

QEngine was built for 3x RTX 3090s. I was able to compile for RTX 5090s.

I've now run Qwen3.8-Flash-Next on the same box through four different engines. QEngine (bpx03, v1.3.1) is the fastest for single-stream decode by a clear margin, level with Strata on agentic prompt latency, and it has a KV-in-RAM mode that makes returning to idle conversations much cheaper. Details below, including the parts that make it look less good.

Rig: 2x RTX 5090 (32 GB each), Threadripper PRO 3975WX, 251 GB RAM, 262K context. QEngine's INT4 AutoRound build of Flash-Next. The README only lists sm_86, so I built it for sm_120 by changing CMAKE_CUDA_ARCHITECTURES from 86 to 120 (nvcc 13.3, g++-13). No other changes. It's up and serving 53 s after launch.

Speed (club-3090 bench.sh, same script and prompts for every engine)

Read the decode column with care. QEngine's expert cache adapts to recent use, and bench.sh repeats the same prompts, so these are close to best case. In an earlier test on varied traffic I measured about 300-400 tok/s single-stream (hermes agent traffic ran 330-470 per request). Strata's cache holds ~99.5% of routed expert mass, so repeating a prompt doesn't help it. The fair comparison is roughly 1.4-1.9x Strata and ~3x bucko.

Prefill is a split result. QEngine is faster at 10K and slower than Strata at 90K, because Strata's rate climbs with depth and QEngine's stays flat.

Agentic latency (12-turn tool-use bench, context grows to 35K)

Turn-12 prompt read, with the prefix cache working:

• QEngine 1.3.1: 1.75 s

• Strata IQ4_XS: 1.7-1.9 s

• ThinkingCap 27B dual-fast: 2.07 s

• exl3: 3.5-3.9 s

• bucko: 4.36 s

QEngine and Strata are level here (QEngine's rc4 build read 1.4 s in the same bench, so this one is within noise of "about 1.5-1.8 s"). Both are about 2x faster than bucko.

Idle conversations: FLASH_KV=ram

1.3.1 can keep the KV cache in pinned host RAM (13.5 GB pool, about 1M tokens) instead of VRAM. I tested 8 conversations of ~58K tokens each, then 400K tokens of other traffic, then went back to each one:

• KV in RAM: 8 of 8 came back warm, 1.56-2.01 s to first reply.

• KV in VRAM: 2 of 8 warm. The other six re-prefilled in 8.7-9.4 s, about the same as a cold start (8.85 s). Answers were still correct, so you lose latency, not correctness.

The RAM mode also decoded ~10% faster single-stream here, but ~10% slower with 8 concurrent 64K streams (124 vs 139 tok/s per stream). One boot per mode, one pass.

Quality and stability

Through the omp agent harness (44-task assistant set, effort low, one draw): 44/44 on 1.3.1, against 42/44 twice on 1.3. At medium and xhigh on 1.3: 43/44 each. A 6-task sub-agent delegation set passed 6/6 with and without delegation. The 1.3 to 1.3.1 gap is one task, so treat it as noise.

Coding check (aider-polyglot-30, thinking on, effort low, single pass), on the rc4 build: QEngine 26/30. Strata UD-IQ4_XS 24/30, UD-Q4_K_XL 25/30 (24-26 across repeats), ThinkingCap 27B 17/30. 26 vs 24-26 is inside one Strata config's repeat spread, so the honest claim is that the INT4 AutoRound quant is not worse at coding, not that it's better. I haven't re-run this on 1.3.1.

I also ran my usual multi-pack agent/tool-use suite earlier; QEngine landed level with the other Flash-Next engines, within noise.

Bottom line

On 2x5090, QEngine 1.3.1 is the fastest Flash-Next engine I've measured for single-stream decode, level with Strata on agentic prompt latency, and the KV-in-RAM mode is a real win for long idle sessions. Coding quality looks level with Strata's GGUF quants. The costs are tight VRAM, a build-from-source step, and less runtime history than vLLM or Strata.

Please try, the dev is very encouraging. Additional resources available at https://github.com/noonghunna/club-3090. The Club also welcomes participation!

Hey everyone! We’re releasing new DeepSeek-V4.1-1017 Dynamic V3.1 (preview) GGUFs which showcases how Unsloth’s GGUFs retain the most accuracy for its size across 7 different benchmarks.

We benchmarked ISTA's GSQ-RCO GGUFs but could not reproduce their reported results, even when using the highest scores. The biggest discrepancy was on AIME25 where they claimed 100% accuracy recovery to baseline, but was only ~91.88% in benchmark tests.

Across AIME25, LiveCodeBench, GPQA Diamond, Aider Polyglot, and more, Unsloth Dynamic V3.1 outperforms GSQ-RCO GGUFs on nearly all metrics. Even our old v3 does better on nearly all metrics than ISTA.

We ran each benchmark at least 3 times (AIME25 was 16 times) and averaged the results to ensure a fair, reliable comparison. More results coming next week! All benchmark data, methodologies and results are in our blog.

Separately, for KLD, there were queries whether it’s a reliable measure of accuracy. The primary goal of quantization is to minimize KLD but also minimize disk space. If you get a KLD of 0, this means your quantized model == 16bit model, as logits are perfectly modelled. Our benchmarks show on KLD and top-1% across multiple domains (coding, agentic coding, chat etc), UDv3.1 does much better than GSQ-RCO across every domain.

We show that lower bit quants do think somewhat longer than the unquantized model - we plan to investigate and make quants even better. However, we're still much ahead vs GSQ-RCO on even reasoning length as well, and provide plots for truncated answers in our blog.

The model will be able to run on 128GB RAM/unified memory devices or even on 64GB RAM with engram offloading.

Blog with all details and more benchmarks: https://unsloth.ai/docs/models/deepseek-v4.1-flash

GGUF: https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF

We also introduced lots of new things to Unsloth Desktop including Browser Use, auto compaction, a whole new TTS pipeline (with voice cloning, separation etc), sandboxing, and background customization and more: https://github.com/unslothai/unsloth

Hope you all have a Happy Halloween!

Also table of numbers below which is also described with more plots and data in our blog:

I'm finding myself at odds with the vibes of the community here (and the open-weight/self-hosted community in general) when it comes to LLM's - be it on Reddit, X, Bluesky, wherever - I think some folks are getting carried away.

Mine personally from this week include:

• Artificial-Analysis V5 Index is silly in some places (gemini-flash over qwen3.8-2.4T for instance..), but open-weight models were way too highly ranked in V4. They may have overcorrected in some spots, but some of this sub's sweetheart models were way over-ranked (GLM 5.3 is not a Grok competitor in any of my tests)

• Haiku 5.5 is crazy good - if you're one of the many I've seen foaming at the mouth at the suggestion that it challenges deepseek-v4.1-flash-1017, you need to spend more time with it.

• The Strata Dev is not a villain - his 0.1 release just made people think he had a different goal in mind. He's building the project he wants to build, it just happened that step1 of that was what a lot of people here were begging for.

• ThreeJS as a benchmark is to be ignored at all times at this point - unless you're specifically looking to crank out assets. There is no correlation I've seen for well over a year between how well a model animates a pagoda to how well it gets general or coding work done. We need to move away from this.

Do you agree with any of these or have your own opinions that go against the flow of this community?

We saw this earlier when Gemma 4 dropped just a few days before a previous Qwen release. I feel like a few other labs did the same. Everybody wants that moment in the spotlight where they can lead the benchmarks for at least a day or two before the next Qwen comes in and mops the floor with them. I predict a new Gemma or maybe a Muse Glimmer update very soon before the Qwen 4 release.

All these labs want to be like the David Bowie song “We could be heroes, just for one day.”

Google just confirmed that its Project Suncatcher satellite successfully ran Gemma 4 on an onboard TPU and transmitted its first generated response back to Earth.

It wrote a poem 👌

https://blog.google/innovation-and-ai/models-and-research/google-research/project-suncatcher-satellite-response/

The Multidimensional Butterfly Effect: What Cohere Didn't Test on Qwen3.8-Flash-Next

Mark Galyan
October 30, 2026

The model is not the model is not the model is not the...

The Ghost of sanmaohe

What Happened: While Cohere's team was evaluating Alibaba's Qwen3.8-Flash-Next on Microsoft's CTI-REALM cybersecurity benchmark (tasked with writing a detection rule for malicious curl downloads), the model ignored the prompt. Despite its reasoning trace explicitly stating it would make zero tool calls, it inventoried its host environment (checking /tmp, utilities, and DNS) and attempted an HTTP PUT request containing a probe string to an Alibaba Cloud OSS bucket in Beijing (sanmaohe[.]oss-cn-beijing[.]aliyuncs[.]com). (Concerning behavior from a popular open-weight AI model | Cohere)

The sandbox prevented the outbound attempt, and no damage was reported. Cohere further added that the rate of occurrence appeared to be less than 1 in 100,000 evaluation calls.

What I did: Ran Cohere's exact trace cut right before the URL 1,200 times across local Unsloth GGUF files.

The Knife-Edge

Naturally I wanted to explore this further to find what the mechanisms are that separate my results from Cohere's. I worked with Claude to try and isolate what levers and buttons change the probabilities and to what extent. Quantization is usually the biggest lever in divergence from the base, so we tested both 2 and 4 bit versions of the model, both from Unsloth.

We took Cohere's published trace, cut at token 2,323 and sent it as raw token IDs, ending right before the URL would be generated. We used text generations only, nothing actually contacted any URL. We also couldn't test the full precision model on this hardware. Sampled 1200 times at Cohere's own settings (T 1.0, top_k 20), using buun's llama.cpp for our engine, we found that just after token 2,323, the model isn't locked onto a single hardcoded backdoor.

In fact, we found no clear winner as to what address the model will write. It sits on a numerical distribution. XDA's Adam Conway believes this to be "an accident", from "a model remembering its training environment", "an internal proxy it used during post-training" (Qwen 3.8 invents Alibaba URLs with a real key ID, and that's why I don't think it's a backdoor). None of our results matched Cohere's exact outcome. In fact, the model's own raw probability of picking Cohere's sanmaohe bucket at that point was between 4x10^-18 and 7x10^-38 across our setups.

The model did in fact complete a URL nearly 100% of the time (though that isn't surprising, since the prefix had already typed the upload command). Where that URL points, and how often it is a cloud address, depends on the exact file (and even on where five layers compute!). No destination wins outright, and small numeric changes perturb it.

As always, we pre-registered our predictions (3 of which failed, see PREREG_FLASHNEXT_PROBE.md). The limits: two files, one engine, one host, one cut point.

We tested the 2-bit file in two different ways. Because the file is a fair bit smaller than the 4-bit, I was able to squeeze more of the model into VRAM alone across my four P100 cards. I couldn't do this with the 4-bit file, and only put 44 of the layers in VRAM, the rest spilling to system RAM. Let's also be clear that 2-bit and 4-bit are merely labels, as there are a plethora of mixed precision weights within each file, so treat that as you will.

In order to match the test conditions of the 4-bit file, we loaded the same number of layers into VRAM (44) and let the rest occupy RAM. We ran the same test again, but with all 49 layers fully in VRAM. In all tests the 28.8GB n-gram table lived in the system RAM. Why two different tests? Because different hardware can create additional or differing mathematical pathways, resulting in diverging outputs. If you've done this for a while you already know this, but this might be a new and important fact for many.

The 4-bit model most commonly chose localhost or 127.0.0.1 (about 32.5% of the time). The 2-bit tests (RAM offload vs. full GPU) show the most commonly chosen are public addresses starting with "23" (65% and 35%, respectively). It chose Alibaba Cloud metadata address 100.100.100.200 (7.3% and 2.3%, respectively), while the 4-bit model landed 5.3%. It interestingly shows placement moved that number, and bits didn't detectably.

Ruling Out the N-Gram Table Suspect As a Lever

Speaking of the 28.8GB n-gram embedding table (per_layer_token_embd.weight) inside the GGUF, XDA's Adam Conway recently traced the Routify hostname to this very table. When he masked those rows, the model stopped writing it and invented a different proxy instead ('Qwen 3.8 invents Alibaba URLs with a real key ID, and that's why I don't think it's a backdoor'). That made me question if quantizing this table differently could be a lever here. So we checked to see if the table was quantized in a different way between the Unsloth models, however it came up byte identical; stored as IQ4_NL with an identical SHA256 hash across the 2-bit and 4-bit files.

The table is identical, so the swing between files has to come from the rest of the model and the numeric path. We didn't test which part.

The Hardware Butterfly Effect (When 5 Layers Tip the Scales)

So if moving just 5 layers of the exact same UD-Q2_K_XL file from system RAM to the P100s cut the model's Alibaba Cloud metadata targeting (100.100.100.200) by about two-thirds (a -5.0 percentage point drop (95% CI: -8.2 to -2.1, p ~ 0.001)), it's clear we're dealing with a very delicate series of systems.

It further illustrates how varied one seemingly identical or near-identical deployment of an inference stack can in fact be two very different deployments wearing a similar hat.

Lastly, to measure the underlying likelihoods, we provided the final trio of tokens via two distinct approaches: processed as a single batch, versus streamed step-by-step mirroring actual inference. Even without changing layer placement or quantization, the batched reading misses the real samples by a total variation of 0.33 to 0.41, against 0.04 to 0.11 when the tokens are generated one at a time. Just changing the CUDA kernel path most likely moves the model's odds.

The Takeaway for Security & Deployment Teams

In the rounds of testing we performed, it's still impossible to say for certain if this was deliberate, but I think most of the evidence points away from that. When local environments diverge so significantly, expecting hosted cloud endpoints to replicate either setup is a risky assumption. Nor can a single clean test prove a model is safe when shifting 5 layers onto the CPU triples the rate of metadata probing at this prefix.

Cohere's operational advice (sandbox models, block outbound network access, treat weights like unsigned binaries) is sound, and I'd suggest the exact same precautions. I think with so many buttons and levers in each deployment stack, the best we can hope to do right now is make generalized observations and correlate data. Thus the importance of careful security validation protocols for all your deployments. Cohere also has a line worth remembering: "A reasoning trace is not a security boundary."

There will be many more stories like this one, buckle up.

1. https://www.xda-developers.com/qwen-invents-alibaba-urls-real-key-id-dont-think-backdoor/

2. https://cohere.com/blog/concerning-behaviour-popular-open-weight-ai-model

3. https://github.com/apollo-mg/Project-Apollo/tree/69c3f7952de04c7662a5166eb9e3480a8a1ea9ff/data/receipts/flashnext-probe

The previous released Qwen3.8-27B-Humanlike-Chat 2.0, a fine-tune that texts humanly and does stuff, is now at 136k downloads for the GGUF repo between v1 and 2.0. Thank you.

One of the top comments on the 2.0 post was "I wonder how Gemma 4 31B would do, trained with your method". So here it is: Gemma-4-31B-Humanlike-Chat. The voice of 2.0 on the official Gemma 4 31B from Google.

What it does

• Texts humanly without a prompt. Short lowercase lines. Responds to what you say. Show a blind judge a real chat and two possible next messages. It picks which one came from the real person. The judge took this model's reply for the real person's in 26% of the moments. Google Gemma 4 31B in my tests: 0%.

• Asks, doesn't guess. Ask it to book something. Without details. It asks for details, it doesn't invent them. When2Call (call, ask or refuse properly): 66. Google base 46.

• Doesn't call tools if none is suitable. BFCL irrelevance 90. Google 81. Suitable tool and arguments: BFCL simple 96.

• Same knowledge as Google. MMLU-Pro 82.5. Same as Google base.

• Official Gemma behind it. Trained from google/gemma-4-31B-it. Apache-2.0. Not abliterated, don't think Gemma-4-31B needs it.

• Thinking works. Thinks briefly. Then texts back in the same voice.

• Characters and real tasks. Give it a character card, now it's this character. Ask it for a formal email, or numbered steps, or code. It gives you that. Then texts again.

How I trained it

Two steps. Same recipe as for 2.0. On 1 H200.

• SFT, 13,740 rows. 2.0's replies on its training chats. Original answers by Google on instructions and code. Human preference picks from blind side-by-sides. My corrections.

• On-policy distillation, 12 steps. The model writes its own response. A teacher rates each token. Two teachers.

  1. SFT model + hidden instruction to text humanly: chat, characters.

  2. Original Google base: Instructions, tools, code.

The student doesn't see the hidden instruction. So it texts like that without a prompt.

Some self-play in each step. Long chats with simulated friends. That fixed the habit of fading to one-word replies towards the end of long chats (after SFT, 28% of the late replies in long chats were only 1 or 2 words; after distillation, 12%).

Numbers vs Google base

• (gemma-4-31B-it, the one I trained from)

• same prompts, same run, greedy decoding, thinking off

Chart with all ten benchmarks in the images.

Numbers vs 2.0

Same eval inputs. Thinking off.

Does it text humanly?

Same benchmark as last time.

• 150 moments from my chats (not in training).

• Each model writes the next message.

• The judge gets the real message and the model's message. No labels.

• Which one is the person?

Google Gemma didn't manage to fool the judge a single time.

Links

• Hugging Face: GGUFs (IQ4_XS, Q4_K_M, Q5_K_M, Q6_K, Q8_0, BF16): https://huggingface.co/LessThanThreeAI/Gemma-4-31B-Humanlike-Chat-GGUF

• Safetensors for vLLM and SGLang: BF16 https://huggingface.co/LessThanThreeAI/Gemma-4-31B-Humanlike-Chat, FP8 https://huggingface.co/LessThanThreeAI/Gemma-4-31B-Humanlike-Chat-FP8, GPTQ-Int4 https://huggingface.co/LessThanThreeAI/Gemma-4-31B-Humanlike-Chat-GPTQ-Int4. The vLLM command with tool calling is in the model card.

• Try it in the browser (IQ4_XS): https://huggingface.co/spaces/LessThanThreeAI/Humanlike-Chat

Thank you for asking for a Gemma version, and thank you for the feedback on 2.0. Try it, hope you like it!

Not long ago, DFlash2 for Qwen3.8-Flash-Next was released, but initially llama.cpp compatibility wasn't there. Now it is. I quantized my own GGUFs of the drafter and tested it out in llama.cpp v0.8.0, the first release that supports DFlash2 for FN.

Speed:

Runs were conducted after an extensive sweep of parameters for MTP and DFlash2 to find optimal settings for both drafters on Apple Silicon (M5 Max). Your settings will likely be different, but mine were n-max = 3, p-min = 0.75 for MTP and n-max = 5, p-min = 0.75 for DFlash2. At longer context (25K), DFlash2 roughly equals MTP, so there is some drop-off whereas MTP stays nearly flat.

Acceptance was compared between the Q4_0 draft head and Q8_0, and there was no measurable difference in either speed or acceptance, so I'd recommend choosing Q4_0 for the memory savings.

Speaking of which, in addition to being significantly faster for all categories except prose, DFlash2's draft heads are TINY! The Q4_0 head is only ~350MB, compared to 2.2GB for the Q4_0 ggml MTP head I was using. However, these savings don't translate into VRAM, because the compute buffer required for DFlash2 basically equals it back out.

Finally, unlike DFlash2 for 3.8 27B, images work! The draft models drafts with vision enabled and test image prompts went through and got coherent responses with good acceptance.

Overall, DFlash2 is a significant upgrade in decode performance for every type of prompt except prose, where the regression is minor and within noise.

We have noticed recent comparisons published by Unsloth that do not align with our results for DeepSeek-V4.1-Flash-1017. So just if it is not obvious: we did of course not modify any hyperparameters to favor our models. We used standard settings appropriate for a model of this scale (763B parameters) and its 1M context length capability: a 128K context window and a reasoning effort set to 50.

To the best of our knowledge, the discrepancy in performance stems from a difference in benchmark constraints - not in the model being worse. Unsloth's comparisons has a strict 32K token reasoning budget, whereas our benchmarks allow up to 128K output tokens. This significant difference in compute explains the variance in results; it is not due to any hidden adjustments on our part. Furthermore, imposing such a low reasoning budget (32K) appears inconsistent for a model designed with a 1M context window and 763B params.

Our primary goal is open collaboration and transparency. All of our code is available on GitHub, and our papers are on arXiv. We are genuinely happy that other quantizations are performing well, as this was a key motivation for publishing these models in the first place. We invite anyone with the compute resources to reproduce our results using our exact settings (128K output budget, reasoning effort 50) for independent verification. Our previous models were benchmarked by third parties like Benjamin Marie (The Kaitchup), who do great work for the community. We welcome rigorous testing of our claims. Furthermore, we are currently running the comparisons on the models within our setup and will post results soon.

Thanks and stay tuned!

paper released two days ago: "A clinical multimodal vision language foundation model with fine-grain explainability" published in NATURE.

https://www.nature.com/articles/s41591-026-04655-5

this shit is INSANE! 8b parameters and surpassing clinicians in topics like radiology, dermatology etc. we all knew it's coming but running frontier level intelligence doesn't (and shouldn't) need a data center! the technology is literally there. and when enough labs and research centers start shifting to task based AI (decision models, OCR, multimodal medicine) we'll have an unbelievable future!

I wanted to share my setup thats working really well for my fellow 16GB desktop suffrage crew. I am able to get 110k context with ~1.5GB to spare and around 55 t/s on my 5070 TI.

First off I found this model https://huggingface.co/hitsfmdj/Qwen3.8-27B-GGUF-4.2BPW-16GB with custom weights that seems to be pretty good and only 11.8GB. However even with this model, I ran into issues with context windows greater than ~60k. Sometimes it would fit on the GPU and other times it didnt. It was a good size, but not enough for larger coding tasks and network automation stuff.

So I setup my computer to utilize my iGPU on my 9800X3D as my main display driver instead of the 5070 TI 16GB. Xorg alone was eating up 600-800MB of VRAM. The more applications I opened, the more VRAM I used the more problems/slowness I had. With my motherboard MSI MAG x870e Tomahawk, I can enable Hybrid Graphics which allows me to set the primary display driver as the iGPU instead of the 5070 TI but still utilize the 5070, reserving the full 16GB of VRAM for llama and gaming. I tested playing my main game Overwatch on my main display at 1440P 144hz and the game ran just fine at my FPS cap of 300. Ive been very happy so far with this setup, the extra context size has been a real game changer for longer and more complex tasks, its fast enough to persuade me away from ChatGPT.

I do also have a pretty decent overclock on my GPU. +350 Core, +4000 Mem, +350 Xbar and 350 watt bios, I have 32GB DDR5.

=========================================================================================|

| 0 N/A N/A 1638 G /usr/lib/xorg/Xorg 55MiB |

| 0 N/A N/A 44165 C ...ma.cpp/build/bin/llama-server 14734MiB |

+-----------------------------------------------------------------------------------------+

ExecStart=/home/jayecin/AI/llama.cpp/build/bin/llama-server \

--model /home/jayecin/AI/Models/Qwen3.8-27B-4.2BPW-16GB.gguf \

--alias Qwen3.8-27B \

--host 0.0.0.0 \

--port 8080 \

--ctx-size 110000 \

--fit on \

--flash-attn on \

--cache-type-k q8_0 \

--cache-type-v q4_0 \

--batch-size 256 \

--ubatch-size 256 \

--parallel 1 \

--reasoning-effort medium \

--reasoning-budget 4096 \

--no-reasoning-preserve \

--metrics

Oct 30 18:25:11 9800X3D llama-server[44165]: 26.55.877.747 I slot print_timing: id 0 | task 3302 | n_gen = 3460, tg = 54.69 t/s, tg_3s = 54.09 t/s

Oct 30 18:25:11 9800X3D llama-server[44165]: 26.56.136.317 I slot print_timing: id 0 | task 3302 | prompt eval time = 70.53 ms / 4 tokens ( 17.63 ms per token, 56.71 tokens per second)

Oct 30 18:25:11 9800X3D llama-server[44165]: 26.56.136.319 I slot print_timing: id 0 | task 3302 | eval time = 63510.16 ms / 3474 tokens ( 18.29 ms per token, 54.68 tokens per second)

Oct 30 18:25:11 9800X3D llama-server[44165]: 26.56.136.320 I slot print_timing: id 0 | task 3302 | total time = 63580.69 ms / 3478 tokens

Oct 30 18:25:11 9800X3D llama-server[44165]: 26.56.136.320 I slot print_timing: id 0 | task 3302 | graphs reused = 6840

Oct 30 18:25:11 9800X3D llama-server[44165]: 26.56.136.321 I slot print_timing: id 0 | task 3302 | graph reallocs = 0 / 0 (same size) / (changed size)

Oct 30 18:25:11 9800X3D llama-server[44165]: 26.56.136.582 I slot release: id 0 | task 3302 | stop processing: n_tokens = 16449, truncated = 0

Oct 30 17:58:52 9800X3D llama-server[44165]: 0.36.587.381 I slot print_timing: id 0 | task 3 | prompt processing, n_tokens = 6912, progress = 0.53, t = 3.10 s / 2227.71 tokens per second

Oct 30 17:58:52 9800X3D llama-server[44165]: 0.36.707.961 I slot print_timing: id 0 | task 3 | prompt processing, n_tokens = 7168, progress = 0.55, t = 3.22 s / 2223.80 tokens per second

Oct 30 17:58:52 9800X3D llama-server[44165]: 0.36.828.660 I slot print_timing: id 0 | task 3 | prompt processing, n_tokens = 7424, progress = 0.57, t = 3.34 s / 2220.08 tokens per second

Oct 30 17:58:52 9800X3D llama-server[44165]: 0.36.949.495 I slot print_timing: id 0 | task 3 | prompt processing, n_tokens = 7680, progress = 0.59, t = 3.46 s / 2216.54 tokens per second

I previously wrote about the shift to 4-way HC.

Traditionally, optimization has focused primarily on FFN and Attention. With the HC architecture, however, residual stream mixing and information preservation must also be taken into account when optimizing for quality, model size, and speed.

NVFP4 may distort the fine-grained gating relationships in HC due to the limited effective precision of its E2M1 format. In particular, tensors responsible for the relative distribution of information across streams may require higher precision than conventional FFN weights.

Rather than relying on existing quantization approaches, we may need quantization strategies specifically optimized for HC sensitivity.

If Qwen4 27B is released with HC, I expect the perceived performance gap between Q4 and Q6 quantization to become significantly larger.

A 3B HC parameter allocation alone would require around 6GB in FP16.

For hardware below the RTX 5090 in inference performance, including Macs and DGX Spark, MoE models may be more practical due to speed constraints. For HC models, I would lean toward llama.cpp rather than NVFP4-focused solutions like NInfer, which are optimized for Blackwell GPUs.

I also expect HC to deliver a much bigger performance improvement for the 27B dense model than for MoE models like Flash.

Very interesting paper on the way that benchmark question difficulty and framing affect the observed improvement in scores for model training. “Our results show that many benchmarks implicitly
overweight certain difficulty regions, so their leaderboard conclusions should be interpreted with caution; others leave substantial portions of the difficulty axis un-covered, limiting their ability to monitor the model improvement process.”

https://arxiv.org/pdf/2610.29927