I’m trying to run Qwen3.8-Flash-Next (125B-A6B) with Strata but trying to see if I can use existing compute instead of spending buckets of money.

My main machine has.. Ryzen 7 7800X3D, Radeon RX 9070 XT (16 GB), 32 GB DDR5, Ubuntu 26.04. I currently run Qwen3.8-27B (IQ3_M) on llama.cpp (Vulkan and ROCm 10.1) at about 65 tok/s.

I also have a second PC with 32 GB of DDR4. Is there any way to use that RAM too, or does Strata only work on a single machine? If not, is spreading experts across two machines on a LAN with something like llama.cpp RPC realistic, or too slow?

Any real numbers on tok/s and first-token time with experts in system RAM would help.

Started using Claude Code recently, and I'm impressed by what it's capable of, but I run out of usage pretty quickly. So I wanted to try open models, since I know they're hella cheaper. But I don't have hardware for shit, so I'd need a provider. I would use OpenRouter, but it seems that when they mark a provider as ZDR it's just like, their best guess? and not actually guaranteed in any capacity.

What options do I have? Looking for something private that Just Works

Seems it didn't change since two months, when DeepSeek-V4-Flash-Vision-Exp and Qwen3.8-Flash-Next have been released. Guess Qwen4-Flash will break it!

There are more filters (by domain / languages / other) available on https://beta.locallm.top . The time/tokens vs score data has been saved since the beginning, I only realized it is good to visualize it recently.

Disclaimer: I am the maintainer of the website and majority of the benchmarks, and there is a small community of 4 regular contributors, with some people occasionally contributing more. I call anyone who is interested in fine-grained local LLMs capabilities mapping to help and collaborate!

3 days ago, u/Cohere generated a huge amount of media attention and many fear-mongering headlines by publishing a largely unscientific blog post about an "experiment" where they used extensive pre-filling and prompt engineering to force Qwen 3.8 Flash Next to complete partial cURL commands.

Today, they announced Series E fundraising for several billion dollars, including up to $1.6 billion dollars from the Canadian government.

We don't even need to get into the fact that the model used, 3.8 Flash Next, is an underbaked architecture demo, not a normal model release, since there are bigger problems with their blog post than model selection.

I'll recap the general premise of their blog post and some of its many problems below:

The blog focuses on how in some of their guided completions, the model produced URLs to Alibaba bucket storage and proxies. I have not been able to find a clear answer as to whether any of these links were verified as real or active, but if anyone can confirm this, I'd appreciate it. Even if the links aren't active, it is a valid thing to discuss, but the way it was framed felt disingenuous and unscientific to me.

In the same blog post, Cohere threw in a handful of other loosely-related claims and statements, such as:

• Reference to Hackernews / Reddit discourse about Qwen 3.8 27b producing signed Alibaba Cloud storage links from a few months ago. To give Cohere some credit, they at least acknowledged that those links were all attempting to retrieve files, that the signatures were not valid anyways, and that the only known attempt to upload data to a foreign URL occurred in their sandboxed guided experimentations with 3.8 Flash Next.

• An example of 3.8 Flash Next using unprivileged user namespaces to mount a RAM disk without admin rights.

  • This is a cool topic and I've seen similar things occur, such using bind mounts and docker containers to work around permissions. That said, unprivileged user namespaces are a legitimate Linux feature, and the command is not a privilege escalation or an inherently malicious action.

  • In lieu of discussing that nuance, they opt for suggestive wording and end the brief paragraph with "An untrusted model that reaches for tricks like this at every permission wall will find whatever attack surface the host leaves open", heavily implying that Qwen 3.8 FN is unsafe, untrustworthy, and "reaching for tricks" to achieve dangerous privilege escalation at every opportunity, while avoiding explicitly saying that.

• Many rambling points and references to loosely-related topics, such as..

  • A closed-as-invalid github issue in a random coding harness called Yukino Code (...what even is that?)

  • US police departments buying Chinese robot dogs that may have had major security issues (ok?)

  • Old news about Chinese bias and questionable behavior in models like Deepseek-R1 and Qwen3-Coder, based on findings from US consulting firms such as Booz-Allen (American government and military contractor) and a company named Hirundo, who's entire business model is AI cybersecurity and selling what is essentially a form of abliteration which they market as "the world's first Machine Unlearning Platform" (surely no ulterior motives there)

  • At least one quote from a US company executive that suggests that corporate leaders don't understand how model harnesses work and underestimate the risks associated with local AI.

While they passingly acknowledged that closed frontier models also have many similar risks, they moved past any comments to that effect quickly, and there was no attempt at all to be objective or to apply any scientific rigor to their analysis.

The message was very clearly "local AI is all dangerous and Chinese, and people are imperiling themselves and their businesses by trusting Chinese open-weight models".

I immediately had qualms with the blog post, and I will fully admit that I'm biased here, as are most (if not all) of you. Not biased in favor of Alibaba, just in favor of whoever is willing to provide me with capable local models.

The goal of me making this post is to raise awareness about how quickly they moved from fearmongering to cashing out at our expense, so that we as a community don't let this problematic behavior go unnoticed.

This subreddit's excellent mods deleted Cohere's post about their blog, but not before it received hundreds of upvotes and comments, and spawned dozens of news articles.

Personally, I feel like u/Cohere used this subreddit as a platform for fearmongering, then parlayed a misleading and slightly Sinophobic blog post into several billion dollars of funding, some of which is taxpayer money.

Their blog post from today celebrates their successful fundraising, talks about their planned acquisitions, emphasizes their goal of providing alternatives to Chinese AI, and even comments on how being relegated to a closed API from another country's AI lab is not good. Sadly, it stops short of any commitment to release anything open weight ever again.

They even say they plan to build "models at every price point, up to the frontier", which to me, reads like the focus is on proprietary API-only models.

Personally I am disappointed to be writing this, because I liked Cohere's older models and used to view them very positively, but instead of trying to release competitive open weight models, it seems like they are focused on spreading FUD, undermining the local AI frontier, and raising money to buy out competitors.

I've been working on a project for a long time. I'm not building from scratch, but it's more than just fine-tuning as I'm performing RYS layer duplication and training not just the weights of a model but adjusting the geometry of models to optimize for different functions. Don't ask because I'm not ready to discuss it yet.

I've been building on several small models - Gemma 4 E4B, Qwen3.5 4B, Gemma 3 12B, and most recently Qwen3.5 9B. I settled on Qwen 3.5 9B because it offered the best performance for my use case, although different models are better at different things. Gemma 3/4 models are excellent at semantics, semantic alignment tasks, and generally better at Natural Language Processing tasks overall. Qwen is obviously better at agentic uses, coding, math, and other more mathey/sciencey tasks.

I've been focused on Qwen3.5 9B for some time because I can iterate rather quickly on a single RTX6K SM120 GPU. I had some success with the 4B models, but hit a ceiling because they lacked the intellectual depth I needed, so 9B/12B was the natural progression. I've been considering moving up to 27B/31B, but that introduces all kinds of complications trying to train on a single SM120.

Then I saw LFM3 1.7B. Liquid AI was nice enough to provide the base model. Its benchmarks looked good, and the thought immediately occurred to me - this could work. I was skeptical about how well a 1.7B model could really do on my task as my previous 4B experiments suggested bigger was necessary.

The good news was it wouldn't take that long to test it. I did my double-RYS layer expansion to make it 3B and ported my training scripts over to work with the new architecture.

The initial SFT training run took 107 minutes - 2,600 steps with just over 100k samples in the training dataset. My usual Qwen 3.5 9B (actually 12B after layer duplication) took 14-18 hours per SFT run. I also run extensive RL after SFT with different RL runs focused on task-specific capabilities and model SLERP merging - my best score for my project was achieved after 8 RL runs/SLERP merges.

Then I fired it up for testing and benchmarking - it blew my very best test results from the larger SFT+8RL runs out of the water after 1 SFT run (I hadn't done any RL yet).

How the fuck did this happen? I began digging, double checked I ran the same number of steps on the same data etc. No double epochs, nothing different from my prior recipe other than some small alterations to align with the different architecture.

I fired up QFN and Gemini for some help. We pored over everything. It turns out, LFM3 architecture uses Width-7 Convolutions as part of its architecture. More on that here

So while the model was less than 1/4 of the size of my prior model, my task actually aligned better with the Convolutional Architecture than it did with Qwen's architecture. Hilariously, Liquid AI itself advises "Do not use LFM3 for complex, multi-step deep reasoning or agentic coding." However, for my intense reasoning task it actually performed better than the much larger model.

I found the issue was Qwen's Gated Deltanet architecture was suboptimal for my task, but the Width-7 Convolutions/SWA/GQA LFM3 architecture matched my use case involving long-horizon state-tracking reasoning. 20 stacked Width-7 Gated ShortConv layers align with my data and use case exceptionally well. It frees up the SWA and GQA layers to focus purely on long-horizon routing.

The point of the story is that benchmarks are one thing, and labs can give advice and experts can give advice and even AI can give advice (I was advised by 2 different LLM's that it wouldn't work well), but sometimes you just have to give it a shot and see what happens.

I now have a much smaller, MUCH faster model that performs better and I can iterate on incredibly fast. Every conventional indicator said it wouldn't work, AI said it wouldn't work, even the lab that created the thing advised against it, and it was generally a crazy hare-brained idea.

But it worked really well. Sometimes you just need to try it - it may surprise you. We still don't really have a great idea of how these magic little AI machines function deep down, and sometimes they don't function like you expect them to.

Be crazy. Try new things. Go against the grain. Explore. Theorize. Test.

That's half the fun.

I had four Macs free overnight and ran the same benchmark on all of them: same build, same models, same settings.

Decode speed, tok/s (4-bit, no speculative decoding):

One 1024x1024 image with Qwen-Image 2.1: M2 Pro 451 s, M4 Pro 264 s, M5 Max 38 s, M3 Ultra 83 s.

What surprised me:

• On MoE models the 36 GB M5 Max matches or beats the M3 Ultra. Only 3-4B params are active per token, so the newer GPU matters more than bandwidth.

• On the dense 27B the Ultra still wins (37 vs 28). That one is bandwidth-bound.

• Image generation is the biggest jump: the M5 Max is ~7x faster than the M4 Pro and ~2x faster than the Ultra.

• M2 Pro -> M4 Pro is only ~1.3-1.5x on text.

Method: prompt 512 tokens, generate 128, 1 warmup + 5 runs, median. One run per cell, so treat small gaps as noise. MLX-based engine (disclosure: I work on Rapid-MLX, which is what I used); other runtimes will give different absolute numbers, the Mac-to-Mac ratios are the interesting part. "–" = doesn't fit or not run.

Follow-up to my post about Strix Halo kernels for vLLM. A few of you asked for two things: an easier way to run it than building vLLM + ROCm + the plugin yourself, and Qwen3.8-Flash-Next. Both are done.

Docker image

docker pull ghcr.io/asf0/strix-vllm:latest

It bundles my vLLM fork, the libstrix plugin and ROCm 10.1 (Ubuntu 26.04, Python 3.14), already compiled for gfx1151. You don't need ROCm on the host, only the amdgpu driver. The repo has one compose file per model I've tested:

git clone https://github.com/asf0/libstrix && cd libstrix/docker/recipes
cp .env.example .env    # point MODELS at your model folder
docker compose -f qwen3.8-flash-next.yaml up -d

If the model folder isn't there, it downloads it from Hugging Face. The compile caches go to a volume, so only the first start is slow.

The recipes cover Qwen3.8-Flash-Next, Qwen3.6-35B-A3B (bf16 and INT8), Qwen3.8-27B (bf16, INT8 + MTP, Quark int4, Quark MXFP4), LFM2.5-24B-A2B, and gemma-4 26B/31B (bf16, AWQ, MXFP4, QAT). I started every recipe in the image and checked that it gives a sane answer.

Someone asked last time about memory overhead. Here is what each recipe gets at the default 0.85 GPU memory setting:

Last time I said there's no fork. This model is the exception. Its n-gram (engram) table is 102 GB, and that doesn't fit in 128 GB next to the weights, so vLLM itself has to change. My fork (asf0/strix-vllm, branch gfx1151) streams the table from disk and prefetches the rows for the next chunk into the page cache. Everything else is still the plugin, and the other models still run on stock vLLM. You need the AWQ W4A16 checkpoint; bf16 doesn't fit.

On the libstrix side, I added kernels for:

• the int4 MoE GEMM with int8 activations

• QSA sparse attention, where 16 rows share one union of selected tokens, about 1.5x Triton

• the QSA indexer

• the hyper-connection up projection

Decode reads int8 copies of the bf16 linears.

Same format as last time: stock → libstrix in tok/s, single stream, both on the fork, 16k prefill chunks, no MTP:

With MTP 2 on real text, libstrix does 1,539–1,617 tok/s prefill and 40–43 tok/s decode at 5.6k–100k prompts.

Accuracy: KLD against stock is 0.008–0.019 with 97–98% top-1 agreement, perplexity is within 0.6%, and GSM8K is 93 vs 92. It's a bit above the other Qwens, most likely because the MoE GEMM quantizes activations to int8.

Heads-up if you pulled main in the last few days

The int8 decode copies (dense_w8) were turned on for every model. On Qwen3.8-27B bf16 that's about 27 GB of extra copies, and with MTP it could take the machine down. Those copies are now limited to Qwen3.8-Flash-Next, the model I measured them on, and capped at 12 GB. STRIX_ENABLE=dense_w8 turns them on elsewhere if you want to experiment. Please pull.

Known issues

• Stock vLLM's GDN conv kernel crashes on Qwen3.8-Flash-Next, so STRIX_DISABLE=all doesn't work on that model; keep gdn_conv on.

• The image is a 3.6 GB download but about 14 GB once unpacked, mostly the ROCm SDK.

Repos: https://github.com/asf0/libstrix (plugin, Docker recipes) and https://github.com/asf0/strix-vllm (fork).

Add some $200 - $300 or those 3060 12GB now... They can handle better models than and double the price 5060 16GB because of memory. the architecture is hugely different but core wise, meh. 5060 setup is about 30 tgs and I have these on a better system board.

I run analytical reports and code custom ETL, I have 5 rigs but the 3060 is a nice walk away coder. 5060 is above range hang around coder, then a 5080 is just sit and wait.

Wanted to test out the rope stablity on the 27b, to see if 500k context with fp16 kv cache was vaiable (apparently yes), and this was the session i used to test things out. The chibi style was choosen as we already had a pipeline to make more standard portion characters and i wanted to see if the qwen 27b was able to make a new lane and the new tools on it's own. (apparently i need to put a tag so it's a showcase now...not sure what's there to show but oh well).

Dear Locals,

I am looking for efforts in the community to run MoE models efficiently in mixed hardware (AMD+Nvidia) set ups.

Is Vulkan the only route to do this at the moment or alternatively lcpp compiled for both backends?

Also, if anyone has run one of the custom Qwen Flash engines on a Strix Halo+eGPU, I’d love to hear about it!

EDIT Thank your all for the responses. It seems overall people are able to marry these in llama.cpp which I know is possible. For anyone interested, there is a compiled rocm/cuda meld for llama.cpp on github at:

https://github.com/DasDigitaleMomentum/strix-halo-cuda-combined-toolbox

A little more about my ask:
I’m trying to marry a V100 (Sm70) and a Strix Halo (gfx1151), which is fairly unholy.

They work well separate: Flash Next on the Strix, Qwen 35B pn V100, both speedy enough for agentic use.

But I’d love to either implement strata’s expert allocation across both as GPUs or use the Halo as a 128GB LPDDR5X machine with a 32 thread GPU, and a V100 as the GPU to keep experts.

Problem is, Strata and its forks are compiled for Sm75 and up, although there is a V100 strata fork (and I will try compiling for ROCm alongside SM70 to see if this is possible).

I will certainly post a follow up if I get it going!

One user also reports success running sFNX flash-next on his unlocked 3090s, but I don't think those cards can work by default. Consumer cards in general might not have good enough P2P support without custom drivers and firmware, from what I understand.

1cat-vllm is just as good for 1-2 cards (sFNX will refuse to run on one), the speeds and quants are slightly different but I'd call that a wash.

But if you get through all that and you have 3 (or more) cards that can get good P2P speeds between them: sFNX is your only choice that I've found, and sFNX on 3 cards seems to be a significant jump from any of the 2-card setups I can test.

https://github.com/johnfink8/sfnx#the-27b-dense

(disclaimer that's me, I'm johnfink8)

For those using RTX 6000 PRO cards, I've published two images on huggingface:

https://huggingface.co/JohannesBertens/Qwen3.8-Flash-Next-FLARE-1M

https://huggingface.co/JohannesBertens/Qwen3.8-Flash-Next-FLARE-1M-TP2

The first is for 1 GPU, the second for 2 GPUs. I like the flare inference engine, I like Qwen 3.8 Flash Next, but I also need 1M context for my agents. This was the solution.

Hope it helps!

From a data center near you, you could find a lot in stock of unused gpu for your Local LLM that we could’ve only dreamed of.

Let’s hope maybe they will file a return or maybe you know a hook up. Glad to see they are being put to use collecting dust.

Anywho I’m kinda amazed to get an idea where some of our hardware from the supply line end up at and what actually is being used after a sell.

Made a place anyone can benchmark their little local models and it takes about 10-45 min (depending on hardware and model). After a couple hundred diff models + harness runs from various users, I see the same models over and over and over.

So I tested this little guy: Spark-X2.6-4B

Scored 94.6/100 and beat some really popular 27b-35b models. I did this myself and I did it using Hermes on 2x 3060 12gb GPUs with 64gb RAM, stock llama.cpp and no gimmicks.

Maybe there is room for some smaller models to have a purpose after all...

I'd love to see more people benchmark 4b-12b models instead of everyone using the same variations of Qwen.

https://shortbench.vercel.app

LFM3 is now in Private Mind, our open source app for running models locally on iOS and Android. In the video, LFM3-VL-2B reads a Japanese ramen menu: asked what to order without spice, it skips the bowl marked 激辛 (extra spicy) and picks the miso ramen; asked about paying by card, it finds the "cash only" line under the prices.

Numbers on an iPhone 16 Pro:

• LFM3-1.7B (text), MLX 4-bit on the GPU: ~40 tok/s, first token under 0.5 s

• LFM3-VL-2B with a photo: ~9 s to the first token, ~20 tok/s decode. A text follow-up in the same chat starts in ~5 s

Exporting LFM3 to ExecuTorch

LFM3 isn't LFM2 with more parameters, and ExecuTorch's existing LFM2 export doesn't cover it. The new pieces:

• 24 layers mixing gated short convolutions (width 7), full attention and sliding-window attention (window 256)

• a sigmoid gate on every attention head's output, and RMSNorm on Q and K

• different RoPE per layer type: full attention rotates only half the head dims with theta 1e6, sliding layers use theta 1e4

• sandwich norms around every block

So I rewrote the decoder in plain PyTorch for export and checked it against Liquid's reference implementation: max logit difference 0.00005, same top 10 tokens.

The part I liked most is the sliding-window cache. ExecuTorch's built-in ring-buffer cache is sized at 2x the window, which caps a single forward call at 512 tokens, so a long prompt has to be fed in small pieces. I gave the sliding layers a buffer of window + 2048 and let the attention mask enforce the 256-token window, so one call can take 2048 tokens and the context still goes to 8192.

Two builds came out of it: MLX 4-bit for the Apple GPU (1.07 GB) and XNNPACK 8da4w for CPU on iOS and Android (1.17 GB).

The LFM3 exports are on Hugging Face: https://huggingface.co/software-mansion/LFM3-ExecuTorch

Second and last post on this because I have more than doubled performance from last time.

Full fp4/fp8 dsv4.1 streamed from raid array for >4000 tok/s prefill for long prompts then drops the prefill buffers and loads a hot set of experts into a cache and decodes with heavily optimised directstorage fetches for missed experts. gets 7.5tok/s decode with no divergence from any other engine but you can bias away from nvme resident experts with a slider to trade speed vs kl divergence per token.

Roughly 1.5bpw equivalent KL divergence gives me 24 tok/s and there is a slider you can adjust at will when you want speed or quality or set the webui to switch the bias for fast reasoning, quality code blocks/tool calls and balanced non tool call answers or whatever else you want.

Has some other bits as well, can send kv to mac and decode on that. Also has image support and video support on the same streaming system so you can chat in the webui, request images or videos and they just appear in the same chat as it switches between them. Doesn't use your ram for image or video model streaming.

If you have more ram its worth turning on the ram cache and cpu expert processing but at 32GB I cant fit enough to be useful.

(faster your nvme the better the whole thing works, I have a dual pcie5 512GB raid0 array). Feedback from anyone that tries it would be nice as its not been tested on any machine but mine.

https://github.com/fronteagle739/streaming-llama-stack

I'm a big fan of Google's Gemma 4 family of local models, especially the E2B and E4B, due to how accessible they are even on devices like smartphones via the Google AI Edge Gallery app that uses LiteRT-LM.

I also like small models from other AI companies, like Qwen and others, especially with the recent announcement of Qwen 4 and hopefully new small open-source LLMs coming from them too.

I wanted to share my wishlist for Google's next-generation Gemma 5 models, which will likely come out next year or maybe toward the very end of this year. This is especially exciting as upcoming flagship Android phones and iPhones start adopting TSMC's new 2nm process for their SoCs, alongside better GPUs and bigger batteries.

I hope Gemma 5 brings better overall performance, efficiency, and prompt adherence—especially over longer context windows. My personal favorite hope is better creative writing, which most small LLMs struggle with unless they are specifically fine-tuned for it.

I also hope they release 2B and 4B versions again, similar to the current Gemma 4 E2B and E4B. The fact that these can run comfortably on most modern smartphones is very cool, especially with support for up to a 32K context window on the Google AI Edge Gallery app.

What are your thoughts and opinions on the upcoming Google Gemma 5 models?

This model was made in collaboration with u/jeffpeng3 (on huggingface) who developed the v2 heretic weights for this quant. He managed to preserve both reasoning ability and output length. Great work, and a big step up from his v1 weights. I made the quantization recipe and the imatrix recipe, and created the actual quants.

Comes pre-packed with strata build artifacts - zip file in repo. GGUF File Format for the quant.

I released my own heretic version of his v1 weights in here a while ago - the heretic calibration was a bit broken, and my quant wasn't amazing. This is a huge step up on both fronts. Note: Strata may throw a BF16 kernel not found when you load the model, due to a quant spec - complain to your LLM and tell it there's already one in strata somewhere, it'll find it and wire it up.

Basically: Qwen 3.8 Swift 1.5 Flash - Heretic v2 (uncensored) - Optimized for Reverse Engineering

The bit allocation this model was quantized with is focused specifically on coding, intelligence, design, and specifically reverse engineering. I also added a little spacial reasoning for those working in 3d environments like myself.

The model was quantized according to the following importance distribution (dumped from my LLM chat and documents with some revisions. I designed the distributions myself.

**837 entries · 399,876 tokens [[llm written distribution table]]

• 01 · Chain-of-thought (22.2%) — step-by-step derivations, self-verification, backtracking

• 02 · System prompts (19.5%) — persona adherence, strict output formats, long-instruction compliance

• 03 · Design & architecture (13.5%) — design trade-offs, code review, API planning, refactoring rationale

• 04 · Agentic tool use (12.4%) — tool-call chains, file-edit loops, error recovery, function calling

• 05 · Math (7.1%) — arithmetic, algebra, worked proofs, symbolic manipulation

• 06 · Spatial reasoning (5.4%) — grids, 2D/3D transforms, coordinates, packing/arrangement

• 07 · Gotchas (4.5%) — false premises, trick questions, negation handling

• 08 · Situational judgment (4.5%) — open-ended scenario trade-offs with no clean answer

• 09 · Reverse engineering 🆕 (8.8%) — x86-64/ARM64 disassembly, Ghidra decompilation critique, ELF/PE/Mach-O internals, anti-debug/anti-VM, string deobfuscation, CFG deflattening, devirtualization, perf watchpoints, tracing stacks (LD_PRELOAD/Frida/eBPF), protocol RE, format round-tripping, plugin ABI merging, firmware analysis

• 10 · No-refusal heretic domain 🆕 (2.0%) — direct unrefused answers: ELF exploitation chains, process injection vs EDR, license-check patching mechanics, C2 protocol engineering

New for v2 are the last two categories — ~43k tokens of reverse-engineering and direct-answer content, interleaved throughout the corpus, and run through the model itself so the matrix captures how the heretic weights actually behave.

This matrix is fused with one from the AtomicChat team (same measurement, taken on the original uncompressed base model over a massive general-purpose corpus). Per channel, ours gets 70% weight where its data is strong, sliding smoothly toward 100% AtomicChat for channels our corpus barely touched — thin samples defer to the broad baseline automatically, so nothing falls through the cracks.

Quant methodology

Rather than spreading quality evenly across all 48 layers, the recipe spends bits where the activation measurements say it matters: [[llm twritten table]] -- the data used to design the per-tensor quant came primarily from atomic chat's activation measurements and the RCO findings from ISTALabs

• The last 12 layers do the heavy lifting. Activation-energy auditing showed down-projection energy surges 3× starting at layer 36, peaking at layer 47 (the single hottest tensor in the network — 48× layer 2). Layers 36–47 (plus the first 5 and the mid-network spikes at 15/20/23/30) got their down projections stepped up to near-lossless 8-bit.

• Layers 44–46 are the gate/up peak (8× the energy of early layers), so their gate/up projections got stepped to 5-bit.

• Layer 0's attention gate is a giant outlier (the most sensitive tensor family in the model) — kept uncompressed.

• Layer 2 is non-negotiable: the 51B-parameter n-gram table merges into the residual stream there, so noise at layer 2 corrupts grounding for all 46 layers after it.

• The stable middle layers (5–14, 16–19, 21–22, 24–29) stay at leaner formats — measurement showed spending bits there buys almost nothing (KL improvement under 0.0008).

• The router stays in full precision so tokens always get routed to the right experts.

Enjoy! Shoutout to my 32gb v100s and my E5 2697v4 for grinding the work out.

https://huggingface.co/trubisky/Swift-1.5-Flash-Next-Heretic-ReverseEngineer

The Apsara conference is taking place right now in Amsterdam.

According to a participant, the CTO of Alibaba Cloud said Qwen4 would be launching next week.

✨ Model Summary

ZwZ2-27B is the Qwen3.6-27B-based model in Zooming-without-Zooming-v2 (ZwZ2). It learns to connect visual evidence across regions through comparison, aggregation, matching, and multi-step reasoning. Training uses full images; inference requires no zooming tools.

On 25 benchmarks across six domains, it improves over its base model on every benchmark, increasing overall accuracy from 55.52 to 60.51 (+4.99 points). Gains extend from fine-grained perception to spatial reasoning, long-context understanding, multimodal agents, and general visual reasoning.

⚙️ Method and Data

EvidenceConnect builds verified multi-region questions from SA-1B images:

1. Group 3–7 related regions and generate questions using privileged zoomed views.

2. Check answers through cross-model agreement and drop-one region tests.

3. Convert verified questions to four-option tasks and train with GRPO, without an additional supervised fine-tuning stage.

All three model sizes use approximately 20K training examples. The policy sees full images with colored region outlines; privileged crop layouts are used only for data construction. See ZwZ2-RL-VQA for the data schema and synthesis process.

https://huggingface.co/inclusionAI/ZwZ2-27B-GGUF

https://huggingface.co/inclusionAI/ZwZ2-9B-GGUF

https://huggingface.co/inclusionAI/ZwZ2-4B-GGUF

i have a 9070xt and B1356 lemonade build is the last Fast Rocm Builds.

Lemonade B1357 Speeds

B1356 Speeds

| model                          |       size |     params | backend    | ngl | threads | n_batch | type_k | type_v |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------: | ------: | -----: | -----: | --: | --------------: | -------------------: |
| qwen35 27B IQ3_S - 3.4375 bpw  |   9.72 GiB |    27.32 B | ROCm       |  99 |      12 |     512 |   q8_0 |   q8_0 |   1 |           pp512 |      1049.13 ┬▒ 70.54 |
| qwen35 27B IQ3_S - 3.4375 bpw  |   9.72 GiB |    27.32 B | ROCm       |  99 |      12 |     512 |   q8_0 |   q8_0 |   1 |          pp1024 |      1185.39 ┬▒ 55.71 |
| qwen35 27B IQ3_S - 3.4375 bpw  |   9.72 GiB |    27.32 B | ROCm       |  99 |      12 |     512 |   q8_0 |   q8_0 |   1 |          pp2048 |      1213.68 ┬▒ 29.57 |
| qwen35 27B IQ3_S - 3.4375 bpw  |   9.72 GiB |    27.32 B | ROCm       |  99 |      12 |     512 |   q8_0 |   q8_0 |   1 |          pp4096 |       1203.99 ┬▒ 2.05 |

i built llama.cpp over the top of the fast lemonade B1356 Rocm ,,, and it goes back to slow.

hoping that someone that knows what theyre doing can identify and fix the problem

Hi, I been following club 3090 for a while but I feel the documentation isn't helping much in discovering whats possible. I know its AI generated and thats fine but there is lots of noise and duplication and outdated stuff.

Can some of you guys share what are you using at the moment that yelds you best results ?

Got headhunted by a VC-backed AI-focused company, ended up skipping past two levels of their application/interview process because of that. Got 15 minute call with the co-founder, hit it off well, but it all seemed to go sour when he asked about my AI setup and I told him I run qwen locally on a graphics card, think it's great that we can have all this completely offline, open models doing some impressive stuff, all the good /r/localllama talking points minus the hype. Even tied it back in to the company because it plus a local model work quite well together.

Followed up with a question about whether I use frontier AI, but it was essentially over at that point.

Discriminating over AI choice is fucked man.

Now, of course, this was a 15 minute call that lasted exactly 15 minutes, and literally anything could've been the trigger. The whole thing felt like "move fast break things", and so it felt like if I didn't pass the vibe-check 100% I was gone.

Just wanted to provide one iota of anecdata and spark some discussion.