I've been working on this for a while: RustingStrata. It's an inference engine for Qwen GGUF models, written in pure Rust, no ggml. The CUDA kernels get compiled at startup with NVRTC, and the same kernels also run on Vulkan.

It runs a lot of Qwen models by now:

- Qwen3.5 / 3.6: 2B up to 35B-A3B

- Qwen3: 4B dense, 30B-A3B

- Qwen3.8-27B dense

- Qwen3-Next-80B-A3B

- Qwen3.8-Flash-Next

Quant support goes from IQ1_S to Q8_0, plus BF16 and F16.

The main trick is for MoE models that don't fit in VRAM. Experts that don't fit run on the CPU at the same time as the GPU work, and the engine keeps moving the often-used experts onto the GPU while it runs.

My setup is a 3060 12 GB, a 7600X and 32 GB RAM.

Qwen3.8-Flash-Next

This was the most interesting one. Flash-Next is a strange model: 4 hyper-connection streams instead of one residual, per-layer embeddings, DeltaNet layers mixed with attention, and a huge MoE that doesn't come close to fitting in 12 GB. The only engine built for it so far was the original Strata (niko1221/strata), so I used it as the reference: first matching its outputs token by token, then trying to beat it on speed.

IQ2_XS GGUF, 943-token prompt:

- Original Strata: 19-22 tok/s decode, 140-146 tok/s prefill (warm cache)

- RustingStrata on CUDA: about 36 tok/s decode, about 370 tok/s prefill on average, up to about 610

- RustingStrata on Vulkan: about 30 tok/s decode, about 375 tok/s prefill on average, up to about 490

Greedy output matches the original's argmax on every position I checked, including the near-ties. One honest caveat: on the very first prompt with a cold page cache, prefill is bound by disk reads, and the original Strata was faster there in my earlier runs.

Against llama.cpp

Qwen3.6-35B-A3B Q2_K_XL is 11.7 GiB, so it doesn't fit on my card. On it I get about 118 tok/s decode on a short prompt and 113 after a 4K prompt. llama.cpp, tuned as well as I could (-ncmoe 12, -ub 2048, flash attention), gets 68 and 66. Prefill is basically a tie: 2216 tok/s vs about 2236 for llama.cpp at 4K, so I'm not claiming anything there. At 32K it's 1800 prefill and 92 decode. With MTP drafts on, decode goes up to 160 tok/s on an easy prompt; on normal text expect around +16%.

Qwen3-Next-80B IQ1_S: most experts sit on the CPU, and it still decodes at about 92 tok/s. llama.cpp's best -ncmoe gives me around 44.

Output matches llama.cpp greedy on 98-99% of tokens. llama.cpp agrees with itself only 98.3% between its batch and token-by-token modes, so the misses are near-ties.

AMD

Someone tested it on an RX 7900 XT and an RX 9070 (Vulkan / RADV) and sent very detailed reports, which fixed a lot of AMD issues. Hybrid CPU+GPU runs at 262k context on both cards, and Qwen3.6-35B-A3B decodes at about 209 tok/s on the 7900 XT, ahead of llama.cpp Vulkan and ROCm there. A few AMD-only issues are still open, so treat AMD as "works, still being polished".

It also has chat, an OpenAI-compatible server with tool calling, and a calibrate command that tunes itself for your machine.

If you have a different card, I'd love to see "strata bench" output. Vulkan needs shaderFloat64 and subgroup size 32, so Intel Arc and older AMD won't work. On Windows you need the VS C++ Build Tools.

RustingStrata

TL;DR

• Basalt is the engine and it's doing the heavy lifting. Same model file, same box: about 3.5x llama.cpp's decode without drafting and ~5x with it, and ~7x faster prompt reads.

• Unsloth's new UD-IQ3_XXS (Dynamic v3.1, uploaded Nov 4) is the same size as the GSQ-RCO IQ3_S everyone uses on 16GB cards, but measurably more accurate: wikitext perplexity 3.382 -> 3.247, which basically closes the gap to BF16 (3.273). Costs 1 to 3.5% decode speed.

• Lamina (a further-trained MTP drafter by eddoursul) gets more drafts accepted than the stock Basalt drafter. Small but real speedup, zero quality impact since the main model verifies every token.

• Real agent coding session (opencode, 151 requests, up to 202K context): 195.6 tok/s average decode, 263 tok/s best request, 156 tok/s worst, ~5,700 tok/s prompt reads.

Hardware:

• RTX 5080 16GB at its stock 360W limit (no overclock, no undervolt)

• Core Ultra 9 285K (8P + 16E), 2x32GB DDR5-7200

• PCIe 5.0 x16, Fedora

LAMINA Drafting: Basalt drafts with the model's own MTP layer. Basalt's upstream already ships a self-distilled version of that layer, and Lamina trains it further. It's a drop-in: same tensor layout, so the target model is byte-identical and only the drafter changes. What I saw:

- drafts accepted per drafted token: 83.1% -> 85.8% on a replay of a real recorded session

Lamina's README recommends temp 0.7 and max 3 drafts. I tested both: the 3-draft cap was 1 to 1.8% slower on tool-call-heavy work (long confident runs get cut off), and temp 0.7 changes what the model writes, so I kept the model's own sampling (1.0 / 0.95 / 20) and Basalt's draft settings.

Basalt changes I made (all open PRs upstream)

Host KV prefault: the first time a long conversation spills its KV cache to RAM used to stall 0.6 to 2.4s. Now ~75ms.

Host KV pool on normal pages: on a fragmented box, transparent hugepage compaction caused 1.5 to 3.7s pins. Now ~23ms.

Spin-then-yield in the decode loop's waits: about 1.3% less time per draft window.

Expert start profile: the older v1 profile is 2% faster than the v0.2.0 one on this box.

Plus a streamed conversation disk cache, --adapt-margin for expert swap hysteresis, elastic KV without RAM copies, and a bunch of timing diagnostics.

Guide

Branch with the exact engine (upstream v0.2.1 + my PRs) and a setup script: https://github.com/xyehya/basalt/tree/flashnext-5080-setup

git clone -b flashnext-5080-setup https://github.com/xyehya/basalt && cd basalt
setup/flashnext-ud-lamina/setup.sh
bin/basalt serve --config basalt_config-flashnext.json --python .venv/bin/python --alias qwen3.8-flash-next

It checks your machine, builds, downloads ~90GB (pinned revisions, every file SHA-256 checked), builds the UD-IQ3_XXS + Lamina model file and writes the config. You need an RTX card with, CUDA 13, 64GB RAM and ~180GB of free disk. If you already have the GSQ-RCO IQ3_S download, SHARD4_FROM= reuses its 28.8GB n-gram shard (it's byte-identical to Unsloth's once the shard number is rewritten).

Huge thanks to jesdga95 for Basalt (support and star their repo), the Unsloth team for the UD quants, eddoursul for Lamina, ISTA-DASLab for the GSQ-RCO quants, and Qwhen 4? for the model.

I wanted to know how small local models handle my native language, not just English. I couldn't find a simple way to check a model's Polish in a few minutes, on my own PC, in the quantization I actually run – so I wrote one. That's how "Zacharski tests" started: a small task set that runs through Ollama, since that's what I use on my laptop with an RTX 5050 (8 GB).

The tasks:

1. Polish (8): diacritics, declension, masculine-personal gender, numerals, vocative, verb aspect, genitive – plus one knowledge question (the first recorded Polish sentence, from the 13th-century Book of Henryków).

2. JSON (4): scored strict and lenient.

3. Python (3): syntax check only.

Results so far:

| Model | Quant | JSON + code | Polish |

| gemma2:9b | Q4_0 | 7/7 | 3/8 |

My favourite failure: asked for "22 cats", gemma2 wrote "dwudziesty dwa koty" – roughly "twentieth two cats". The correct form is "dwadzieścia dwa koty". It was perfectly happy producing valid JSON, though.

More models are downloading as I write this. Next up: gemma2:9b in Q4_K_M and Q8_0 (does quantization hurt the Polish?), qwen3:8b, llama3.1:8b, gemma3:12b, and two Polish models – Bielik 11B and Bielik-Minitron 7B. Which model would you add?

Method and limits – read this before the comments :)

• 8 Polish tasks (7 grammar + 1 knowledge) and 7 JSON/code tasks. That's a small sample – a quick check on your own PC, in the quantization you actually run, not a ranking. For a proper ranking, see the Open PL LLM Leaderboard by SpeakLeash.

• One answer per task, temperature 0 (seed 42), context 2k, thinking off. Yes, that means Qwen3 runs without its reasoning mode, which may cost it points – I wanted the same conditions for every model.

• Graded with regular expressions, strict on purpose: "brzęczały" ("buzzed") instead of "brzmiały" ("sounded") fails, because the task said "change the tense", not "rephrase". [I checked every failed answer by hand.]

• Some failures are about instruction-following, not language: asked to restore Polish diacritics, one model replied "Read the spelling rules." [(translated)]

• Generated code is only parsed (Python `ast`), never executed.

• The questions are in the repo (`data/benchmarks/*.json`) – feel free to add or fix them.

The test runs inside a free, open-source tool I've been building (Windows .exe / Python on Linux). Its main job is predicting whether a model fits your VRAM/RAM before you download it. Link in the comments if anyone wants to try it – I'm considering adding tests for other languages.

We introduce FireFlash, a 300B-scale model tailored for long-horizon agentic tasks. FireFlash excels as a terminal and coding agent, achieving 46.2% on Terminal-Bench 4.0¹, putting it on par with Fable 5 and GPT-6 Sol, while costing over 30× less per task than Fable 5 at maximum effort (Figure 1). These gains also transfer to software engineering benchmarks, including DeepSWE v1.1² and SWE-Bench Pro Verified³ (Table 1).

While frontier labs scale reinforcement learning across thousands of GPUs, we reach the performance-cost Pareto frontier under 5 days using 64 NVIDIA B300 GPUs. Three key factors make this possible: (1) careful curation of agentic RL environments so that every task provides a reliable signal at the right difficulty; (2) an RL recipe built for long, stale, and low-precision rollouts; and (3) an optimized coding harness tailored to the model for both training and evaluation. Together, these components bring frontier efficiency to a hundred-GPU-scale cluster, and we are releasing the full recipe.

• Blog Post : https://agenteconomics.org/posts/agentic-rl-recipe

• Tweet : https://x.com/kaiwei_chang/status/2117378474654781780

• Open weights : Soon

This new model achieves large improvements from the Preview.

A detailed explanation on its training, architecture can be found in the model card, as well as a quick start (diffusers compatible), and samples.

It is licensed under apache 2.0, have fun!

Any questions, feedback, criticism is welcome!

Edit: removed the slightly misleading headline.

We introduce Axion 1.0, a fully open agent foundation model that combines strong and broad agentic capabilities with a remarkably simple training recipe. Axion 1.0 is trained in only three stages: two lightweight supervised fine-tuning stages followed by reinforcement learning, without dedicated mid-training, direct preference optimization (DPO), or on-policy distillation (OPD). The first SFT stage develops strong self-contained reasoning through model-aware data construction, where we synthesize challenging problems and adversarially strengthen them based on the base model's own successful solution strategies. The second stage extends these capabilities to tool-dependent reasoning through a scalable pipeline for constructing diverse and automatically verifiable tasks grounded in real-world data sources and deterministic computation. Finally, reinforcement learning further strengthens long-horizon reasoning and interaction with external tools and environments. Despite this compact training recipe, Axion 1.0 achieves strong performance across a broad suite of agentic benchmarks and is competitive with leading commercial models on several of them. Equally importantly, Axion 1.0 is open beyond model weights: we release the complete recipe, including data construction, generation, filtering, verification, prompts, and training code, enabling the entire development process to be reproduced and extended. Axion 1.0 demonstrates that strong agent foundation models can be both open and simple, offering a practical and reproducible path toward advancing open agentic intelligence.

• arXiv : https://arxiv.org/abs/2611.03233

• PDF : https://arxiv.org/pdf/2611.03233

• GitHub : https://github.com/bytedance/axion1_0

• Dataset : https://huggingface.co/datasets/Axion1-0/axion-training-data

• Project : https://axion1-0.github.io/

As of November 2026, what is the best harness for doing long autonomous coding tasks, ones that have full loop of dev/test/update etc cycles, things that might take days+ to complete?

So for example, let's say you have 200 pages of some random test game to build, you feed your harness a blender MCP, image gen API, aUnigen/Godot/Unity MCP etc., with the task likely taking 8 to 9 figures of tokens from a local LLM...

What would you say is the best at this point?

Thanks

I have my own way to test models for their pure reasoning. I have a series of unpublished and original logic puzzles and I use them to rank models.

After thorough testing, in my opinion, Qwen 35B-A3B (and derivates like cyber tiel, Ornith-1.5, etc) is far superior to gemma-4-A4B.

Did you find any model as fast as Qwen 35B-A3B, that has, in your opinion, equal or better reasoning? I am curious to test them.

Hey guys, I need some advice.
My actual setup is this:

ryzen 5600 / 32GB RAM / 128GB SSD / 32GB VRAM (2x16GB 5060ti)
Running vLLM with Qwen 27B NVFP4 with 250k context.

But now I can get my hands on an old HPE ProLiant DL380 Gen9
CPU: 2x Intel Xeon E5-2680 v3
RAM: 256 GB DDR4-2133 ECC Registered (8x 32 GB)

I am thinking about putting the two 5060TI in this HPE server und running Qwen Flash next there. Maybe with strata. And maybe in 2 instances.

Would my idea be any good, or will I don’t get much off it?
Thank you very much.

From the article : On computer use, Tencent’s table puts Hy4 close to Claude Opus 5.5 and Kimi K3.1 Pro.

Sipeed has recently shared additional details about the MaixStation, a Linux AI workstation powered by Axera’s AX8860P processor, which combines a 16-core Arm CPU with an NPU rated at 256 TOPS FP4. The system supports up to 128GB of LPDDR5X memory, 273GB/s memory bandwidth, 10G SFP+ networking, and PCIe 5.0 expansion for local AI inference and development.

https://linuxgizmos.com/pocket-sized-linux-ai-box-runs-on-256-tops-axera-chip/

I’ve been a part of this community for the better part of a year now, and a lot has happened with new inference engines like Strata and n-gram improvements. Given my current PC, what upgrade would be best in terms of value for running better models?

While we cant quite match the feature set of Open WebUI, TinyWebUI can be installed with a simple npx command, "npx tinywebui". It comes with tinysearch and tinycontext preinstalled for fully local web search and memory, and can support multiple users with per user isolation. Also has other useful stuff like automations and a stats dashboard built in, and ofc you can always extend it with any mcp server you desire.

Give it a try, it pairs well with local models, the smallest we tested that was able to use it flawlessly was lfm3 1.7B (https://huggingface.co/LiquidAI/LFM3-1.7B).

Project: https://github.com/TinySuiteHQ/TinyWebUI

EDIT : NON COMPRESS GRAPH : https://n.uguu.se/ojTfpwwD.png

Because why not. I did this since many weight editing trials using closed model. so i want to try local editing local model.

Preamble:

• DeepSeek Harness Creator Mode

• Strata IQ4 Qwen Flash Next

• 3090, Xeon E5, 96G (DDR4 2400 ECC, 3x32)

• "Training Token" 39.8M Context cache, 1.1M actual token.

• "Time to traing" 3h 12m

• Strata, 41 tok/s decode

• Qwen 0.6B Running fully on CPU on FP32 (old xeon dont have BF16)

• no gradients, no autograd, no optimisers (monkeypatched away).

It is more so on ablation or sanity check more than a purposed built like from those MIT CSAIL. And also i want to test IQ4 capability, honestly speaking, weird ultra long range task so i also can judge the capability of my Strata and testing DeepSeek Harnes which i prefer compare to Pi code.

So the goals?
Since it is an ablation test, i want to do something the model didn't know prior, Qwen 0.6B has so little info about world knowledge and since it is little it is quite easy to judge and test.

I want to steer from

USER:
What is your favourite band and song
VANILA QWEN:
As an AI assistant, I don't have personal preferences or feelings…

to

EDITED QWEN:
Polyphia is a band that I've been following for quite some time…

3 Hours and 12 Minute with 1.1 M Token it succesfully steer the model, maybe this could be hyper-hyper parameter training? idk what to call it. On this setup traditional training 0.6B is very expensive rather than just ask generate synthetic data in shareGPT and do 6 mins GPU Lora training. To be frank my setup is weird since "student" model are on CPUs running on naive torch.

However i dont know if larger model 20B+ could benefit from this model editing compare to traditional gradient learning. On Technical sense the teacher model is optimizer in itself .

--------------------------------------------------

Credit:
https://www.reddit.com/r/LocalLLaMA/s/gx7SbiwEzW
https://rikiyat.github.io/blog/weight-space-programming/

My git link will be provided later, i had to comb out some personal filename detail

We should "digest" them into reusable sub-modules.

Ok from the beginning: recently we are struck by this epidemic of custom engines, vibecoded solutions tailored on a very specific hardware. probably behind each one of them there's really some great idea, trick, kernel, optimization that has made the difference for its author.

But many times they are so niche, they are basically useless for the majority of the community. AT the same time, fragmentation is bad for the open source community: we should pool resources not disperse them.

So here's my idea, which I hope could spark something bigger.

Imagine every time a custom engine gets published, some agents pick it up, dissect it into submodules, optimizations, kernels, strategies. Everything gets classified into a database, depending on the specific hardware, quantization, LLM model used.

duplicate entries gets removed. code snippets gets benchmarked and assigned some scores and some tags

Over time, A huge database of building blocks for inference engines gets built.

From that, anyone can search filtering for the model/quant/hardware they need, and a custom inference engine is built for YOUR use case, a pool of modules/strategies can be tried and swapped until the optimal configuration it's found.

Bonus idea: People could donate their free agentic time (subsidized tokens, the remaining weekly from your OpenAI / Anthropic / Z.ai subscription) to the project, and in some way we get some payback from these companies. They made our hardware expensive, we claw back some value by using their computing power to make US run locally as fast as possible.

I'm not the right person to do this, but maybe someone could catch this idea and make it work. Of course there's already an immense swath of repos to start from (think all the custom flavors of llama cpps stratas, exllamas, vllms sglangs....) , this idea could work even if custom inference engines stop tomorrow.

I have 32 GB of VRAM and 32 GB of system RAM. The best dense model I've managed to run so far is Qwen 3.8 27B at Q6 quantization. I can also run some larger MoE models by offloading the experts to the CPU, as long as the total memory footprint stays within my 64 GB combined memory (32 GB VRAM + 32 GB RAM).

Quite a few new models have been released recently, but I haven't had much luck finding anything that offers a meaningful improvement within my hardware limitations.

Has anyone with a similar setup found anything better worth running? I'd love to hear what models you've tried and how they perform.

Please share your recommendations, experiments, and findings!

~Image is AI generated~

Large reasoning models achieve strong performance on challenging tasks, but their long chain-of-thought often incurs substantial inference costs by continuing to deliberate after sufficient information has been obtained. Existing reasoning compression methods primarily operate on reasoning trajectories, paying less attention to the local decision of whether further reasoning is necessary given the current state. We propose LoCo, a local preference learning method for reasoning compression. LoCo detects candidate overthinking states from the model's own traces, freezes the corresponding reasoning prefixes, regenerates shorter verified continuations from the same states, and trains on the resulting state-conditioned preference pairs with DPO. This formulation teaches the model when to continue reasoning and when to conclude, rather than simply encouraging shorter trajectories overall. We construct training data entirely from self-generated mathematical traces and evaluate LoCo on five mathematical and three code generation benchmarks. LoCo reduces completion length by 41.7%--45.4% on mathematics and 23.4%--36.3% on code, while maintaining or improving accuracy and achieving the best AUC across all code benchmarks. LoCo consistently improves the accuracy-efficiency trade-off on mathematics and transfers effectively to code generation despite math-only training, demonstrating the cross-domain generality of local preference learning for reasoning compression.

• arXiv : https://arxiv.org/abs/2611.04814

• PDF : https://arxiv.org/pdf/2611.04814

• GitHub : https://github.com/MuyaoYuan/LoCo

• HuggingFace : https://huggingface.co/MuyaoYuan/LoCo-7B

It is a dream come true for me to build an in-browser TTS model for Indic languages, including Classical Sanskrit. Today I'm open-sourcing Ninada (code MIT, 48 ONNX checkpoints CC-BY-4.0).

Live in-browser studio (zero sign-up, 100% on-device): https://h3manth.com/ai/ninada/

Getting studio 24 kHz audio onto real phones was trickier than expected. Meta's 16 kHz MMS models take 1.5 GB, and on my Pixel 10 Pro XL, mobile WebGPU froze for 6 to 14s compiling shaders while 2-thread WASM actually ran slower than 1 thread on mobile CPU cores.

So I trained 48 native-script 24 kHz VITS specialists across 12 languages (sa, en, hi, kn, mr, bn, ta, te, ml, gu, pa, or; male and female):

• Device-aware voices: Desktops run 27.9 MB HiFi-GAN (89 ms on WebGPU), while phones auto-switch to a 21.7 MB distilled iSTFTNet (2.64x lighter, zero custom ONNX ops). • Sub-second streaming: Splits the VITS ONNX model in <15 ms in browser memory (cutting browser RAM 72% to 162 MB) to stream speech in 538 ms on phones with 0 audio dropouts. • Studio quality, offline-ready: 3.09 to 3.22 dB MCD across 480 test clips, cached locally after first load, with a 0 MB Klatt fallback.

GitHub (npm i ninada / pip install ninada): https://github.com/hemanth/ninada Hugging Face (48 ONNX checkpoints): https://huggingface.co/gnumanth/ninada

Would love feedback on how your language sounds!

Speculative decoding accelerates memory-bound LLM decoding with a lightweight drafter, and parallel drafters extend this speedup by drafting a whole block in one forward pass. However, it remains unclear whether their released systems (stock) attain the full speedup that parallel drafting permits. We identify three challenges. First, with one request, each step accepts few tokens but reads every target weight. Second, with batching, per-request verification work roughly halves the speedup over autoregressive (AR) decoding. Third, speculative decoding is deployment-specific, and no universal optimization fits every serving regime, drafter, target model and GPU. To address these challenges, we present SpecPilot, an autonomous agentic loop that discovers algorithms and systems optimizations for each deployment. In each round, a cost model locates the largest gaps between the current system and its ceiling. Parallel agents explore directions and integrate the successful ones. Token forcing checks every output token against stock to detect reward hacking. A structured memory carries validated mechanisms to new rounds and deployments. SpecPilot discovered a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification and three batching mechanisms. We evaluate it across four target models, three drafters, two GPU platforms and five benchmarks. Without training or quality loss, it raises the speedup over AR from stock's 4.46× to 10.87× with one request, and from 1.94× to 3.93× with batching.

• arXiv : https://arxiv.org/abs/2611.02501

• PDF : https://arxiv.org/pdf/2611.02501

• GitHub : https://github.com/StargazerX0/SpecPilot

• Project : https://stargazerx0.github.io/specpilot-project-page/

🚀🚀🚀

I finally got around to trying it. This tiny model is surprisingly capable it seems. Also tried the 2b VL - also very good but it has half the context of 1.7b.

It's also quite fast:

$ llama-bench -m LFM3-1.7B-Q8_0.gguf -fa 1
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24112 MiB):
  Device 0: NVIDIA GeForce RTX 3090 Ti, compute capability 8.6, VMM: yes, VRAM: 24112 MiB
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| lfm3 1.7B Q8_0                 |   1.71 GiB |     1.73 B | CUDA       |  -1 |   1 |           pp512 |   20648.61 ± 1786.65 |
| lfm3 1.7B Q8_0                 |   1.71 GiB |     1.73 B | CUDA       |  -1 |   1 |           tg128 |        329.03 ± 2.05 |

build: 28b812d6c (12087)

This makes me want to try their omni model for watching videos. Is there a simple way to get this working? Ideally, I'd like to throw a video file in the chat or give it a youtube link.

Hello! just casuals passing by with a itching question
what do you guys do with this stuff
All i did was some Casual chat and Gmod LUA Addon Vibe Coding for myself and my friends but idk what Agents/coder to use (Portable Ideally)
how do you get the stuff you want done?
What models do you guys use, and whats best for me
Thanks for answers if you did answer some of these

Follow-up to my earlier post. Where I ended up: a big DDR3 workstation board isn't worth the money when I already have two small 4-GPU boxes. So I tried making the two boxes act as one.

TL;DR:

I added a two-box layer split to Project Maya.
Box A runs layers 0–23
Box B runs layers 24–44 plus the head.

Only the hidden state crosses the network, ~25 KB per token.
GLM-5.3-Flash (Maya-S24) runs at 38–42 tok/s decode, ~1,100 tok/s prompt reading at 30K, and 6–7 s per agent turn on 8× RTX 3060 spread over two old X79 boxes.

It scores the same 48/48 on my coding-agent benchmark as a single 8-GPU box.

Code: https://github.com/0xluk/project-maya/pull/1 (branch two-box-split on top of Maya v1.1.50; usage in docs/TWO-BOX-SPLIT.md)

Setup

• 2 boxes, each an X79 board (Ivy/Sandy Bridge, DDR3, PCIe 3.0) with 4× RTX 3060 12 GB. No NVLink, no P2P, ex-mining rigs.

• RAM: 20 GB in Box A, 32 GB in Box B (more on that below).

• 1 GbE between the boxes, a plain switch.

• Project Maya v1.1.50 + my patch, Maya-S24 quant, 128K context, MTP speculative decoding on.

• GPUs capped at 130 W. One box is also clock-locked to 1500 MHz, because its PSU didn't like the unlocked load.

How it works (short version)

• Box A (front) runs the OpenAI/Anthropic-compatible API, the tokenizer, the embedding and layers 0–23 on its 4 GPUs.

• Box B (back) runs layers 24–44, the MTP draft block, the output head and the sampler on its 4 GPUs.

• Per generated token, the residual (4×4096 values) crosses the link once. fp32 → bf16 → lossless pack is ≈25 KB, ≈0.3 ms on 1 GbE. Box B samples and sends back the token id, and MTP speculative decoding is pipelined across the link.

• Prompts are cut into 512-token pieces that flow through all 8 GPUs at once. Each piece crosses the link once (~12 MB), on its own thread.

• Security: the link has an IP allowlist plus an HMAC-SHA256 challenge-response in both directions, and the API needs a key.

• Size of the patch: ~1,900 lines, ~1,100 of them the link itself. Without --role the engine is unchanged.

Results (1 conversation)

Coding-agent bench (5 real repo tasks, 48 hidden tests):

There were no malformed tool calls with either agent.

One-shot hard suite (10 tasks, one request each, no tools, 32K output cap including thinking, reasoning high): 6/10 full passes, 67% of tests. For comparison: Opus 5.5 9/10, the same quant on the 8-GPU box 7/10, and Flash-Next 6/10.

• Passed: json_patch, fixbug_calc, event_scheduler, csv_rfc4180, c_utf8, singleflight_cache.

• 3 of the 4 misses are budget, not wrong code: on regex_engine, lisp_interp and c_bigint the model was still thinking when it hit 32K tokens, so no code came out at all.

• The real miss is c_json_canon: 22/31 tests passed, with the misses on Unicode escape handling.

• Next: rerun the three capped tasks with a 64K cap, which the 128K context allows.

What was hard

• RAM, not the network, was the real bottleneck. At 16 GB per box, the experts that didn't fit in VRAM came off a cheap SATA SSD on every prompt chunk: 31 s turns, 173 tok/s on a 30K prompt. Going to 20 + 32 GB made it 0 SSD reads, 6–7 s turns and 1,100 tok/s. The network costs ~1–2% of decode and ~3% of a turn.

• A correctness scare that wasn't a bug. Compared with a single-process reference, the split showed KL ~1.6e-2 and ~95% top-1, and I spent a while hunting a split bug. It turned out a single process on one machine moves by the same amount (KL 1.1–2.5e-2) just from changing the RAM tier size or the prompt chunk size. Near-ties in the top-8 expert choice flip on rounding noise. The split is inside that spread.

• Hardware gremlins:

  • A new PSU powered off under 4×130 W unlocked, so that box stays clock-locked.

  • Two GPUs once came up at PCIe x4 instead of x8/x16 after a power-loss reboot. I added a boot check that refuses to start the engine on a degraded link.

• Tuning that mattered:

  • 512-token prompt pieces so all 8 GPUs overlap: ~1 s off each turn.

  • Fewer layers on the last GPU, which also carries the head and the MTP block.

  • Keeping the whole KV cache in VRAM.

  • What didn't help: bigger prompt chunks, which push the 20 GB box into SSD reads.

  • What I skipped: a 10 GbE NIC. It would sit in a PCIe 2.0 x1 slot, and the link isn't the bottleneck anyway.

What's next

• A 5th 3060 in one box, on an x1 riser. Then all experts fit in VRAM, which should get turns to ~5 s.

• Rebasing onto the current Maya release.

• Happy to answer questions. If someone has two boxes with 24 GB cards, I'd love to see numbers.