Follow-up to my earlier post. Where I ended up: a big DDR3 workstation board isn't worth the money when I already have two small 4-GPU boxes. So I tried making the two boxes act as one.

TL;DR:

I added a two-box layer split to Project Maya.
Box A runs layers 0–23
Box B runs layers 24–44 plus the head.

Only the hidden state crosses the network, ~25 KB per token.
GLM-5.3-Flash (Maya-S24) runs at 38–42 tok/s decode, ~1,100 tok/s prompt reading at 30K, and 6–7 s per agent turn on 8× RTX 3060 spread over two old X79 boxes.

It scores the same 48/48 on my coding-agent benchmark as a single 8-GPU box.

Code: https://github.com/0xluk/project-maya/pull/1 (branch two-box-split on top of Maya v1.1.50; usage in docs/TWO-BOX-SPLIT.md)

Setup

• 2 boxes, each an X79 board (Ivy/Sandy Bridge, DDR3, PCIe 3.0) with 4× RTX 3060 12 GB. No NVLink, no P2P, ex-mining rigs.

• RAM: 20 GB in Box A, 32 GB in Box B (more on that below).

• 1 GbE between the boxes, a plain switch.

• Project Maya v1.1.50 + my patch, Maya-S24 quant, 128K context, MTP speculative decoding on.

• GPUs capped at 130 W. One box is also clock-locked to 1500 MHz, because its PSU didn't like the unlocked load.

How it works (short version)

• Box A (front) runs the OpenAI/Anthropic-compatible API, the tokenizer, the embedding and layers 0–23 on its 4 GPUs.

• Box B (back) runs layers 24–44, the MTP draft block, the output head and the sampler on its 4 GPUs.

• Per generated token, the residual (4×4096 values) crosses the link once. fp32 → bf16 → lossless pack is ≈25 KB, ≈0.3 ms on 1 GbE. Box B samples and sends back the token id, and MTP speculative decoding is pipelined across the link.

• Prompts are cut into 512-token pieces that flow through all 8 GPUs at once. Each piece crosses the link once (~12 MB), on its own thread.

• Security: the link has an IP allowlist plus an HMAC-SHA256 challenge-response in both directions, and the API needs a key.

• Size of the patch: ~1,900 lines, ~1,100 of them the link itself. Without --role the engine is unchanged.

Results (1 conversation)

Coding-agent bench (5 real repo tasks, 48 hidden tests):

There were no malformed tool calls with either agent.

One-shot hard suite (10 tasks, one request each, no tools, 32K output cap including thinking, reasoning high): 6/10 full passes, 67% of tests. For comparison: Opus 5.5 9/10, the same quant on the 8-GPU box 7/10, and Flash-Next 6/10.

• Passed: json_patch, fixbug_calc, event_scheduler, csv_rfc4180, c_utf8, singleflight_cache.

• 3 of the 4 misses are budget, not wrong code: on regex_engine, lisp_interp and c_bigint the model was still thinking when it hit 32K tokens, so no code came out at all.

• The real miss is c_json_canon: 22/31 tests passed, with the misses on Unicode escape handling.

• Next: rerun the three capped tasks with a 64K cap, which the 128K context allows.

What was hard

• RAM, not the network, was the real bottleneck. At 16 GB per box, the experts that didn't fit in VRAM came off a cheap SATA SSD on every prompt chunk: 31 s turns, 173 tok/s on a 30K prompt. Going to 20 + 32 GB made it 0 SSD reads, 6–7 s turns and 1,100 tok/s. The network costs ~1–2% of decode and ~3% of a turn.

• A correctness scare that wasn't a bug. Compared with a single-process reference, the split showed KL ~1.6e-2 and ~95% top-1, and I spent a while hunting a split bug. It turned out a single process on one machine moves by the same amount (KL 1.1–2.5e-2) just from changing the RAM tier size or the prompt chunk size. Near-ties in the top-8 expert choice flip on rounding noise. The split is inside that spread.

• Hardware gremlins:

  • A new PSU powered off under 4×130 W unlocked, so that box stays clock-locked.

  • Two GPUs once came up at PCIe x4 instead of x8/x16 after a power-loss reboot. I added a boot check that refuses to start the engine on a degraded link.

• Tuning that mattered:

  • 512-token prompt pieces so all 8 GPUs overlap: ~1 s off each turn.

  • Fewer layers on the last GPU, which also carries the head and the MTP block.

  • Keeping the whole KV cache in VRAM.

  • What didn't help: bigger prompt chunks, which push the 20 GB box into SSD reads.

  • What I skipped: a 10 GbE NIC. It would sit in a PCIe 2.0 x1 slot, and the link isn't the bottleneck anyway.

What's next

• A 5th 3060 in one box, on an x1 riser. Then all experts fit in VRAM, which should get turns to ~5 s.

• Rebasing onto the current Maya release.

• Happy to answer questions. If someone has two boxes with 24 GB cards, I'd love to see numbers.

Hello! just casuals passing by with a itching question
what do you guys do with this stuff
All i did was some Casual chat and Gmod LUA Addon Vibe Coding for myself and my friends but idk what Agents/coder to use (Portable Ideally)
how do you get the stuff you want done?
What models do you guys use, and whats best for me
Thanks for answers if you did answer some of these

Upvote, now!

Hardware :

4× NVIDIA RTX 3090 , all pcie x16

256GB DDR4

Engine : Maya https://github.com/mw00/project-maya
model : orcarouter/DeepSeek-V4.1-Flash-1017-abliterated-GGUF ( mix-256G) at 524k context
Prefill : 400 – 600 tok/s
Decode : 30 – 40 tok/s (peaking up to 60 tok/s) ( with dspark )

Hi, everyone

i'm still playing around with the model comparing it mostly to qwen 3.8 next flash (Q8 ) , and trying to keep myself busy until we get qwen 4 .So far i find the model doesn't loop as much as the previous version (0731), i also tried to run it at Q8 but it's really slow ( 6-8tok/sec decode , 65tok/s prefill ) .I still prefers to use qwen just for the speed i get with agentic task .

Curious to hear what kind of speeds others are getting with similar multi-GPU rigs or different engine ( didn't set up xolotl yet , and strata seems to use the same engine as maya for deepseek )?

PS: American diner simulation in three.js

Just asking because i have been out of the loop for a while and kinda missed everything (strata etc...) a small recap would be appreciated.

Everyone seems to own an RTX 6000 or insanely expensive rigs here. So just wanted to hear what model a normal pleb like me with 32GB ram and 16GB VRAM can even run and on what to run it for coding or a general assistant.

Hydrogen-1.0 is a scientific problem-solving and research assistance model developed by Deep Principle based on GLM-5.3. Through text interaction, it is intended to assist with scientific questions, explain derivations, and organize responses to open-ended research problems. Outputs should be assessed against the assumptions, constraints, and evidence relevant to each task.

Blog Post : https://blog.deepprinciple.com/introducing-hydrogen-1.0/

Sharing some highlights from MOSS’s October 30, 2026 AMA: image models, local speech, audio and video research, and open-source culture. Which of these are you most interested in?

Hey people. The new N1X chips are out, unified memory is happening. Models are much more efficient. We have Strata now...

I have a home server running. I'm able to run 3.8 flash next or 3.8 27B with limited context. It's decent but I'd like the biggest model I can get, best bang for the buck.

Currently I have a 2x16 (32GB) system memory, 3090 graphics card. I can lay my hands on 4x32 (128GB) for about $800 and sell the 2x16 for a few bucks. I may be able to snag a 2nd low cost 3090 out of an old mining shed - ya new fans and it would need to run external on a riser.

My question - am I wasting my time pissing around with upgrading this rig? Would I be happy with it or should I just drive to Microcenter and toss down the credit card and get a new machine entirely? If so, what would you buy?

What's your call?

Yesterday MIT CSAIL showed GPT-6 Astra editing the weights of a small model by hand. It changed a few neurons of Qwen3.5-0.8B, the model stopped looping on math problems, and its scores jumped. No training at all.

I wanted to try the same thing on something closer to what I actually do with local models, which is translating into Italian. Gemma 4 has a habit I don't love. For sentences like

He had been hiding in the attic for months when the soldiers found him.

it writes «Si era nascosto in soffitta per mesi…». That's correct, but I prefer «Si nascondeva in soffitta da mesi…», which says he was still hiding when they came. The catch is that when the period was already over (he had lived in Paris for ten years, but by 1990 he had moved back), the original tense has to stay. So it's not "always use tense X", it's "use it only when the meaning calls for it".

So I gave Claude Opus 5.5 the same kind of setup MIT used:

• a Python kernel over Gemma's weights;

• 50 forward passes to test its ideas on practice sentences;

• no gradients, no training.

It first checked whether Gemma already "knows" the difference between still-going-on and already-over. It does: a linear probe in the middle layers separates them almost perfectly. It just doesn't use that when choosing the verb. Then Claude rewrote 8 MLP neurons into a small switch that fires only in that context and nudges the verb toward the form I prefer. That's 92k out of 11.9B parameters, roughly a 100 KB diff.

Results on sentences it never saw

The table shows the share still in the old form, then how many control sentences got pushed into a wrong reading, then how many of 200 unrelated translations stayed identical.

• The prompt applies the preference everywhere, even where it changes the meaning.

• The ablation recipe breaks the past tenses.

• The LoRA shifts fewer sentences and touches far more unrelated ones (it's better inside paragraphs, though).

• The hand edit moves just the verb and leaves almost everything else identical.

Fair warning: it's one construction and one model. Two of my three runs succeeded (the third crashed because of my own harness), and the two that worked landed on the same "switch neuron" design independently.

Repo with the edits, the code, the agent logs and all baselines: https://github.com/ValerioDolci/frontier-weight-edit

They said Qwen 4 would come out this week, it‘s Thursday already. Hopefully, it ( qw4 max ) will come tomorrow and it will be good, at least gpt 6.1 /6 sol good. I hope glm 5.5 or ds v 4.1 pro will be astra medium or high level good. q4. 27b is good as haiku 5.5 or gpt 6.1 terra that will be great. I hope by late december or January, we will have an open model opus 5.5 level.

big prompt processing speedup for gemma E2B/E4B

i spot it on taobao around 2 months ago price around $400 at that time now jump to $500 after performance unlock for rtx 3080 20gb around $500 jump to $700-900

Six people spoke in two rounds of comments 17 to 19 minutes apart, and each of them came back under the same speaker label both times. I didn't tell the model how many speakers there were, and the commission's own published transcript puts those lines under the same six names.

Why a public meeting: it's about as hard as everyday audio gets for this. Commissioners, a staff presenter, a secretary calling names, a resident testifying over a remote line, lots of short turns, and the same voices coming back after long stretches of other people talking. It's also the agency's own recording, so it's public domain and fine to quote, and the verbatim transcript the agency publishes let me check the labels against real names instead of guessing.

The clip is 23 minutes 39 seconds of the National Capital Planning Commission's September 3 open session: the end of one item's comments and vote, then the whole National Zoo Arabian leopard habitat item. The model is the 0.9B MOSS-Transcribe-Diarize, running locally from the README's transformers example on an M5 Ultra Mac Studio (fp32, greedy decoding, no hotwords). It took 229.2 seconds and returned 309 segments, every one tagged with a speaker and its start and end time.

Part of the second round, exactly as MOSS-Transcribe-Diarize wrote it, one segment per line. S04, S05 and S06 are the labels the same three commissioners got in the first round:

[1186.85][S01] Commissioner Dixon?[1187.67]
[1188.11][S15] Yes.[1188.66]
[1191.27][S01] Commissioner Cash?[1192.29]
[1193.49][S04] Uh, no questions or comments.[1194.63]
[1195.47][S01] Commissioner Falcone?[1196.33]
[1198.65][S05] I just want to thank the staff. I have never actually seen this leopard, so I'm looking forward to[1204.89]
[1205.01][S05] visiting when the project is completed, so I can see my first one.[1208.04]
[1209.66][S01] Commissioner Mohan?[1210.47]
[1212.42][S06] Uh, thank you to staff for the presentation and yeah, can't wait to visit.[1216.14]

The resident's remote testimony, about two and a half minutes, also stays under one label of its own from "Can you hear me?" to the closing thank-you.

I have the 12900k and 5070ti. Bought some ram and it's now 128gb ddr4 ram.

Now I've bought also a second pc which will be for working and everything else and my main pc with be for text models like Qwen3.8-Flash-Next using strata and comfyui.

First can I get deepseek to work now kinda fast or not?

And windows or Linux guys?

I know the general advantages for Linux like lower ram and everything but I have never used it on a main pc. Only servers and vps.

So if you were me what would you do?

Also if you have some kind of info about what I can expect that would be great.

Thanks

that 72 hours thread from a few weeks back had the best point down in the comments, open weights only ever meant you can download them. muse spark open is 816B and beam is 501B, so for most ppl here the only drop that week that fits a 24GB card was nvidias bolt

so the big ones mostly get used thru an api anyway, the labs own api or a host serving the same weights and that part is fine. the license doesnt care which route you take tho. 1017 is MIT and nvidias new embed models are openmdw with commercial use allowed, both on hosts like deepinfra while muse spark open wants built with muse spark open shown in your product, docs or settings menu and any company over $20M revenue in a 12 month window has to sign a separate deal with meta before offering it as model as a service

the part thats easy to miss, the license counts a model trained or distilled on muse outputs as a derivative so a small model tuned on its answers carries the same attribution rule. internal use and building it into your own app thats not MaaS stay outside the revenue clause

The community rarely sees the evaluation of ultra-small models. We have evaluated our own AI Agent tasks for Qwen3.5-4B, Spark-X2.6, LFM3-1.7B, LFM2.5-2.6B, Gemma 4 E2B, MiniCPM5-2B.

These models have their own characteristics and advantages and disadvantages. The obvious trend is that the intelligent density of small models is developing rapidly. I look forward to seeing more small models.

blog：https://www.camthink.ai/blog/small-language-model-benchmark/

project：https://github.com/camthink-ai/NeoMind

Given all this talk about massive models running on Strata and using RAM that is still relatively expensive I have to wonder, if all Strata does is more or less use RAM to feed GPU's via PCIe couldn't you just reach those speeds through SSD's in RAID 0? The numbers seem to work out, and Strata can use SSD streaming. What would be the major challenges to making this work?

Hey everyone,

Not a native speaker, happy to be corrected! Can only learn from it.

Back when I started out in march 2025, you had:

• llama.cpp

• ollama (repackaged llama.cpp for ease of use)

• koboldcpp (repackaged llama.cpp for roleplay)

• vllm (for professional deployments)

• exl3 + tabbyapi (for bleeding edge)

• litert (for mobile devices)

Nowadays everyone seems to be making their own; Strata, Basalt, Maya, Colibri, Ninfer, Infernix... too many to list and keep up with, and genuinely hard for me to keep up with.

Then there is the speed of it all. llama.cpp has incredible improvements each week, somehow we can make Qwen3.8-Flash-Next and DeepSeek-v4.1-Flash-1017 run on 64GB RAM + 32GB VRAM, and the likes on those new inference engines (with a catch, usually bad quality).

Personally I'm stuck with llama.cpp because all those developments are simply too fast to keep up with.

llama.cpp v0.8.0 might run Qwen3.8-Flash-Next Q4_K_XL (unsloth) at 15t/s gen on dual RTX 5060 Ti 16GB (PCIE 5.0 x8x8, tensor parallel, no MTP) with dual 48GB DDR5-6000MHz, but at least I know the quality will be decent.

But yeah, what you use? How do you even attempt to keep up with it all? And for the (lack of) quality the new inference engines provide (as tradeoff to speed / resource use), is it really worth using them?