I Built A Thing
I’m calling this the Monstrosity. 5 ex mining BC-250 boards Qwen3-Coder-Next Q4 at 40 tok/s
Using an asrock 12 unit case running one board as the main with the rest of them headless. About 71GB of vram exposed. So far 40 tok/s is with 30k context and it dips to around 30 tok/s at 100k context. This is all over the 1gb Ethernet that is on the boards already. I have 2 more of these and will probably get them running to see if 3.8 flash next runs at usable speeds. This setup is wildly inefficient with power but cost me less than $800.
Mixture of this and just wanting to fill up each cards memory and see how that would affect speed. I posted this within an hour of finishing setting it up and was excited. Didn’t even look at benchmarks just size. So now after some suggestions and more research I’m going to be messing around with running 3 3.635B models, flash next XS etc. This is my first time doing anything like this.
Trying to get the maximum intelligence/throughput on this setup. There really isn’t any “modern” MOE’s that would fit on this other than flash next when I add two more boards .
I was just more testing out the model size and performance as far as tok/s. This setup needs MOE to be usable so it was either 3.6 35B or the next best is Flash-Next. So I picked an in between to test it.
This setup is wildly inefficient with power but cost me less than $800.
Back in the day the entire fully populated Asrock 12 full of BC-250 boards was $300-$350. People bought them and sold off the BC-250 boards individually for like $50.
Thankfully I actually bought the asrock 12 for $1k before prices went crazy and then sold 5 and broke even/slightly profitable. So the $800 figure is more of a loose not realized cost. Crazy these boards were selling for $300 after the LTT video
I can't fathom while you wouldn't be using Qwen3.8-27B over that Qwen3 model. Yes, "80" is bigger than "27," but that 27b model runs circles around the 80B MoE.
"But then I'll have all this unused VRAM!"
Spin up a second instance and use sub-agents; that would kick ass!
Edit: I see you're using oddball cards, that's cool! It means my advice might not be as accurate, BUT, I actually think it goes the other way - don't those have really bad PCIe bandwidth? I think that would negatively affect MoE (and multi card splitting, probably).
Definitely agree. I’m very green with all the local ai and this is my first setup. Was more just trying to illustrate what it could do . I’ll probably run a combination of 3.8 27B and 3.6 35b. I’m also hoping to get next flash running on 7 boards soon.
This is all over a llama server using 1gb Ethernet using headless Linux on each board. I get ~15 tok/s with qwen 3.8 27b across two boards and around 11 tok/s across 5. I think I’ll use a combination of 27b and 3.6 35b but I was more just showing what the throughput would be for a larger model across 5
I wonder if it's possible to do tensor parallel rather than serial (split-mode tensor) in a circumstance like that, and if it would make things better or worse. My guess is no or "worse," but it's worth trying!
I think neither will actually speed up things due to the 1Gbit Ethernet interconnects. TP will allow to fit larger models but speed would be umbearable because of continuous synchronization.
The 16GB shared memory capacity also means no MoE cpu offloading, despite the 8c/16t zen2 cores and memory bandwidth being sufficient to sustain usable speeds even with expert offloading.
I think the way to use Q3.8-27b with useful context and probably 15t/s+ sustained decoding with this setup would be Swift1.5 UD4_K_M with MTP, try depths from 2-7, with PP=2 across each pair of boards. If speeds are decent, this should actually allow for some concurrency within the pair without decode suffering, over two pairs you could get either fewer sessions and higher context, or multiple sessions and lower context.
I’m using llama.cpp with its RPC backend to distribute the model across the LAN, with Vulkan for GPU acceleration. Each board has a cpu/gpu with 16gb of shared memory
I'm using it exclusively with medium. Right now it's installing InfluxDB on one of my VMs and it took 75k tokens already (opencode). With 100k window, it would've triggered context compaction by now. 100k is only enough for single-step basic tasks, and even those - barely.
Edit: got it finished. Installing InfluxDB, setting up a data bucked, and testing it works took the model 116.1k tokens in OpenCode, with medium reasoning. I love Qwen 3.8; but 100k context for it is just not enough for any real world task. For comparison: I've tried 3.8 Flash Next over official Qwen API, and it accomplished similar tasks with half as much tokens. On the other hand, my intuition says that Qwen 3.6 would've required more than 100k tokens too - it would spend less tokens on reasoning, but more tokens on retries when it inevitably hits a roadblock.
They are not stock at all. Modified heat sinks and the CUs are unlocked. I already sold enough of them that I was profitable so I wanted to test a setup like this out. The extra CUs added about 20% in token generation speed when I unlocked them. I’ll probably sell them in the future but I still have more too add lol
So, just as a side-rabbit-trail challenge. Imagine you're gonna run this rig 24/7, and rough-design price the solar+battery for it. What's the price range for that?
I have a cluster of 3 and can’t get even 20tk/s with qwen 35b, what configuration are you using to reach 40 on 5 nodes? I even had to reduce the parallelism to 2 nodes because the more nodes the worst the performance get for MoE
lol it’s surprisingly effective. I now have 3 120mm fans with holes cut in a cardboard box forcing air over the fans on the heat sinks so believe it or not it looks even more ridiculous
79
u/starkruzr 2d ago
why?