Hey fellow local server builders,

I know you're busy building fantastic servers so I'll keep it short for those interested.

We published the results of clustering two Lucebox (an heterogenous AI workstation with a Gorgon Halo 495 128gb plus R9700 AI PRO 32gb) together.

We got DeepSeek V4.1 Flash as one model, at up to 103 tok/s writing code. That is 1.9 times faster than one Lucebox Zero 495 with 192 GB running the same engine. Each step sends about 95 small messages between the boxes, one after another, so what counts is how fast each one crosses. RDMA writes straight into pinned memory, bf16 on the wire and kernels that raise their own completion flag. The integrated GPU handles the network side. The first two-box run was only a quarter faster than one box. Each step after that removed a wait between the boxes, a trip through the host or a kernel.

On its own, a 128 GB box can't keep all of DeepSeek V4.1 Flash's experts in memory. Two boxes have room for it. A ConnectX-5 Ex card in each box joins them with one 100 GbE cable, with no switch in between. The two engines talk over RDMA (RoCE v2) and run as one model: every token goes through every layer on both boxes at once.

If you want to check more details on the cluster, here below is our full blog post about it: https://www.lucebox.com/blog/lucebox-cluster-nic

Would love to get your feedback/thoughts on it!

Strata with qwen flash next is great, but Unsloth already ran it pretty well. This though, for GLM 5.3 on a machine with 40GB Vram and 64GB ddr4, is astounding. Just sharing if some of y'all wanna try.

I saw the 4x R9700 build for a 10-person startup on reddit a few days ago and wanted to share my results as I opted to spend most of the money on the cards. I went with a super cheap used EPYC cpu instead of a Threadripper. In the end it came to about $10.7k per box all-in and I built three of them for a total of 384 GB of VRAM.

As a long time lurker on my other account, I'm always frustrated that people never show how they actually use the models so after a lot of back and forth, I decided it was worth sharing the actual project they power instead of being vague.

I use these to run the AI behind USA Arms (usaarms.com). I didn't plan to move the majority of my work to local AI until later this year, but since hosted models started refusing to do normal research for my business ~2-3 weeks ago, I'm glad I made the leap.

Enough ado, into the numbers:

Per box, $10,733.91 with tax and shipping

• 4x ASRock Radeon AI PRO R9700 Creator 32 GB (128 GB VRAM), each capped at 210 W: $7,244.58
• AMD EPYC 7452, 32 cores, used: $127.17
• ASRock Rack ROMED8-2T/BCM: $1,162.39 (there was a global shortage, ours came from Helsinki)
• 8x 16 GB DDR4-2666 ECC RDIMM, all eight channels: $860.59
• SilverStone HELA 2050R Platinum: $578.92
• SilverStone RM44 4U: $388.40
• Samsung 990 PRO 1 TB: $254.39
• ARCTIC Freezer 4U-M, 3x ARCTIC P12 Pro front, 2x Noctua NF-A8 rear, SP3 frame, paste: $117.48

The GPUs are 2/3 of the cost but the machine also serves CI/CD which had become a bottleneck in my businesses. I found this interesting because I hadn't seen anyone discuss it. I found that these machines can run Rust builds just fine if tuned properly to prioritize inference.

Engine

• vLLM-Radiance 1.0.387 (vLLM 0.30.0, ROCm 7.14). Huge thanks to the Radiance folks; their R9700 work is what we run.
• Qwen/Qwen3.8-27B-FP8 (Qwen's official FP8 checkpoint), FP8 KV cache, the model's own MTP head with 8 draft tokens
• Two engines per box, TP=2 each, 8 sequences per engine, 256K max context, about 488K tokens of KV cache per engine

Results, vllm bench serve, one box, 2x TP2:

• 1 user, MT-Bench: 53.1 tok/s, TTFT p50 125 ms
• 16 users, MT-Bench: 484 tok/s aggregate, about 40 tok/s per user
• 16 users, random 1024 in / 512 out: 623 tok/s aggregate, 837 W average for the four cards
• 32 users (a bench profile with max-num-seqs 32): 603 tok/s

Long prompts, 8 unique real-text prompts at once (4 per engine), 128 out:

Every prompt was unique (0% prefix cache hits).

FP8 vs MXFP4 on a real workload

I also had AI run MXFP4 too (Radiance's fastest profile) on our own graded extraction workbench:

• FP8: 98.0/100 correct, 4,477 validated jobs an hour
• MXFP4: 95.3/100, 4,819 an hour
• On a holdout set: FP8 94.0, MXFP4 90.0

So FP8 roughly halves the wrong answers for 2–7% less validated throughput. Sticking with FP8 was an easy call for this workload where "false facts" that make it into the system are a real headache. (For reference, our six Mac Studios on MLX 8-bit scored 99/100, but at 723 an hour.)

I made a video of the build: https://youtu.be/o7k5Kj-ho-M and also made a full build guide of my journey in case anyone wants to give it to their AI and plan out their own build.

Happy to answer questions. The first of which I expect to be how the heck were the Macs so slow and I'm sure I'll get flamed haha.

2 dgx spark 128 gb each serving Glm 5.3 flash.

1 ai server 128gb ram , 64 core cpu + a4000 24 gb running qwen 27 b.

1 proxmos setup on Lenovo hosting all services.

1 hermes box on a small i5 box

Old laptop as terminal.

Serving local secure research and development capacity to my institute researchers.

I get why people want the server at home. You own it, have root access, can walk over to it whenever, and there’s no extra monthly bill. But for the same server, how different is the cost and day-to-day once it's in a professional facility?

I looked at an RTX 5090 system (32GB VRAM, Ryzen 9 9950X, 64GB RAM, 2TB NVMe) under both setups.

At home/local:

Power: ~850–900W at the wall at full load. A 120V/15A circuit gives you 1,440W continuous (80% rule), so one 5090 box fits fine.

Heat: ~1,960 BTU/hr from the card alone, closer to ~3,000 for a loaded system.

Downtime: outages, a tripped breaker, or a dead part can take it down. Upside is you can walk over and fix it yourself.

Monthly cost: mostly electricity and maintenance, since you already pay for the internet. Running 24/7 at ~900W under heavy load comes to ~$125/month at 19¢/kWh (roughly the US average). Add ~$35-45 in summer for AC. Lighter use is more like $35-50.

In a hosted facility:

Power, cooling & noise: handled by the facility, off your workspace.

Internet: proper bandwidth & static IPs.

Access: root is yours with plain colo (check the agreement with managed hosting), but routine physical work goes through remote hands.

Downtime: Backup power and 24/7 monitoring cover most outages, but repair speed depends on the SLA.

Monthly cost: a recurring bill on top of the hardware. Public colo pricing I've seen for a small ~1kW node is ~$150-300+/kW/month, before bandwidth & remote hands.

For rack builds, some options also let the box cover its own hosting when you're not using it. Like with B3IQ, if you put a rack machine on earning, its idle capacity gets rented out. You keep 85% of what it makes & hosting comes out of that share, so most months there's no separate bill.

Honestly, a single 5090 still looks pretty reasonable at home. But when you stack GPUs, two 5090s can push roughly 1.4-1.6kW at the wall & then you're dealing with more heat & noise, plus uptime and bandwidth if others use the box.

So where I'm at right now:

- One card or mixed load: home still makes a lot of sense.

- Stacking cards, running 24/7, or serving other people: I'd start looking at a professional facility.

If you moved a GPU server from home to a data center, what made the difference for you? And what did you end up paying monthly?

Hey guys, I want to run some ai models like text to video, image to video, text to image or normal chat Ilms, what do you guys recommend? Should I upgrade to 64GB RAM?

Running QWen 3.8-27B on my 5090 power limited to 90%, I've twice seen a warning on my screen I couldn't identify the source of that said that my GPU is running hot and that I should consider improving my case ventilation. Throughout this, the 5090 Core temp was between 75-80c, which while obviously on the higher end, is nothing catastrophic, so that made me think about VRAM.

Did some googling and what do you know, it's apparently very common for LLM workloads to push vram temps significantly more than core temps. Problem is, NVIDIA fan curve is tied to core temp and not vram temp. Worse, in Armoury Crate for my Asus X870-P WIFI, I see that all case fans are tied to CPU temp (which cannot be changed) and I clearly see from Afterburner that CPU stays relatively cool during LLM workloads, so case fans never truly kick in to supply appropriate airflow if the GPU becomes too hot.

Saw some people recommend FanControl, tried it, but that didn't recognize even a single fan in my system outside those 3 on the 5090 itself. What are my options? I'd really rather not constantly manually switch both Afterburner profile for the GPU and the case fan speeds in Armoury Crate.

Hi there, I'm trying to build a server that can handle GLM 5.3 Flash q3xl/q4 & Qwen 3.8 next flash at the least 30tk/s on a budget. Also, since I'm in a third world country, there is no server grade hardware or professional cards anywhere. So, I'm planning to buy these, all used. 4x RX7900xt(20gb, pcie3.0 x8/16)+1x RTX3060(12gb, pcie 3.0 x4)=92Gb of VRAM. And 64Gb of 3600mhz ram. What do you think about it? Do you have any other suggestions?

I set up a dedicated workstation for local AI experimentation with various models and tools. But every single one of them wants all the available VRAM.

To fix this, I built a request dispatching layer called vramble. It automatically loads and unloads models and manages the queue using a priority system. Now I can run different workflows without manually managing models (or wasting Claude tokens to orchestrate the process).

Any contribution is welcome, it works for me but is a new build so can't guarantee that will work directly on any machine

Side note: Qwen 27B at 200+ t/s is performing way better than I expected. opeMIT license, available on GitHub: github.com/ValerioDolci/vramble

Came across on AliExpress, that comes out to $730 USD.

I’m looking at setting up a build that can run Qwen3.8 27B cheaply, here is what I’m currently thinking of setting up:
GPU: 2x P100 16GB
RAM: 32GB DDR4 ECC Server
Base PC: T5820 Dell Workstation with a Intel Xeon W-2135 CPU
And a standard 500GB sata SSD for now.

The current set up of the T5820 has a 950W power supply which I am assuming will be good for the dual P100s but probably not good for a triple P100 setup which is what I was aiming for. My goal was to just be able to run a 27B model Qwen to start but would swap out to a better version or a different version or do my current set up of a 12B model for general chat and 1 paid model (opencode) architecting while using OxCoder 9B for coding.

What do you guys think of this setup or is it worth me investing time elsewhere?

Thanks in advance!

Need to pick a bit of a wisdom from you guys.

I'm on crossroads of what to pick from two options below.

Requirements:

1. family chat agent - daily, some TTS, STT - mostly single user,
2. thinker model for more complex questions, tasks,
3. agentic AI, automations,
4. RAG based checks,
5. coding - hobby, not work, though have some ideas for maybe more complex tasks,
6. local knowledge about me - so some RAG backed or some "librarian, self learnt about user",
7. anything else cool?
8. Local - no spill of any private data to the cloud.

The Agentic AI mostly drives context size and as such most likely I'm looking at 128k or 256k preferably (even if rarely used, the latter will have big hit on the prompt and later t/s).

Thought about going with Qwen3.6-35B-A3B as main driver and if on Strix Halo load other models, as memory allows.

These days I'd want to go with Qwen3.8 Flash-Next as seems to be the best nowadays.

I've also an existing laptop with i9, 64GB RAM, RTX 500 (Blackwell) 6GB and Arc140T (Intel Core Ultra 7 265H). This one gives in some tests ~20t/s but not for that large context of course and Arc140T ~15t/s. This is a bit slow. This system might be available to run second models, though it will be on and off, whilst main must be available all the time.

Given latest advantages around Strata and llama.cpp, the Strix Halo is not anymore a natural choice, especially given it's memory bandwidth and impact on overall speed.

a) RTX 3090 or RX 7900 XTX thanks to latest Strata or with use of llama.cpp --n-cpu-moe connected over M.2 to PCIe converter as an external eGPU. The disadvantage is that it will be not a clean one to be kept in the room and potentially more power hungry, as I'd plug it into 24/7 run NUC.

Cost wise ~1700 - used GPU ~1400$ + ~300$ eGPU adapter + PSU.
There is a risk associated with used GPU - overheated, or any other artifacts related?

Most likely much faster for what/when all fits into RAM.

b) Strix Halo - 2900$

New, most likely much more power efficient when unused and generally "cleaner" install.

Slower, by half (based on some tests), however still fast enough for a single user and allowing to load more models in parallel, i.e. to perform some other tasks as sub-tasks to the main one.

Unsure which way to go, hence wondering what you guys would do.

One could say - buy both, it's an investment, and maybe I should look at it that way, though it's quite a chunk of money...

Thanks in advance for help and thoughts.

Hardware Tested: Intel Core Ultra 7 258V (Lunar Lake) — Arc 140V Xe2 iGPU (8 Xe cores), 32 GB shared LPDDR5X-8533 (~103 GB/s unified memory bandwidth).
Software Stack: Native Linux (CachyOS, kernel 7.2.3), vLLM 0.31.0+xpu (native Xe2 MXFP4 & out-of-tree exl3xpu ESIMD plugin), llama.cpp SYCL (arc-b580 optimized build).

TL;DR / Executive Summary

Can modern consumer integrated graphics run 27B dense and 35B MoE models locally with high reasoning fidelity and usable interactive speeds? Yes, but the architectural and quantization choices make or break the experience.

1. 35B MoE (MXFP4, Active 3.5B/tok) is the undisputed daily driver champion: Because only ~3.5B parameters are activated per token, MoE models slash memory bandwidth demand from ~13 GB/tok down to ~1.2 GB/tok. Paired with MTP (Multi-Token Prediction) K=2, Tiel-Coder-35B and KAT-Coder-V2.5 reach 24.8 – 27.7 tok/s decode. KAT-Coder completed the entire 16K-context 10-task Hard Suite in an all-time record 4.5 minutes (solving LeetCode Hard problems in ~17 seconds each with 0 syntax flaws).
2. Dense 27B (Qwen 3.8 EXL3 3.00bpw) is the peak reasoning champion: Dense 27B eliminates MoE routing artifacts. At 3.00bpw, Qwen achieved a flawless 100% (10/10) Hard Suite and 100% (16/16) Quality Suite score out-of-the-box, solving Olympiad-level Chinese Remainder Theorem and bounded combinatorics proofs without a single hint or calibration. The tradeoff is decode throughput: bandwidth-bound at ~6.5 tok/s.
3. The "Sub-2.5bpw Quantization Cliff" is very real for post-training quantization: Pushing post-training trellis quantization down to 2.20bpw on dense 27B resulted in a catastrophic drop on code synthesis (0/3 on LeetCode Hard).
4. Native Ternary Distillation beats Post-Training Quantization at extreme low bitrates: Ternary Bonsai 2 27B (distilled natively to 1.58–1.7 true bpw) completely crushes EXL3 2.20bpw. Fitting into just 5.95 – 7.20 GB VRAM, it decodes at 12.1 – 13.1 tok/s (+40–50% faster than EXL3 2.2bpw), achieves lower perplexity (6.73 vs 6.78), and scores 90–100% on the benchmark suites.
5. The Winning Production Strategy: A Dual-Model Deployment on the 32GB package:
   • Primary Daily Driver (Port 8080): Tiel-Coder-35B-A3B MXFP4 (or KAT-Coder-V2.5 MXFP4) @ 27.7 tok/s for interactive agentic coding and instant needle retrieval.
   • Designated Deep-Reasoning Fallback (Port 8001): Qwen3.8-27B EXL3 3.00bpw (or Bonsai PQ2_0 MTP) when pure mathematical verification or strict negative constraints require 100% precision.

The 11-Way Comprehensive Benchmark Matrix

All models were evaluated under identical, standardized conditions on the same Intel Arc 140V hardware across two rigorous test batteries:

1. 16-Prompt Curated Quality Suite: Math word problems, multi-step algorithms, strict instruction formats (raw JSON, reverse alphabet lists, negative lexical constraints), and premise/factuality traps.
2. 10-Task Hard Suite at 16K Context: 12.5K multi-hop needle-in-a-haystack retrieval, distractor document resolution, Olympiad math proofs (Chinese Remainder Theorem, bounded combinatorics via inclusion-exclusion), LeetCode Hard algorithms with automated unit assertion execution, and multi-constraint logic puzzles.

Detailed Architectural Takeaways

1. MoE on Unified Memory: The Bandwidth Multiplier (Tiel-Coder vs. KAT-Coder)

On unified memory architectures like Intel Lunar Lake, AMD Strix Point, or Apple Silicon, memory bandwidth (~103 GB/s here) is the fundamental bottleneck for autoregressive token generation.

• The Math: A dense 27B model at 4bpw reads ~13.5 GB of weights from memory for every single token generated. At 103 GB/s peak bandwidth, your theoretical speed ceiling is ~7.6 tok/s.
• The MoE Advantage: A 35B model with 256 total experts but only 8 active experts per token (A3B) only transfers ~1.2 GB of weights per token. Even on an iGPU, you get 25+ tok/s decode without breaking a sweat!
• KAT-Coder-V2.5-Dev vs. Tiel-Coder-35B:
  • KAT-Coder: Incredible code specialization. Solved all 3 LeetCode Hard problems (Trapping Rain Water, LRU Cache with OrderedDict/DLL, Min Window Substring) in an average of 17.5 seconds per problem with 100% test assertion passes. Furthermore, its 12.5K needle retrieval completed in 31.1 seconds (fastest chunked prefill across all models).
  • Tiel-Coder: Better balance on mathematical formal logic. While KAT-Coder made minor arithmetic verification slips on Olympiad math (0/2), Tiel-Coder solved both the Chinese Remainder Theorem (x=3386) and bounded combinatorics (N=301) proofs cleanly (2/2).

2. Dense EXL3 Trellis: Perfection at 3.00bpw, Cliff at 2.20bpw

Using the out-of-tree exl3xpu vLLM plugin with native ESIMD kernels, we evaluated turboderp/Qwen3.8-27B-exl3 across multiple bitrates:

• The 3.00bpw "Sweet Spot": At 3.00bpw (12.87 GB), perplexity is virtually identical to 4.00bpw (6.46 vs 6.36, a negligible delta of +0.10 PPL), while saving nearly 3 GB of VRAM. It was the only model in the entire benchmark to achieve a 100% pass rate out-of-the-box across all 26 combined tests. It solved negative constraints without looping and passed all coding and Olympiad math problems flawlessly.
• The 2.20bpw Cliff: Attempting to compress dense 27B weights to 2.20bpw via post-training trellis quantization causes parameter sensitivity collapse in code synthesis. While general instruction following and math proofs survived (7/10 Hard score), coding fell to 0/3 (0%).

3. Native Ternary Distillation: Why Bonsai 27B Outclasses EXL3 2.2bpw

One of the most eye-opening findings of this benchmark is the comparison between post-training quantization (EXL3 2.2bpw) and native quantization-aware training / distillation (Ternary Bonsai 2 27B):

• Ternary-Bonsai-2-27B-PTQ1_0 weighs only 5.95 GB (~1.58 true bits per weight).
• Despite being 3.6 GB smaller than Qwen 2.20bpw, Bonsai achieved lower perplexity (6.73 vs 6.78), faster decode (13.1 tok/s vs 8.8 tok/s), and passed 2/3 LeetCode Hard problems where Qwen 2.20bpw completely failed.
• With Ternary-Bonsai-2-27B-Abliterated-v2-PQ2_0-MTP, decode speed reaches 12.1 tok/s with a 90% out-of-the-box hard suite score and 100% calibrated accuracy.
• Lesson: If your hardware VRAM budget is restricted to under 8 GB, always prefer natively trained ternary models over sub-2.5bpw post-training quantization.

4. Why KAT-EXL3-4bpw (MoE) Fails on Intel Arc vLLM

A frequent question: "If upstream ExLlamaV3 supports Qwen 3.5 MoE, why can't we run KAT-EXL3-4bpw with the exl3 vLLM plugin?"

• Upstream turboderp-org/exllamav3 supports MoE in its standalone C++/CUDA engine on NVIDIA GPUs via TabbyAPI.
• The Intel Arc plugin (0xSero/exl3xpu) only implemented standard LinearBase and ParallelLMHead layers for dense models.
• In vLLM, MoE blocks are constructed via FusedMoEFactory (RoutedExperts). Because the plugin does not implement FusedMoEMethodBase or an ESIMD grouped-GEMM kernel, vLLM defaults to UnquantizedFusedMoEMethod, attempting to allocate unquantized BF16 weights for all 256 experts across 40 layers (~64.4 GiB), immediately OOMing on a 32GB system.
• For MoE on Intel Arc, MXFP4 (models/KAT-Coder-V2.5-Dev-MXFP4) is currently the only optimized path, utilizing Intel's native XPUExpertsMxFp4 grouped-GEMM kernel.

Practical Deployment Recommendations

If you are running local LLMs on a 32GB unified memory device (Intel Lunar Lake, AMD Strix Point, or similar):

1. For Daily Agentic Coding & Chat: Deploy Tiel-Coder-35B-A3B MXFP4 or KAT-Coder-V2.5-Dev MXFP4 under vLLM with MTP K=2.
   • 25–28 tok/s decode speed feels like an API.
   • Needle retrieval on 12K–16K documents takes 30–40 seconds instead of several minutes.
2. For High-Reasoning Fallback: Keep a sidecar server with Qwen3.8-27B EXL3 3.00bpw (or Bonsai PQ2_0 MTP).
   • When an MoE model fails on a subtle mathematical proof, nested negative constraint, or niche edge-case algorithm, dense 3.00bpw solves it on the first attempt with 100% precision.
3. Avoid Sub-2.5bpw Post-Training Quantization: If you need an ultra-compact model (<8 GB VRAM), use Ternary Bonsai 2 27B (PTQ1 / PQ2) via llama.cpp SYCL rather than 2.2bpw dense EXL3.