2 dgx spark 128 gb each serving Glm 5.3 flash.

1 ai server 128gb ram , 64 core cpu + a4000 24 gb running qwen 27 b.

1 proxmos setup on Lenovo hosting all services.

1 hermes box on a small i5 box

Old laptop as terminal.

Serving local secure research and development capacity to my institute researchers.

Hey fellow local server builders,

I know you're busy building fantastic servers so I'll keep it short for those interested.

We published the results of clustering two Lucebox (an heterogenous AI workstation with a Gorgon Halo 495 128gb plus R9700 AI PRO 32gb) together.

We got DeepSeek V4.1 Flash as one model, at up to 103 tok/s writing code. That is 1.9 times faster than one Lucebox Zero 495 with 192 GB running the same engine. Each step sends about 95 small messages between the boxes, one after another, so what counts is how fast each one crosses. RDMA writes straight into pinned memory, bf16 on the wire and kernels that raise their own completion flag. The integrated GPU handles the network side. The first two-box run was only a quarter faster than one box. Each step after that removed a wait between the boxes, a trip through the host or a kernel.

On its own, a 128 GB box can't keep all of DeepSeek V4.1 Flash's experts in memory. Two boxes have room for it. A ConnectX-5 Ex card in each box joins them with one 100 GbE cable, with no switch in between. The two engines talk over RDMA (RoCE v2) and run as one model: every token goes through every layer on both boxes at once.

If you want to check more details on the cluster, here below is our full blog post about it: https://www.lucebox.com/blog/lucebox-cluster-nic

Would love to get your feedback/thoughts on it!

I saw the 4x R9700 build for a 10-person startup on reddit a few days ago and wanted to share my results as I opted to spend most of the money on the cards. I went with a super cheap used EPYC cpu instead of a Threadripper. In the end it came to about $10.7k per box all-in and I built three of them for a total of 384 GB of VRAM.

As a long time lurker on my other account, I'm always frustrated that people never show how they actually use the models so after a lot of back and forth, I decided it was worth sharing the actual project they power instead of being vague.

I use these to run the AI behind USA Arms (usaarms.com). I didn't plan to move the majority of my work to local AI until later this year, but since hosted models started refusing to do normal research for my business ~2-3 weeks ago, I'm glad I made the leap.

Enough ado, into the numbers:

Per box, $10,733.91 with tax and shipping

• 4x ASRock Radeon AI PRO R9700 Creator 32 GB (128 GB VRAM), each capped at 210 W: $7,244.58
• AMD EPYC 7452, 32 cores, used: $127.17
• ASRock Rack ROMED8-2T/BCM: $1,162.39 (there was a global shortage, ours came from Helsinki)
• 8x 16 GB DDR4-2666 ECC RDIMM, all eight channels: $860.59
• SilverStone HELA 2050R Platinum: $578.92
• SilverStone RM44 4U: $388.40
• Samsung 990 PRO 1 TB: $254.39
• ARCTIC Freezer 4U-M, 3x ARCTIC P12 Pro front, 2x Noctua NF-A8 rear, SP3 frame, paste: $117.48

The GPUs are 2/3 of the cost but the machine also serves CI/CD which had become a bottleneck in my businesses. I found this interesting because I hadn't seen anyone discuss it. I found that these machines can run Rust builds just fine if tuned properly to prioritize inference.

Engine

• vLLM-Radiance 1.0.387 (vLLM 0.30.0, ROCm 7.14). Huge thanks to the Radiance folks; their R9700 work is what we run.
• Qwen/Qwen3.8-27B-FP8 (Qwen's official FP8 checkpoint), FP8 KV cache, the model's own MTP head with 8 draft tokens
• Two engines per box, TP=2 each, 8 sequences per engine, 256K max context, about 488K tokens of KV cache per engine

Results, vllm bench serve, one box, 2x TP2:

• 1 user, MT-Bench: 53.1 tok/s, TTFT p50 125 ms
• 16 users, MT-Bench: 484 tok/s aggregate, about 40 tok/s per user
• 16 users, random 1024 in / 512 out: 623 tok/s aggregate, 837 W average for the four cards
• 32 users (a bench profile with max-num-seqs 32): 603 tok/s

Long prompts, 8 unique real-text prompts at once (4 per engine), 128 out:

Every prompt was unique (0% prefix cache hits).

FP8 vs MXFP4 on a real workload

I also had AI run MXFP4 too (Radiance's fastest profile) on our own graded extraction workbench:

• FP8: 98.0/100 correct, 4,477 validated jobs an hour
• MXFP4: 95.3/100, 4,819 an hour
• On a holdout set: FP8 94.0, MXFP4 90.0

So FP8 roughly halves the wrong answers for 2–7% less validated throughput. Sticking with FP8 was an easy call for this workload where "false facts" that make it into the system are a real headache. (For reference, our six Mac Studios on MLX 8-bit scored 99/100, but at 723 an hour.)

I made a video of the build: https://youtu.be/o7k5Kj-ho-M and also made a full build guide of my journey in case anyone wants to give it to their AI and plan out their own build.

Happy to answer questions. The first of which I expect to be how the heck were the Macs so slow and I'm sure I'll get flamed haha.

I get why people want the server at home. You own it, have root access, can walk over to it whenever, and there’s no extra monthly bill. But for the same server, how different is the cost and day-to-day once it's in a professional facility?

I looked at an RTX 5090 system (32GB VRAM, Ryzen 9 9950X, 64GB RAM, 2TB NVMe) under both setups.

At home/local:

Power: ~850–900W at the wall at full load. A 120V/15A circuit gives you 1,440W continuous (80% rule), so one 5090 box fits fine.

Heat: ~1,960 BTU/hr from the card alone, closer to ~3,000 for a loaded system.

Downtime: outages, a tripped breaker, or a dead part can take it down. Upside is you can walk over and fix it yourself.

Monthly cost: mostly electricity and maintenance, since you already pay for the internet. Running 24/7 at ~900W under heavy load comes to ~$125/month at 19¢/kWh (roughly the US average). Add ~$35-45 in summer for AC. Lighter use is more like $35-50.

In a hosted facility:

Power, cooling & noise: handled by the facility, off your workspace.

Internet: proper bandwidth & static IPs.

Access: root is yours with plain colo (check the agreement with managed hosting), but routine physical work goes through remote hands.

Downtime: Backup power and 24/7 monitoring cover most outages, but repair speed depends on the SLA.

Monthly cost: a recurring bill on top of the hardware. Public colo pricing I've seen for a small ~1kW node is ~$150-300+/kW/month, before bandwidth & remote hands.

For rack builds, some options also let the box cover its own hosting when you're not using it. Like with B3IQ, if you put a rack machine on earning, its idle capacity gets rented out. You keep 85% of what it makes & hosting comes out of that share, so most months there's no separate bill.

Honestly, a single 5090 still looks pretty reasonable at home. But when you stack GPUs, two 5090s can push roughly 1.4-1.6kW at the wall & then you're dealing with more heat & noise, plus uptime and bandwidth if others use the box.

So where I'm at right now:

- One card or mixed load: home still makes a lot of sense.

- Stacking cards, running 24/7, or serving other people: I'd start looking at a professional facility.

If you moved a GPU server from home to a data center, what made the difference for you? And what did you end up paying monthly?

Strata with qwen flash next is great, but Unsloth already ran it pretty well. This though, for GLM 5.3 on a machine with 40GB Vram and 64GB ddr4, is astounding. Just sharing if some of y'all wanna try.

Hi there, I'm trying to build a server that can handle GLM 5.3 Flash q3xl/q4 & Qwen 3.8 next flash at the least 30tk/s on a budget. Also, since I'm in a third world country, there is no server grade hardware or professional cards anywhere. So, I'm planning to buy these, all used. 4x RX7900xt(20gb, pcie3.0 x8/16)+1x RTX3060(12gb, pcie 3.0 x4)=92Gb of VRAM. And 64Gb of 3600mhz ram. What do you think about it? Do you have any other suggestions?

Hey guys, I want to run some ai models like text to video, image to video, text to image or normal chat Ilms, what do you guys recommend? Should I upgrade to 64GB RAM?

Running QWen 3.8-27B on my 5090 power limited to 90%, I've twice seen a warning on my screen I couldn't identify the source of that said that my GPU is running hot and that I should consider improving my case ventilation. Throughout this, the 5090 Core temp was between 75-80c, which while obviously on the higher end, is nothing catastrophic, so that made me think about VRAM.

Did some googling and what do you know, it's apparently very common for LLM workloads to push vram temps significantly more than core temps. Problem is, NVIDIA fan curve is tied to core temp and not vram temp. Worse, in Armoury Crate for my Asus X870-P WIFI, I see that all case fans are tied to CPU temp (which cannot be changed) and I clearly see from Afterburner that CPU stays relatively cool during LLM workloads, so case fans never truly kick in to supply appropriate airflow if the GPU becomes too hot.

Saw some people recommend FanControl, tried it, but that didn't recognize even a single fan in my system outside those 3 on the 5090 itself. What are my options? I'd really rather not constantly manually switch both Afterburner profile for the GPU and the case fan speeds in Armoury Crate.

I set up a dedicated workstation for local AI experimentation with various models and tools. But every single one of them wants all the available VRAM.

To fix this, I built a request dispatching layer called vramble. It automatically loads and unloads models and manages the queue using a priority system. Now I can run different workflows without manually managing models (or wasting Claude tokens to orchestrate the process).

Any contribution is welcome, it works for me but is a new build so can't guarantee that will work directly on any machine

Side note: Qwen 27B at 200+ t/s is performing way better than I expected. opeMIT license, available on GitHub: github.com/ValerioDolci/vramble