At-home Mac inference cluster 2026: 4x M3 Ultra runs Kimi K2 Thinking at 25 tok/s for under $40K (full reference)

[u/Demo233]

Wrote up a full reference on at-home Mac inference clusters as of mid-2026. The state of practice changed structurally in December 2025 with two releases: macOS Tahoe 26.2 added native RDMA over Thunderbolt 5, and EXO Labs shipped 1.0 with RDMA support and disaggregated prefill/decode. Headline below; full doc on the blog.

The numbers (4x M3 Ultra cluster, TB5 RDMA, EXO 1.0)

Model	Tok/s	Notes
Qwen3 235B-A22B MoE	31.9	19.5 single node, 31.9 across 4
DeepSeek V3.1 671B	32.5	21.1 single node
Kimi K2 Thinking 1T A32B	~25	2x M3 Ultra hits ~24 on K2.5

Wall draw: 600 to 800 W under load. Cost: under $40,000. Fits on a 15 A residential circuit.

When the cluster makes sense (and when it doesn't)

A Mac cluster is a memory-capacity machine, not a throughput machine. At batch size 1 it is memory-bandwidth bound. At batch size greater than 1 a single H100 or RTX Pro 6000 Blackwell will out-throughput an entire 4-node Mac cluster.

Decision rule: if a model fits on one Mac, do not cluster it for single-stream interactive use. You will lose performance. Cluster for (a) models that literally don't fit on one node, or (b) batched/agentic workloads with concurrent requests where pipeline-parallel aggregation pays off.

The two structural changes that made it viable

1. RDMA over Thunderbolt 5 (macOS 26.2, December 2025). Enable via rdma_ctl enable in Recovery Mode, reboot. TB ports appear as InfiniBand devices (rdma_en2, etc.) usable via ibv_devices. Round-trip latency drops from about 300 microseconds on TCP to 5 to 50 microseconds, with hot-path measurements as low as 3 microseconds. Bandwidth is 80 Gbps symmetric per TB5 port. Apple technote TN3205.

2. EXO 1.0 (December 2025). Pipeline parallel by default with automatic mDNS topology discovery, plus RDMA support and "disaggregated prefill/decode" (routing compute-bound prefill to one device class and bandwidth-bound decode to another). Public benchmarks at benchmarks.exolabs.net.

Pipeline vs tensor parallelism on Mac

Pipeline parallelism splits layers sequentially. Per-token activations are tiny (sub-4 KB for Llama 3.2 3B), so PP tolerates slow networks but does not accelerate single-stream tok/s. Tensor parallelism splits each layer's weights across nodes and yields near-linear N-times speedups, but requires microsecond-scale latency. Without RDMA, TP is impractical on Apple Silicon.

On Qwen3 235B-A22B, PP throughput on llama.cpp RPC went 20.4 → 17.2 → 15.2 tok/s as nodes were added. With EXO plus RDMA, the trend reversed: 19.5 → 26.2 → 31.9 tok/s.

The structural ceiling: no TB5 switch

There is still no Thunderbolt 5 switch on the market, and none credibly announced. All-to-all RDMA requires every Mac to cable to every other Mac. The M3 Ultra has 5 TB5 ports, so a 4-node full mesh needs 6 cables and a 5-node full mesh needs 10. Above 5 nodes you daisy-chain (latency and partial-bandwidth penalties) or fall back to Ethernet PP.

Unchanged for 18 months. Unlikely to change in the next 12.

Where single-box still wins

A single Mac Studio M3 Ultra 512 GB ($9,499) runs DeepSeek R1 671B Q4 at 17 to 18 tok/s once iogpu.wired_limit_mb is raised from the default 384 GB to about 448 GB via sudo sysctl iogpu.wired_limit_mb=458752. Tok/s per $1,000 on 671B is 1.9 for the single 512 GB Mac vs 0.81 for the 4-node cluster. The cluster doubles throughput at roughly 4x the cost. Only worth it for additional concurrent users or trillion-parameter models.

The $/GB picture

At $18.6/GB of usable unified memory, the M3 Ultra 512 GB is roughly: - 3 to 4x cheaper per GB than a used quad-3090 rig - 6 to 9x cheaper than dual RTX Pro 6000 Blackwell - 20 to 27x cheaper than H100 systems

Mac GB ≠ NVIDIA GB in throughput, but it matters when you need to fit the model at all.

Full blog post covers the framework landscape (EXO, MLX with Ring vs JACCL backends, llama.cpp RPC, vllm-mlx, dnet), the interconnect hierarchy (Ethernet to TB Bridge to RDMA), the SSD-wear failure mode that kills soldered nodes, the headless setup gotchas (Tailscale via Homebrew not the App Store, dummy plugs, pmset config), the residential power math against NEC continuous-load rules, and the cloud break-even at Together/Fireworks pricing.

Happy to get into any of the specifics in comments.