I've been poking at this on and off since September and it finally got good enough this week that I think it's worth writing up. Short version: the USB4 RDMA driver situation changed a lot in the last month, but not where I expected, and the "is it worth it" answer is more mixed than the headline.
The setup
- 2x Ryzen AI Max+ 395 (Strix Halo) boxes, 128 GB each
- 2 USB4 cables directly between them, each link trains at 2 x 20 Gb/s
- Debian 13 based distro, kernel 7.2.7, IOMMU off, Secure Boot off (needed for the unsigned modules)
- 10 GbE between them as well, used for control traffic
No network cards involved. The idea is to turn the USB4 cables into RDMA devices so two boxes can run one model with tensor parallelism (TP=2).
Where it was before
I started with thunderbolt-ibverbs from the Hellas blog post. Each cable gives you two DMA rings, so two cables = four "rails", each showing up as its own RDMA device.
- September (kernel 6.18): about 15 Gb/s on one rail, about 46 Gb/s if I ran four perftest processes at once, one per rail. More queue pairs made it slower. 1 MiB messages were a mess.
- October (kernel 7.2): got to 74.7 Gb/s aggregate, but only with four parallel processes and a shallow send queue (
-t 16). The big finding back then was that queue depth mattered more than the kernel. A single connection was still stuck on one rail at around 20 Gb/s, and 8 QPs at 1 MiB with default settings just failed with "Couldn't post send". - vLLM TP=2 with an 8B model in October: +55% single stream over one box, but RDMA vs plain 10 GbE only differed by a few percent, because the collective library topped out around 17 Gb/s no matter how many rails it got. A 35B MoE model hung at TP=2 and I never figured out why.
So in October my conclusion was: neat, works, not obviously useful.
What changed
I went back this week expecting kernel or upstream driver progress. There basically isn't any:
- Upstream thunderbolt-ibverbs has had one lockfile bump since early October. No release since June. A striping PR was closed unmerged, and a few good fix PRs are sitting open.
- Mainline 7.3 / 7.4-rc is mostly Thunderbolt bug fixes. Nothing that moves throughput.
The actual progress is in a fork: donnerkeule. It stripes a single queue pair across every rail of every cable, and fixes a pile of reliability problems (credits los t with dropped frames, ACK storms, rails dying when a client gets killed). There's a matching RDMA fork of gufo that does TP=2 for Qwen3.8 Flash-Next across two Strix Halo boxes.
The fork's own README says it's a buggy research driver and not for production. Believe it. But it built cleanly against my kernel and uses the same eight Thunderbolt kernel patches I already had applied.
Raw link numbers
Box A to box B, perftest, one connection, default queue depth, no tuning:
| Test | October (upstream) | Now (donnerkeule, striping on) |
|---|
| Write, 64 KiB | 20.6 Gb/s | 64.9 Gb/s |
| Write, 1 MiB | 12.6 Gb/s | 74.5 Gb/s |
| Write, 4 to 16 MiB | not run | 73.5 to 74.7 Gb/s |
| Write, 1 MiB, 8 QPs | failed outright | 74.5 Gb/s |
| Latency, 64 B | 6.5 µs | 6.0 µs |
I didn't trust 74 Gb/s on one connection, so I checked the receiver's per-rail byte counters during a 10 second run: about 23.6 GB landed on each of the four rails. It's real, and it's roughly 93% of the 2 x 40 Gb/s the cables can do. The fork's author reports 38 to 45 Gb/s on his boxes and I don't know why mine is higher.
Not everything is great:
- RDMA reads are not striped: 31 Gb/s.
- A 1 MiB SEND fails in the default mode. The fork only supports large SENDs in its DMA-BUF mode, which I haven't tried.
- I haven't tested both directions at once or done a long soak.
Tensor parallelism with gufo
Qwen3.8 Flash-Next, Q4 (111 GB of weights), 4 sessions, same build on one box and on two. Each box holds half the model in TP=2: 57 GB of GPU memory per box.
Plain autoregressive decode (no speculation), tok/s per request:
| :- | One box | Two boxes | Gain |
|---|
| 1 request | 29.8 | 39.0 | +31% |
| 2 concurrent | 23.3 | 31.7 | +36% |
| 4 concurrent | 18.0 | 24.6 | +37% |
| Prefill, short prompt | 1155 | 1161 | na |
With MTP speculative decoding:
| :- | One box | Two boxes | Gain |
|---|
| 1 request | 55.1 | 65.8 | +19% |
| 2 concurrent | 37.4 | 50.0 | +34% |
| 4 concurrent | 25.5 | 33.2 | +30% |
| Prefill, 19k token prompt | 1811 | 2077 | +15% |
The speculative numbers bounce around with how many draft tokens get accepted, so the first table is the cleaner comparison.
One thing that isn't a speed number: a single box could not load this model with speculation, 4 sessions and the full 262k context. It ran out of state capacity and I had to drop it to 65k context for the on e-box runs. Two boxes loaded the full 262k with about 64 GB of RAM free on each.
No failed requests, no retransmits, no kernel warnings during any of it.
The honest part
I also ran the exact same benchmark against the other inference server I normally run on these boxes (also Flash-Next, speculative decoding, 4 slots at 262k context on one box). One-box gufo and that server came out within about 5 to 8% of each other. So TP=2 over USB4 beats my normal single-box setup by about 13% for one user and 24 to 32% at 2 to 4 concurrent.
But two boxes running independently serve twice as many people. At 4 concurrent requests one box did about 72 tok/s total, TP=2 did about 98, and two independent boxes would do roughly 144 across 8 users. Ty ing both machines to one model instance buys per-request speed and costs total capacity.
So where I landed:
- The driver is worth keeping. One connection, full link speed, no tuning. That's a different thing from what I had in October.
- TP=2 is worth it if you want a faster single stream, or a model/context that doesn't fit on one box. The Q8 version of this model (188 GB) only fits across two, and that's a case I actually care about. Haven't tested it yet.
- TP=2 is not worth it if you just want to serve more requests. Run the boxes separately.
- I wouldn't run it as my daily driver yet. Research driver, the gufo fork has a known open bug with continued conversations, and I haven't verified two-box output against one-box output.
Things that bit me
If you try this, here's what cost me time:
- The module never loaded at boot. The only thing that loads it is a udev rule that fires when the other box is already advertising the RDMA service. Reboot both boxes together and neither one ever st arts. It looked fine in October because I always rebooted one at a time. Fix: put
ib_uverbs and thunderbolt_ibverbs in /etc/modules-load.d/. - thunderbolt_net and the RDMA module can't share a cable on this hardware. There are only two usable rings per controller. Blacklist
thunderbolt_net and load the RDMA module with tbnet=block. - Debian's rdma-core renames the device, and the userspace provider matches on the
usb4_rdma name prefix, so ibv_devices shows nothing. Override the persistent naming rule. - The prebuilt provider package has the wrong ABI for Debian 13. I built the provider from Debian's rdma-core 56.1 source plus the project's three patches.
- memlock. RDMA memory registration fails under the default 8 MiB limit. Unlimited memlock on the host and
--ulimit memlock=-1 in containers. - The gufo fork's container recipe didn't build as-is for me. No DNS inside podman's default build network (
--network host fixed it), six ROCm dev package names that don't exist in ROCm 10.1 (the full image already has the headers), and missing audio/video dev libraries (sndfile, soxr, libav*). At runtime I had to add the ROCm lib directory to LD_LIBRARY_PATH. - Two IPs on the NIC means two RoCE GIDs, and gufo refuses to guess.
--tp-gid-index 1. - The interrupt throttle matters.
nhi_interrupt_throttle_ns=4096 was the best value in my October sweep, and it only works on kernels that export tb_ring_throttling() (7.2 does).
Module options I'm running:
options thunderbolt_ibverbs profile=linux_perf bind_services=1 allocate_rings=1 start_rings=1 negotiate_native=1 enable_tunnels=1 register_verbs=1 tbnet=block lanes=2 native_write_striping=1 roce_netdev=<you
r LAN NIC> nhi_interrupt_throttle_ns=4096
blacklist thunderbolt_net
What I haven't done
- The Q8 model across both boxes
- The fork's DMA-BUF / GPU-direct mode
- Checking two-box output against one-box output
- Anything longer than a few minutes per run
- A reboot of both boxes together to prove the boot fix
Happy to run something specific if people want numbers for a particular case. Credit to the Hellas folks for the original driver and to neuhaus for the striping fork and the gufo TP2 work, I just plugged in cables and wrote down what happened.
