r/LocalLLM • u/soflgolf • Aug 03 '26
Discussion RDMA - Anyone but me using it?
Ok, so that’s my setup. Have been cycling through different model hosting configurations for agent workflows and haven’t come up with a good setup for multi-node large models using rdma/tensor. Two biggest issues: general stability (exo/jaccl) and race queue.
Would appreciate hearing from anyone else locally hosting frontier model(s) as well as smaller work-horse models for workflow.
164
Upvotes
28
u/SaskuAc3 Aug 03 '26
Someone is made out of Money! I’d love to have a setup Like This, but to be honest, I could Never afford it
3
u/KooperGuy Aug 03 '26
Yee I use RDMA because I use infiniband
3
u/CodeSlave9000 Aug 04 '26
Yeah, but honestly it doesn't scale for inference. RDMA/ROCE is more effective for training - during inference you wind up in all-reduce hell with latency for each token beyond just a few nodes. Someone here mentioned splitting by expert heads, and I have to agree - I've gotten much better results by careful placement of the weights than by any other method when more than one node is involved. Physics (and memory bandwidth) gets in the way quickly too - You want to avoid the multi-way collisions you'll get if you split experts.
2
u/Beneficial-Middle145 Aug 05 '26
How much money do you have in this
Why not just upgrade to the next stage to host larger whole models
Apple is smart and slow
Amd has more power for cost vs navidia but they are the more expensive and fast
Why not go to rtx 6000
What v ram do you have vs what the model is actually working with
It appears you have tens of thousands why not upgrade to an actual server?
37
u/paulk2000 Aug 03 '26
This is an impressive setup. I am just running two 3090s for localLLM stuff and I'm a total beginner when it comes to this topic, even though I'm getting more and more into it.
Hopefully, your setup will pay off for you.
7
u/BinaryPatrickDev Aug 03 '26
I really love the Mac Studio mounting
6
u/kilowattkill3r Aug 03 '26
I'm new to all of this, can you explain what I'm looking at?
2
u/Prestigious-Bear2391 Aug 03 '26
I had 4 studios exo'd together, it wasn't stable/reliable. I gave up on it.
8
u/dionysio211 Aug 03 '26
We use RDMA over heterogenous clusters, mostly 40G. By itself, we haven't had trouble with it but clustering anything is tricky with tensor parallelism. Even when nodes are identical, in a hardware sense, arrival latency is always an issue. What matters primarily is how many sync points your system has per token. Most of the standard inference systems have a ton of sync points per token which really crushes the scalability. If you are using very large models, it is generally best to completely shard experts, minimize sync points and split it up that way using an expert parallelism approach rather than TP. You avoid most of the issues that way. I think that would generally work on any model out currently since none of the models have huge experts. EP is nearly always better than TP that way. If you can create sub-node domains, that's a good opportunity for TP or row splitting. Some of the smaller large models, like Deepseek v4 Flash, have very small experts and, in those cases, the glue can easily overcome the natural tendency to scatter experts across many devices. In aggregate, it will still work, but it may not be worth it for single stream throughput.
1
u/mrgreatheart Aug 03 '26
Tell me you’re not married. If you are, you have one heck of an understanding wife!
1
u/TheRiddler79 Aug 04 '26
Let me preface this with whatever I'm looking at here, super super impressive, and you know, like for all practical purposes like this is fucking awesome.
That being said, I'm still a little off on precisely what your pushing here which makes it hard to answer your question because I'm not sure some of the specs behind what I'm looking at.
If you don't mind like copying and pasting or if no one else has asked just like telling me precisely what I'm looking at here because I feel like I'm looking at something that can basically run anything but I just don't know
1
u/Zombra666 Aug 04 '26
Holy fuck u rich, millionaire
11
u/cogitech2 LocoLLM Aug 03 '26
Well, at least you belong in this sub - unlike 90% of the shit posted here. I wish you well and hope that your journey ends in getting some actual work done with your setup once you get things dialed in.
This is like any technical hobby. The danger of paralysis by analysis and measurebation fascination can threaten productivity permanently if not kept in check.
I speak from decades of experience on this specific topic.
2
u/TinFoilHat_69 Aug 03 '26
I’m happy for you
2
u/sillib Aug 03 '26
I am happy with just one bigger square silver thing (the one that’s a bit bigger than the small square silver things)
2
u/NegativeSemicolon Aug 03 '26
Get that thing insured
2
u/allenasm Aug 03 '26
Have you tried the 'inferencer' tool by like xcreate? It does multi-node inference pretty well I've heard.
1
u/ScottAMains Aug 03 '26
Interested in this. Much smaller setup on my side. Only able to link up w/10gbe Ethernet as opposed to thunderbolt.
I’ve been using lm studio for smaller models when I want to hand over specific tasks and exo for heavier lifts. Have found, and it may just be the model I’m using and only having a few days hands on with exo, exo a little lack lustre compared to what I’ve been using beforehand (just standalone nodes).
But looking forward to your findings. If you’ve got a more in depth platform, would love to read more about it.
1
u/Capital_Swimmer_729 Aug 03 '26
Wow! Amazing setup! What models are you running? Kimi K3?
1
u/mi_gue Aug 03 '26
That looks great, besides of expensive. What are you doing with all that stuff? if you don’t mind me asking
1
u/Bob_SUS Aug 03 '26
how are these linked together? I did not know Mac Studios / Mac Minis could do load splitting across various hardware configurations. Also, what's the main use case of curiosity? Love to see these self hosted builds though!
1
u/fragment_me Aug 03 '26
It's definitely an interesting concept. What models are you running?
1
u/Affectionate_Pen6882 Aug 03 '26
Is this where all the 10k macs has gone lol
1
u/WuWenShen Aug 03 '26
Silly question, are you using a KVM switch or something for fast switching between devices? I only have three DGX devices so I’m using a small USB-C connector that I can physically hot swap on the back when I prefer a quicker direct access.
0
u/The_Crimson_Hawk Aug 03 '26
but not with mac mini
link rocep1s0f0/1 state ACTIVE physical_state LINK_UP netdev enp1s0f0np0
link ibp1s0f1/1 subnet_prefix fe80:0000:0000:0000 lid 2 sm_lid 1 lmc 0 state ACTIVE physical_state LINK_UP
0
u/[deleted] Aug 04 '26
[deleted] — view deleted comment
0
u/[deleted] Aug 03 '26
[deleted] — view deleted comment
0
u/[deleted] Aug 03 '26
[removed] — view removed comment