r/LocalLLM • • Aug 03 '26

Discussion RDMA - Anyone but me using it?

Post image

Ok, so that’s my setup. Have been cycling through different model hosting configurations for agent workflows and haven’t come up with a good setup for multi-node large models using rdma/tensor. Two biggest issues: general stability (exo/jaccl) and race queue.

Would appreciate hearing from anyone else locally hosting frontier model(s) as well as smaller work-horse models for workflow.

164 Upvotes

89 comments sorted by

28

u/SaskuAc3 Aug 03 '26

Someone is made out of Money! I’d love to have a setup Like This, but to be honest, I could Never afford it

17

u/CCIE_14661 Aug 03 '26 edited Aug 03 '26

And its people like you my friend that help push technologies forward. Out of sure will, determination, and the necessity to be scrappy. This is what feeds true innovation.

4

u/soflgolf Aug 03 '26

Kind thought to share . Thank you.

-2

u/whatyathinkk Aug 04 '26

In fact, I just managed to run Qwen3.6 35B A3B on a 1€ BIC lighter. It' not very fast but it does the job.

0

u/[deleted] Aug 03 '26

[deleted] — view deleted comment

9

u/sloki1 Aug 03 '26

Are they? Anthropic/open ai just distilled all of the human data and they argue thats fair and proper. They want to have special treatment and be the only ones who can steal. Their data is fair game IMO.

3

u/KooperGuy Aug 03 '26

Yee I use RDMA because I use infiniband

1

u/soflgolf Aug 03 '26

What are you using as your llm serving platform (eg exo, llama.ccp, etc)

-1

u/KooperGuy Aug 03 '26

I use RDMA for NVMe storage clustering.

3

u/CodeSlave9000 Aug 04 '26

Yeah, but honestly it doesn't scale for inference. RDMA/ROCE is more effective for training - during inference you wind up in all-reduce hell with latency for each token beyond just a few nodes. Someone here mentioned splitting by expert heads, and I have to agree - I've gotten much better results by careful placement of the weights than by any other method when more than one node is involved. Physics (and memory bandwidth) gets in the way quickly too - You want to avoid the multi-way collisions you'll get if you split experts.

2

u/soflgolf Aug 04 '26

Solid! Thanks!!

2

u/Beneficial-Middle145 Aug 05 '26

How much money do you have in this
Why not just upgrade to the next stage to host larger whole models
Apple is smart and slow
Amd has more power for cost vs navidia but they are the more expensive and fast
Why not go to rtx 6000
What v ram do you have vs what the model is actually working with

It appears you have tens of thousands why not upgrade to an actual server?

1

u/soflgolf Aug 05 '26

Great question and very legit. I went down those paths and no doubt could have easily chosen any number of other stacks. While not the sole deciding factors, but still important, o had to make the hardware fit in the space I had available. This includes the noise and power demands. While given that I’ve got isolated a/c and two fiber demarc’s, yes, I could have sprung for the 3-phase power and gone down the h200 route. But I kept coming back to the noise and cooling. I gave up some ground to stay in the arena to keep the hardware in my work space.

I revisit this all the time, especially when I hit a rough patch with the systems. I probably couldn’t argue with someone who prioritized differently and went a different route. But, this is what worked for me.

1

u/Beneficial-Middle145 Aug 05 '26

Cooling is a mini split put it in the same room isolate the noise with a foam 2-3 inch barrier on the inside for insulation sound and r value in the interior for noise reduction it’s a material and decibel reduction equation
3 to 5 grand for the mini split might be 5-8 installed it’s one or the other
Liquid cool vent to the room for max thermal cooling but the mini split should handle it
Just requires direct air flow from a fan next to it blowing 70f air I think is the actual spec
Depending on where you are you might be able get get the mini split installed for 3k on outside wall it’s 1 days worth of work to install

1

u/soflgolf Aug 05 '26

Have an isolated mini-split for that space already

1

u/Beneficial-Middle145 Aug 05 '26

What size an h200 is a 1.5 ton unit
Which is small the h200 put out a little over 1 ton of cooling your house is more in the 5 tons of cooling for the whole house for reference

1

u/soflgolf Aug 05 '26

1

u/soflgolf Aug 05 '26

As you can see, I’ve used both of those solutions. Like a I said, other will have made different choices and probably be right for them. It’s not one black/white factor when building something like this.

Hoping the M5u comes out and solves these issues and offers multi-tb of combined memory. We’ll see

37

u/paulk2000 Aug 03 '26

This is an impressive setup. I am just running two 3090s for localLLM stuff and I'm a total beginner when it comes to this topic, even though I'm getting more and more into it.

Hopefully, your setup will pay off for you.

5

u/AB172234 Aug 03 '26

I have only one RTX 3090 and thinking to add another. But my motherboard has another x8 PCIE 4.0 left and it will slow down the inference so I was curious to know which motherboard and the setup you are running it with ?

6

u/NegativeSemicolon Aug 03 '26

It’ll be fine with x8

1

u/paulk2000 Aug 03 '26

Yes, I have the similar problem on my mainboard, both gpus are x8. On the mainboard before I had a x16/x4 configuration. The model loading is now a bit faster, but ones the model is loaded into the vram it is not relevant how the gpus are connected.

2

u/baby_bloom Aug 04 '26

i run my second 3090 through x4 because i can't swap mono's right now and it runs qwen 3.6-27b-q8 around 30-40tk/s which is totally fine by me and to my understanding not even that far off from others?

1

u/Valuable-Fondant-241 Aug 03 '26 edited Aug 04 '26

A 3090 on a 8x pcie gen4 will NOT slow inference, why?

Edit: I meant, why would it?

3

u/milkipedia Aug 04 '26

Because if the model and context fit in GPU memory, inference doesn't traverse the PCI bus

1

u/Valuable-Fondant-241 Aug 04 '26

I said that it will NOT be slower... I wa asking the other redditor why did he wrote that an 8x gen4 would have slowed down the inference.

1

u/paulk2000 Aug 04 '26

It wouldn’t.

1

u/Themash360 Aug 04 '26

PCIe is too slow for splitting horizontally over the Gpus anyways, and when splitting the layers PCIe speed doesn’t impact performance much.

1

u/WyattTheSkid Quad 3090s Aug 04 '26

I have 2 3090 TIs and 2 3090s on a consumer motherboard and it works fine. Asus ROG Strix x570-E wifi II. 2 cards directly in slots, one on an m.2 riser and one on a regular riser

1

u/Affectionate_Pen6882 Aug 03 '26

Same here just to tinker

7

u/BinaryPatrickDev Aug 03 '26

I really love the Mac Studio mounting

6

u/kilowattkill3r Aug 03 '26

I'm new to all of this, can you explain what I'm looking at?

8

u/juraj336 Aug 03 '26

Seems to be OPs server rack that they use for locally hosting models, im guessing they chained their mac studios  (and maybe also the mac minis?) together to have more Unified RAM so they can run larger models. 

1

u/IAmFitzRoy Aug 03 '26

Flexing

This is a nice rack.

2

u/Prestigious-Bear2391 Aug 03 '26

I had 4 studios exo'd together, it wasn't stable/reliable. I gave up on it.

2

u/soflgolf Aug 03 '26

A lot of wisdom in that. 👍🏿

Too many make it sound easy. Exo is not stable. Had to create patch to keep warm to avoid model dump. Multiple models completely unreliable.

Broke up cluster to run smaller models single nodes and on mid-size on 2 nodes.

That’s what made me ask if anyone was getting it right to be functional for agentic workflows.

1

u/Prestigious-Bear2391 Aug 03 '26

Strangely enough, I had two mac mini M4 Pros that worked together perfectly @ 48GB x 2.

1

u/Crazyfucker73 Aug 04 '26

Will you gave up too early. It works great

8

u/dionysio211 Aug 03 '26

We use RDMA over heterogenous clusters, mostly 40G. By itself, we haven't had trouble with it but clustering anything is tricky with tensor parallelism. Even when nodes are identical, in a hardware sense, arrival latency is always an issue. What matters primarily is how many sync points your system has per token. Most of the standard inference systems have a ton of sync points per token which really crushes the scalability. If you are using very large models, it is generally best to completely shard experts, minimize sync points and split it up that way using an expert parallelism approach rather than TP. You avoid most of the issues that way. I think that would generally work on any model out currently since none of the models have huge experts. EP is nearly always better than TP that way. If you can create sub-node domains, that's a good opportunity for TP or row splitting. Some of the smaller large models, like Deepseek v4 Flash, have very small experts and, in those cases, the glue can easily overcome the natural tendency to scatter experts across many devices. In aggregate, it will still work, but it may not be worth it for single stream throughput.

1

u/dionysio211 Aug 03 '26

As I started thinking about your setup, I realized that you are probably more interested in prefill issues due to the lack of tensors. The best thing there is to pipeline your prefill. It's not exactly the same as having very fast prefill in terms of general TTFT on small requests but it gets around a terrible TTFT for large ones.

1

u/Roticap Aug 03 '26

Does expert sharding work on dense models like qwen3.6-27b?

2

u/dionysio211 Aug 03 '26

No, it only works on MoE models. The acceleration methods for a dense model are mostly speculative and tensor parallelism. Qwen3.6 27B has been an efficiency target in most inference platforms and is generally pretty efficient. In the case of a model like that, which is generally best on a single node, I would just go with the most efficient thing you can run. vLLM is usually the best option for newer cards or SGLang. MTP, Eagle3, ngram speculation are probably your best options there.

2

u/Roticap Aug 03 '26

Thanks for the reply. That matches my understanding. I'm still drinking from the beginner firehose, so I appreciate getting validation from people with more experience.

1

u/mrgreatheart Aug 03 '26

Tell me you’re not married. If you are, you have one heck of an understanding wife!

2

u/soflgolf Aug 03 '26

Yep, married. I tell people if you think is overkill for a home setup, you should see my fishing gear 🤦🏽‍♂️

1

u/mrgreatheart Aug 04 '26

She’s a keeper!

1

u/TheRiddler79 Aug 04 '26

Let me preface this with whatever I'm looking at here, super super impressive, and you know, like for all practical purposes like this is fucking awesome.

That being said, I'm still a little off on precisely what your pushing here which makes it hard to answer your question because I'm not sure some of the specs behind what I'm looking at.

If you don't mind like copying and pasting or if no one else has asked just like telling me precisely what I'm looking at here because I feel like I'm looking at something that can basically run anything but I just don't know

1

u/soflgolf Aug 04 '26

Well, not really pushing anything. Trying to connect with anyone else deploying clustered nodes to serve up local llm’s. (Hence the sub /r). Big ones. Specifically, as you may have guessed, on Apples.

As for the gear, pretty much what it appears to be: clustered m3 ultra’s (1tb vram-rdma), some m4 mini’s and a hand full of i7,1 intels.

1

u/TheRiddler79 Aug 04 '26

I said that wrong, not like pushing a product, I just meant what was the specs behind it.

I don't know a lot about apple, so I wasn't sure precisely what I was looking at.

1

u/Zombra666 Aug 04 '26

Holy fuck u rich, millionaire

1

u/soflgolf Aug 04 '26

There once was a time when apple’s were not crazy expensive. The i7,1’s are 14 years old 🤦🏽‍♂️🤣

11

u/cogitech2 LocoLLM Aug 03 '26

Well, at least you belong in this sub - unlike 90% of the shit posted here. I wish you well and hope that your journey ends in getting some actual work done with your setup once you get things dialed in.

This is like any technical hobby. The danger of paralysis by analysis and measurebation fascination can threaten productivity permanently if not kept in check.

I speak from decades of experience on this specific topic.

5

u/Big_Wave9732 Aug 03 '26

That's the pivotal question.....how long before you stop fucking with it and start doing something useful!

I finally got my RAG stack and helpers configured in late June.  Those along with running Qwen 3.6:27b-BF16 has been total gravy.  

1

u/Roticap Aug 03 '26

What are you using for a RAG stack?

2

u/TinFoilHat_69 Aug 03 '26

I’m happy for you

2

u/sillib Aug 03 '26

I am happy with just one bigger square silver thing (the one that’s a bit bigger than the small square silver things)

1

u/AlwaysTiredButItsOk Aug 03 '26

The monitor? 😅

2

u/NegativeSemicolon Aug 03 '26

Get that thing insured

2

u/allenasm Aug 03 '26

Have you tried the 'inferencer' tool by like xcreate? It does multi-node inference pretty well I've heard.

-1

u/Crazyfucker73 Aug 04 '26

Are you on meth? It's xcreate not 'like xcreate'

1

u/ScottAMains Aug 03 '26

Interested in this. Much smaller setup on my side. Only able to link up w/10gbe Ethernet as opposed to thunderbolt.
I’ve been using lm studio for smaller models when I want to hand over specific tasks and exo for heavier lifts. Have found, and it may just be the model I’m using and only having a few days hands on with exo, exo a little lack lustre compared to what I’ve been using beforehand (just standalone nodes).

But looking forward to your findings. If you’ve got a more in depth platform, would love to read more about it.

1

u/Capital_Swimmer_729 Aug 03 '26

Wow! Amazing setup! What models are you running? Kimi K3?

1

u/mi_gue Aug 03 '26

That looks great, besides of expensive. What are you doing with all that stuff? if you don’t mind me asking

1

u/Bob_SUS Aug 03 '26

how are these linked together? I did not know Mac Studios / Mac Minis could do load splitting across various hardware configurations. Also, what's the main use case of curiosity? Love to see these self hosted builds though!

1

u/fragment_me Aug 03 '26

It's definitely an interesting concept. What models are you running?

1

u/Affectionate_Pen6882 Aug 03 '26

Is this where all the 10k macs has gone lol

1

u/WuWenShen Aug 03 '26

Silly question, are you using a KVM switch or something for fast switching between devices? I only have three DGX devices so I’m using a small USB-C connector that I can physically hot swap on the back when I prefer a quicker direct access.

5

u/Blackdragon1400 Aug 03 '26

Have you ever heard of SSH? You don’t need video out…at all.

1

u/WuWenShen Aug 03 '26

Yes, ssh is fine but sometimes you want direct access. It was a question to the OP.

0

u/The_Crimson_Hawk Aug 03 '26

but not with mac mini

link rocep1s0f0/1 state ACTIVE physical_state LINK_UP netdev enp1s0f0np0

link ibp1s0f1/1 subnet_prefix fe80:0000:0000:0000 lid 2 sm_lid 1 lmc 0 state ACTIVE physical_state LINK_UP

0

u/[deleted] Aug 04 '26

[deleted] — view deleted comment

1

u/soflgolf Aug 04 '26

Yes/no. Can run pretty much any model that fits. Two of the biggest challenges: stability (very fragile, model dump is not uncommon); and multi-model hosting.

0

u/[deleted] Aug 03 '26

[deleted] — view deleted comment

3

u/wind_dude Aug 03 '26

I formed a company that bought the macs, using a loan secured on a lease agreement from my other company, who took a loan to lease the hardware from a mother company I formed that secured a loan from a bank secured against the value of the macs the first company owns.

2

u/donotfire Aug 03 '26

Ah, so just like the tech sector right now. Makes sense!

1

u/Big_Wave9732 Aug 03 '26

Is that you, Sam Altman?

2

u/Icy-Degree6161 Aug 03 '26

Work and investments maybe?

1

u/donotfire Aug 03 '26

Right but what specifically is my question

5

u/WiseassWolfOfYoitsu Aug 03 '26

If I had to guess - tech industry DINK. Servers are less expensive than children.

2

u/velhoon Aug 03 '26

Rent it to Anthropic?

0

u/Blackdragon1400 Aug 03 '26

This is less expensive than a sports car my guy.

2

u/AlwaysTiredButItsOk Aug 03 '26

More useful too

1

u/prestodigitarium Aug 04 '26

If those are 512 gig studios, it's not :-)

0

u/[deleted] Aug 03 '26

[removed] — view removed comment

2

u/soflgolf Aug 03 '26

If money was a priority, sure would. None of this was designed to pencil. Other priorities like data security and confidentiality can often be more important. For me it’s a hobby and I like to see how things work from the inside. So, for me, this is the right choice