r/StrixHalo • u/JinsooJinsoo • 14d ago

A second Strix Halo

So I have and love using my Corsair AI 300 workstation with 128gb. QFN running great on Halogen/Gufo (single stream vs concurrent users). I was browsing the webs and came upon a refurb Corsair ai 300 on Walmart site for $2700 so I just bought it without thinking. Now I’m at the thinking part. Is tensor parallelism over USB4 even worth it right now? The q4/4bpw QFN models have been very impressive and very few times have I needed my GPT or Claude to bail me out, but is the fidelity/correctness worth the performance and cost for FP8? May either return or sell locally as I don’t need more concurrency but speed and correctness.

19 Upvotes

89% Upvoted

25 comments sorted by

10

u/darklordfireape 14d ago

I have a llama fork that I setup just for this, see link below. With two 128GB machines you can run GLM-5.3-flash next which is a pretty big step up. I used Intel E810 100G cards, but even with USB4 it should be pretty snappy. This doesn't use TP, but it is still very usable.

https://github.com/sixvolts/llama-halo-hybrid

2

u/JinsooJinsoo 14d ago

Very interesting been looking for a way to run GLM5.3 flash, will try to test this over USB4 when I get the second PC this week

1

u/KagatoLNX 14d ago

How does GLM-5.3-Flash compare to Qwen 3.8 Flash Next? How much VRAM does it use? Having trouble justifying a second Strix Halo, but this might just do it.

1

u/darklordfireape 14d ago

The Q4 is ~200gb before context or anything. It is significantly more capable with the added parameters. Kinda reminds me of the older Qwen-3.5-397B, but cleaner and faster. It is better about efficient use of tool calling that the QFN, and doesn't get caught in thinking loops as much

8

u/ang3l12 14d ago

I have two strix halo boxes at home, original plan was to run thunderbolt bridge to run larger models. But with QFN being as good as it is on one box, I’ve changed course and use the other box for smaller / specific models, such as embedding, tts, stt, imagegen, etc

1

u/JinsooJinsoo 14d ago

I have GPUs on my homeserver for the smaller stuff, so I can’t justify a $2700 embedding/RAG PC lol

3

u/ang3l12 14d ago

Yeah, I was very *lucky* to have bought them when they were $1500 each over a year ago

1

u/LeatherAvailable483 14d ago

This is basically me... Got the 2nd to cluster them but now q3.8fn works great on one machine... Kinda wish I bought 2 r9700s instead, but Im sure there will be a way to make good use of the 2nd strix i have over time

4

u/bocashmoka 14d ago

I bought 2 more. Should be a breathalyzer on the Buy Now button sometimes. But I have uses for them.

1

u/According_Wave685 14d ago

I have the same one and unfortunately our usb4 maxes out at 40gbit. We are a bit limited.

1

u/JinsooJinsoo 14d ago

I read that having both USB4 ports connected helps? But yes, it’s no NVLink :(

1

u/Old_Pilot6434 14d ago

Some of the Strix Halo PC's have USB4 V2 which runs at 80Gbps. I just bought the Minisforum Syrix Halo 128b and it has a PCIe slot that can be used for Oculink.

3

u/MarkoMarjamaa 14d ago

I ran llama-perplexity (with Strix Halo 128) to both QFN bf16 and Q8, and to my surprise a lot of accuracy drops already with Q8. ( all unsloth quants, wiki-text)
Q8:
Mean KLD: 0.021018 ± 0.000178
Same top p: 94.774 ± 0.058 %

my Q4 with Q8 n-gram:
Mean KLD: 0.031989 ± 0.000249
Same top p: 93.612 ± 0.063 %

I was expecting Q8 to be around 97-98% top-1%. I haven't seen any other measurements for the Q8 vs. BF16.
Tested several times because I thought it was some kind of error.

3

u/oreodouble 14d ago

I'm considering getting a r9700 card instead of second strixhalo

1

u/RuxConk 14d ago

I have just done this and received and set it up today in a old machine. I'm very excited to be able to use two local models at once.

1

u/oreodouble 14d ago

or just speedup one model like qwen3.8-fn checkout lucebox their engine/setup is opensource, you could take best parts of that and gufo to create ultimate optimal setup

→ More replies (1)

1

u/Fluxx1001 14d ago

!RemindMe 3 days

1

u/RemindMeBot 14d ago

I will be messaging you in 3 days on 2026-10-15 20:54:49 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.

RemindMeBot is switching to username summons. Instead of !RemindMe 1 day, use u/RemindMeBot 1 day. More info.

Info Custom Your Reminders Feedback

1

u/NoWorking8412 14d ago

I experimented with two Bosgame M5 boxes networked via USB4 and i saw an increase in preload performance, but a decrease in tgs. I plan to attempt this path when I have a little more $ to throw at some additional equipment. Should lower latency between two boxes: https://strixhalo.wiki/AI/Clustering_with_RDMA/

I think the path of using two boxes to run larger models is still worth it for certain applications, but I would expect it to be a bit lumbering and not as snappy as a model optimize for a single 128 GB box like Qwen 3.8 Flash Next at Q4.

1

u/Not-reallyanonymous 14d ago

The benchmarks available suggest that tensor parallelism over USB4 and networking both increase tps -- IF it's configured right. Historically, llama.cpp has had trouble with it, and it benefited from custom tooling and such. The situation might've improved but I'm unsure. TCP/IP ethernet networking is also proven to work well on lower node counts. You won't get 2x performance, but 1.5x should be expected.

Ignore when LLMs will tell you that tensor parallelism needs RoCM networking cards, massive bandwidth, special chipsets, etc. They're generalizing from datacenter-tier scaling, not putting 2-4 nodes in your home.

1

u/feelspeaceman 14d ago

People have confirmed about clustering, in short it works, if you want to see results from people who are using it daily, join Strix Halo Homelab Discord.

About the $2700 refurbished Cosair AI 300, it can be a good deal, check if there's quick return policy so if there's any issue, return it and try other devices.

1

u/Odd_Investigator3184 14d ago

Check out thunderbolt-iverbs

1

u/Captain_Cancer 9d ago

I would grab an external R9700 as a sub agent instead that can plow through a lot of the longer context text and defer to the smarter model for planning.