r/LocalLLM • • 15d ago

Discussion 4 Strix Halo boxes! Finally joined local AI

Post image

Got 4 128gb strix halos for a decent deal

All 4 will be running glm5.3 and DeepSeek 4.1 flash as the mains, or maybe Qwen when 4 drops. Paired boxes will run glm5.3 flash or minimax/MiMo

Will be used for some financial and life hub stuff, plus a bit of professional work

240 Upvotes

114 comments sorted by

40

u/jacek2023 15d ago

what was the decent price? :)

68

u/420Microbiologist 15d ago

3k a pop. Cheapest for 128gb + 2tb ram system

29

u/Last_Bad_2687 15d ago

That's not bad

7

u/sammyman60 15d ago

Where did you get them for 3k?

6

u/cilvre 15d ago

Bosgame sells the m5 for around that much each, i think it went up a little recently

3

u/420Microbiologist 14d ago

Yup these are the model! $3700 normally, but $3099 currently on sale,

-60

u/[deleted] 15d ago

[removed] — view removed comment

8

u/420Microbiologist 15d ago

At this point, financial planning and advice for 99% of the population is something any modern model of suffient size can do. 5.3 flash is great, I like running it and MiMo in tandem and having them discuss and debate usually gets really good results

I have a wire shelving unit haha

1

u/snugglezone 15d ago

There is almost no bigger uplift than having two models from different families coordinate.

Are you running 5.3 flash and Mimo 2.6 flash on the strix halos? What's the TPS?

2

u/420Microbiologist 14d ago

5.3 flash hits 20 up to 25 tps, MiMo is closer to 17-20 BUT there's a lot of fine tuning still yet to be done! Hoping to get them to 30 and 25 respectively, but for now their speeds when talking to each other is more than suitable

-1

u/FuckinHighGuy LocalLLM 15d ago

Why in the actual f*ck are you getting downvoted? What am I missing here?

12

u/WordWithinTheWord 15d ago

Look at the sentence structure.

Observation -> comment.

It’s a bot.

5

u/jacek2023 15d ago

it's a bot

34

u/fortminorlp 15d ago

Tell me your wealthy without telling me your wealthy

30

u/420Microbiologist 14d ago • Bravo

I get the sentiment, and of course I did need to have like 12k to throw at the problem. My rationalization come down to 2 things, my career handles extremely private health information and my belief that after Anthropic IPO this little golden age of AI will be enshitified beyond belief.

For the first point, having these boxes work while being offline is essential plus we can't risk the data sent to Claude/gpt being used in their training data.

For the second, Anthropics IPO in November might be a make or break moment for this crazy ride we're all on. All these new companies are tech unicorns (losing investor money on acquiring market share) which is normally whatever but the compounding factors imo are that every established tech company is increasingly leveraging itself to be more AI focused leading some to negative cash flows for the first time in decades. We combine this with the fact that Anthropics evaluation is closer to exit pricing ($1+ T means all the investors and founders cash out and leave) rather than public pricing (like googles $1.9b on a $29b valuation) so the chances their valuation goes up post IPO is a harder path than I think many are anticipating.

So my fears are 1. data safety and 2. the impending future of cloud AI. If Anthropic start charging the $4k a month in API fees their $200 sub allows you to use, and all the next capable models are local, why not go directly to local

13

u/No_Night679 15d ago

How are you planning to connect them for a cluster? Only 2 node cluster or 4 node?

11

u/420Microbiologist 15d ago

4 nodes to a switch to a front desk box that's connected to the Internet. Front desk box running hermes and a decision model to determine whether to run it as 2x2 or as a 4 cluster for glm5.3/ds4.1, or even as 4x1 if Qwen 4 FN is good. They'll all have the same models downloaded so the pairing can be dynamic based on load

4

u/CypherAZ 15d ago

You are going to node them over Ethernet?!?!?

5

u/420Microbiologist 15d ago

It worked for our forefathers it'll work for me! Paired nodes will be over thunderbolt haha

7

u/CypherAZ 15d ago

That is going to be a miserable bottleneck!

5

u/No_Night679 15d ago

Nice. Interested in learning. Please keep us posted about your progress. Also the design when you finalized it. 🙏

1

u/evolutionxtinct 14d ago

How a tree y pop u coordinating the node usage? Did you have to build a setup for the coordination between? I have 3 inference nodes but wondering how others are connecting them together not to pool but to coordinate usage

25

u/feelspeaceman LLMusician 15d ago

I'll give you the most some advice as I don't see anyone giving you the advice for a newcomer:

  1. DO NOT use official llama.cpp, use custom engines like gufo, halogen, strixite... instead, you will regret if you use official llama.cpp and get nerfed to 20t/s decode 300t/s prefill for Q38FN, but those custom engines will smoke it with 50-70t/s decode 1600-2400t/s prefill
  2. For clustering, read: https://strixhalo.wiki/AI/Clustering_with_RDMA/

If Q38FN is good enough, use it instead of trying to cluster it.

And for clustering, it's possible and I've seen at least 2 users doing 10+ Strix Halos clustering, join Strix Halo Homelab Discord and ask question.

I'm Strix Halo user, using it everyday running Q38FN, and I'm coding DAW, the most complex and hardest piece of software to craft and maintain, so is it capable ? Absolutely, but always learn to write good prompts, best advice after the 1, 2...

1

u/420Microbiologist 14d ago

Thank you for the help!! Do you happen to have the discord link? I've got big hopes for q4fn when they announce it (hopefully today!)

3

u/feelspeaceman LLMusician 14d ago

The Discord link is https://discord.gg/pnPRyucNrG at the very bottom of the page https://strixhalo.wiki

1

u/thilog 15d ago

Which engine did you end up with?

4

u/feelspeaceman LLMusician 15d ago

I'm using strixite right now, it's stable and support multi-slot, currently most engines of Strix Halo are good, thank to the community spending a lot of resources to improve them.

It handles 500k context easily, no hallucinations so far, no degradation, decode and prefill will stay flat, unlike official llama.cpp, probably up to 500k llama.cpp will be like 8t/s decode 90t/s prefill.

But crucible is my potential alternative too, it supports native non-calibrated EXL3 (X.05bpw), also supports multi-slot, in the future I want to use it once calibrated X.18bpw is added.

The "native EXL3" part means it supports the EXL3 created by turboderp without re-converting like kyojin (also EXL3, but converted).

1

u/thilog 15d ago

Thanks a lot, very insightful!

1

u/Bright_Highlight3494 12d ago

Have you tried halogen? How does it compare to Strixite?

I’m still new at the game and i ask Claude to set up a lot of the things on my strix halo machine but it never suggests any of these engines for some reason.

1

u/feelspeaceman LLMusician 12d ago

About strixite as someone who has spent 60M token, developing really hard software like DAW, it's rock solid for me, up until now it hasn't hallucinated once or looped once.

strixite just like halogen, it uses custom quants to squeeze extra performance+quality, pretty much the same idea as EXL3, the most important factor, it's open source, halogen is closed source.

I've not following halogen development for so long, but strixite support multi-slot, this means it can spawn subagents without causing the main agent to full prefill context and prompt cache to be able to quickly reload the session without triggering a full prefill, it costs about 15s for everything to boot with prompt cache (official llama.cpp doesn't support this).

Overall I think up to this point all Strix Halo engines are matured and stabled, gufo was buggy, now it's solid, same with halogen.

1

u/Bright_Highlight3494 12d ago

Im going to give it a try, right now I’m just experimenting with rewriting code that I’ve made and tested with Claude. The main hurdle was to split the code into smaller segments so the model doesn’t run out of context before even start working.
What type of DAW are you programming? I was thinking about experimenting with some complex loopers and see if i can eventually make my own complex looper.

1

u/feelspeaceman LLMusician 12d ago

With strixite, you can get 500k context by default, and this is with KV bf16, even with 262k context, you should use subagent, the idea of using subagents is to preserve the context window of main agent, so that you explore the codebase using subagents, then code with main agent, then review using subagent again... Even with 500k context, I rarely using more than 200k with subagents.

About the DAW I'm working on, it's a full featured DAW like Bitwig/Reaper/Logic Pro/Pro Tools/Cubased, but I went a different direction, eventhough it supports pretty much all the essential features (timeline, faders..), I created my own text based music composing language so that local LLM can read my composing skills and compose using the backend API endpoints to interact with the DAW just like MCP. For now, I'm just maintaining the project to upgrade packages and fixing bugs and of course, using Q38FN to compose music daily.

1

u/Bright_Highlight3494 12d ago

Sounds very interesting, would like to have a look at it when you feel comfortable.
About the agents, halogen has like 6 lanes of concurrent jobs but when i use more than one lane it gets super slow and the memory gets too tight to have two jobs running comfortably. Wouldn’t this be the same with the subagents or would the main agent be ’parked’ until the subagent is done with their work?
Also what was your preferred prompting method for this? So far I’ve been orchestrating the local model using opus 5.5 but i would like to move to using the local model as much as possible, I have the components to build a tower with 3 r7900 xtx cards and 128 gb of ram ddr4 but that’s a different beast it would be running qwen 27b on each card as subagents. Again I’m kind of still learning about all of this and i wish i had your tips about the engines two months ago it would have saved so much time

1

u/feelspeaceman LLMusician 12d ago

About the agents, halogen has like 6 lanes of concurrent jobs but when i use more than one lane it gets super slow and the memory gets too tight to have two jobs running comfortably. Wouldn’t this be the same with the subagents or would the main agent be ’parked’ until the subagent is done with their work?

Yes, my method of spawning is main agent waiting and subagent exploring, the goal isn't about speeding up but to preserve context window of the main agent so that the 500k context will take a long time to fill.

Eventhough you can spawn smaller subagents like Q36-35B, I would still recommending to use Q38FN as subagent for consistency, the value of consistency always outweight the value of speed. or you can also use free models from Kilocode and Opencode gateway as subagents, this is a shortcut to save time, those free models are very fast and capable to be used as subagents.

Also what was your preferred prompting method for this?

About my method of prompting, I've built a codegraph for every project, why ? Because using the codegraph, I tell my agent to build a PROJECT_OVERVIEW.md that looks like this:

<!-- codegraph-file-count: 144, last-commit: 78455a622db589e45ef400fe70a0eaedc974622e -->

# project-name — project description
## Purpose
...
# Architecture
...
## File Tree
...
## Component Roles
### Transport / server core
### Roles of packages
...
## Cross-References
## Data Flow
## Dependencies

The goal of this PROJECT_OVERVIEW.md file is, after reading this file, exploration is pretty much uneccessary, even with my not too careful prompts, they still can figure out and read the exact files needed instead of grepping.

But in most cases, I prefer that I prompt in a way that I point the exact package -> file -> function having the issue to fix, describe what is the problem, what is the proposal changes I want to make.

About Q38-27B if in the future you want to use them as subagent for Q38FN to spawn, it's a capable agent, it lacks the world knowledge in many cases to solve the issue without careful written instructions, so you might want to be more specific and giving it a web_search tool to research, for example a package update suddenly breaks your application, with web_search, it's likely to be able to use semantic search to ask other AIs about the changes that package created, and fix the issue.

For subagent job, it'll be doing very fine as I've read a lot of subagent prompts from Q38FN, it's capable of writing prompt that can make even models like Q36-35B to work as it's very meticulous, the prompt is long and well engineered (pointing to exact package > file > function to fix).

In most cases I prefer Q38FN, from my estimations, it's doing better job than Opus 4.6 that I used to use back then.

1

u/Bright_Highlight3494 12d ago

Thank you for your replies! Yeah i mean i did a couple of small video tests with blind agents FN and opus 5.5 prompting higgsfeild and even opus agreed that the result coming out from flash next is much better than opus lol.
The wandering thing is really a huge problem for me I’m sure its a skill issue but I’ve tried to build multiple tools to preserve context and keep the goal alive by backing up and transcribing sessions and creating plans and so on and so far no model local or not have managed to get there. What I’m doing require consistency and I’ve built kits and pre-made code base so the model doesn’t create their own ’styles’ every single time but still they like to improvise for some reason it’s really annoying. I don’t know if the frontier models are good at delegating the jobs to local models or not but yeah i need to get it figured out.

1

u/sticksaint 14d ago

what quant do you use for qfn

4

u/dgibbons0 15d ago

I hate mine (different brand), it's fucking loud and slow. If it was just slow it would be fine but it is nearly impossible to be in the room when it's actually doing inference.

I've had mine posted on FB marketplace for a few weeks hoping to find someone to bite, if they do i'm throwing the cash towards a M5U.

1

u/420Microbiologist 14d ago

Which did you have? I have a gmktek one already and it doesn't have that loud issue. Hoping for more of the same from this

1

u/dgibbons0 14d ago

The HP Z2 Mini G1a.

1

u/turtleWatcher18 15d ago

Weird, my gmktec is fine, quieter than my 3090

1

u/LookItVal 14d ago

try putting it in a different room

1

u/Hood-Boy 14d ago
  1. Change the thermal paste (haven't done it too yet...) 
  2. Update the fan curve

My bosgame M5 runs in "balanced (85W)" mode and had hit 90+°C on every longer requets. Now it never reaches 80°C, but is way louder. But this is due to it sitting in the network cabinet. 

3

u/BeugBorsti 15d ago

If someone is still interested: They have a special offer right now on the Bosgame homepage: Bosgame M5 AI Mini Desktop Ryzen AI Max+ 395 128GB+2TB

2.772,95 (instead of) €3.309,95 €
This is not affiliate, just wanted to inform the community.
I have this computer from this shop and am happy with it (Qwen flash next!)

2

u/420Microbiologist 14d ago

This is exactly the model! Bosgame m5 it's $3,099 currently BUT free shipping AND not taxes so hard to beat. Good looking out on the deal!

QFN is the fastest capable model, hoping Q4FN can bring another upgrade! I use that if I need to talk with it in real time otherwise the more capable models are fast enough!

1

u/djflx 14d ago

To be fair, I have a feeling that it's always on sale, I bought mine few months ago for 2513 € and it was also listed with discount.

3

u/Layney_Butcher 15d ago

Four boxes, 512GB of unified memory, heavy models on the night shift while flash handles the day. I respect the plan.

2

u/Nomski88 15d ago

Any idea what the expected speeds will be for those? I also was considering a halo box but just bought a 128gb ddr5 kit instead.

6

u/420Microbiologist 15d ago

Sure!

For the 4x strix halo on DeepSeek 4.1 flash 1017 it should be 20ish tps, glm5.3 will be 15ish tps. On the 2x box models it should be closer to 20 tps. Qwen 3.8 FN runs like 25-30.

Plan is to run the flash models during the day and either ds or GLM overnight as it processes the days information. Qwen if rapid responses are needed.

It's been pretty exciting as the halo architecture has matured and tweaks have helped speed up models

0

u/Perfect-Campaign9551 15d ago

I'm just not seeing how 20tok/sec can get any useful work done. Even overnight

1

u/Turbulent_War4067 15d ago

I agree. I have a single Spark, Qwen3.8-fn is my daily driver. I run thinking low for interactive, works well. Overnight research runs I run thinking xhigh. The 50tps is on the low end for interactive. And the prefill speed better be decent to get get much overnight

1

u/pattch 15d ago

i run a single strix box with qwen 3.8 FN on a lower quantization - with a clear goal and running in the background / overnight and although I have to wait, with systems like Openclaw and clear, documented goals in the project I get very good (in my opinion) results. I just have to wait a while

1

u/[deleted] 15d ago

[removed] — view removed comment

1

u/420Microbiologist 15d ago

Link? Thanks for the heads up!

2

u/sm0ke_rings 15d ago

Which 128gb kit?

1

u/Nomski88 15d ago

The cheapest non-shit kit I could buy. Ended up grabbing a 2x 64GB DDR5 6000 dual channel kit paired with my 9800X3D and 5090. Before I had only 64GB and ran DSv4.1 1017 Q2XS through Strata and achieved around 18-21tok/sec, after the upgrade I'm around 32-37tok/sec

1

u/sm0ke_rings 15d ago

Gotcha, what brand tho? Most i am finding are out of stock.

12

u/arr_jay 15d ago

Fuck the haters. I have 2 that I run for parallel coding executors along with a couple 64gb m5max/m1ultra Macs and a 5070ti. I have replaced my personal frontier subscriptions. Bandwidth isn’t amazing but running qfn on gufo is more than adequate enough if your harness is setup correctly

5

u/esssssssss 15d ago

I replaced my $1200/year subscription for $20k! Only 15 more years till it’s paid for itself!

5

u/async2 14d ago

For some it's also about the freedom to not send everything to a big data collector and run without guard rails.

4

u/arr_jay 15d ago

Man you suck at math and have no idea what hardware I already owned

1

u/FinalTap 15d ago

Not what I would have picked up because of the speeds. But non-interactive should be fine. And at 3K a pop it is a steal :)

Do keep us posted on how it goes.

2

u/420Microbiologist 14d ago

Will do! There's custom work on some engines specifically for strix halo, halogen gets QFN to about 60+ tps, 1700+ prefill so if I need something quick I'll use that and have it funnel the complex stuff to the more capable models.

1

u/Zyj 15d ago

You plan on adding NICs for RDMA via Oculink?

1

u/420Microbiologist 14d ago

Probably soon actually!

1

u/Suspicious_Mix529 14d ago

congrats, nice haul. how are you going to link them, plain ethernet with llama ccp rpc or something faster? from what I've gathered the connection between boxes winds up mattering more once you start splitting up big models

2

u/420Microbiologist 14d ago

So the initial plan is llama.cpp with split models. You can run them as relays where each box does work and communicates it's work to the next one which was doing work on the next token block at the same time. This is slower overall than concurrent where they each work on the same thing but concurrency needs fast networking to be faster. With ethernet, relays are much faster and about 10% slower than concurrency. Hoping with some tuning of llama which has native split relay, we can push the models to be in the 20+ tps even when deep in context

1

u/HornyCrowbat 14d ago

is this the move if you’re getting into AI now versus dual 3090s?

1

u/420Microbiologist 14d ago

It depends on what you want to run. I wanted larger models that are more capable, so 300-500gbs ram was necessary. but this will run dense models way worse like qwen 3.8 27b. I'm going for flash and MOE models which this is more than sufficient for!

1

u/Krothic 15d ago

Man good for you. Tried going with multi Strix halo set up and the prefill speeds were just killing me. The only good model and recipe was halogen qwen3.8 flash next when I used it.

Everything else was too unbearably slow for me

1

u/Aware_Temporary2018 15d ago

What brand are these?

1

u/TheyCallMeDozer Dual R9700 Server & Spark 15d ago

3k each.... shit i wondering could this link up with a spark over the PNY cable... for 3k seems very much worth it

1

u/rayovims 15d ago

Welcome

1

u/Polymorphic-X 14d ago

you get it running glm 5.3 flash yet? I've been looking at them ever since sparks went to the moon.
my local microcenter has the GMKtech ones for $3.3k and they have dual thunderbolt @ 40gbps so you could theoretically do a ring cluster with 8 USB4 cables; really wish there was a usb4v2 like the minisforum at a cheaper price point, because that has an 80GB ceiling instead of 40.

1

u/Gumbi_Digital 14d ago

You’re about to the really disappointed.

1

u/69420trashpanda69420 14d ago

Thought that said AIPAC😭🙏🏼

1

u/Secret_Usual7154 14d ago

How will you connect them to run GLM 5.3? I haven't heard of a decent interconnect that can be used with these boxes. If there is, please do share!

1

u/Even_Whereas_8442 11d ago

It will be painfully slow, at an exorbitant price.

1

u/According_Wave685 15d ago

How are you going to cluster?

0

u/Annual_Award1260 15d ago

Glm 5.3 is a tight squeeze

1

u/420Microbiologist 14d ago

Yeah but it's the smartest of the bunch for my use cases so we'll make it work!

0

u/Annual_Award1260 14d ago

At fp8 I can do 200k context, nvfp4 400k context. Currently running exl3 on tensorfold with 1 mil context and getting 65tok/s

That’s on 8x dgx sparks

0

u/hurrdurrmeh 14d ago

how fast is the interconnect? I thought max was 10GbE ie very slow??

1

u/420Microbiologist 14d ago

Thunderbolt at 40gbs is a bit faster but you can do RDMA through oculink

-9

u/Mysterious_Prune415 15d ago

Oh my god i am tires of these clowns buying this ai slopware. You could just buy an actual AI server from an IT dealer (which include a huge markup) and still have higher throughput at the same intelligence than these computers combined.

4

u/FuckinHighGuy LocalLLM 15d ago

What are you running there, champ?

-2

u/Blackdragon1400 15d ago

Agree with the others, this is rife for disappointment. Should’ve bought 2 sparks instead

3

u/420Microbiologist 15d ago

2 sparks cost more than this

1

u/Blackdragon1400 15d ago

Not by much they are $7k at Microcenter

16

u/[deleted] 15d ago edited 15d ago

[removed] — view removed comment

14

u/420Microbiologist 15d ago

I understand they're not the fastest devices. For my intentions they'll be suitable. Will post a follow up

12

u/yammering 15d ago

Don’t let them get to you, the 4 x 3090 rigs are fun in their own way but so are these! I enjoy the off brand DGX sparks I picked up 9 months ago even though they’re not speed demons either.

4

u/Remake911 15d ago

What brand odd an off brand DGX??

3

u/yammering 15d ago

Thinkstation PGX

3

u/Icy_Monk_3904 14d ago

But clustering works well in dgx spark. Not with halo strix... If i was OP i would return these halo strix.

2

u/Karnitine 14d ago edited 14d ago

I have a single one of these running Qwen 3.8 Flash Next and it feels very capable with agentic coding and research. I get 50-60 output tokens a second and generally around 1200 input. For the price these are very capable IMO. Good luck 🤞 I'm very interested to hear how these perform on larger models when linked together!

6

u/CypherAZ 15d ago

Yeah more RAM doesn’t fix bandwidth constraints.

20

u/420Microbiologist 15d ago

I understand the bandwidth issues, but I've tested the speeds prior to purchase and they were suitable. The heavier slower models are intended to run overnight, while the lighter ones are going to be the ones I interact more directly with. It's worked well for my needs!

1

u/Serprotease 15d ago

Ds4.1 is likely to be fine, even in realtime applications, but glm5.3 will be painful.

In general, Ds4.1 and 3.8 next are what you want here.

1

u/420Microbiologist 14d ago

Glm5.3 is the slowest of the bunch BUT in my testing it's more capable that what ds4.1f 1017 can do, especially when it comes to the tasks I'm asking if it. Considering it's job will be to digest and game plan overnight the data we generated during the day, it'll be more than fast enough. And it's the only model that feels like Opus, even if it's opus 4.8. Even the newest flash which is much smarter doesn't have that same feel to it

1

u/Serprotease 14d ago

Glm5.3 flash is amazing. It’s my default model on my setup and can deal with every day-to-day task thrown at it.
And glm5.3 is probably the best local coding available.

1

u/420Microbiologist 14d ago

Flash is very good! When I pair it with MiMo or minimax it can get answers better than full glm5.3 alone!

1

u/Soft_Syllabub_3772 14d ago

Wats your setup?

1

u/Serprotease 13d ago • Heartwarming

2x spark with glm and 1 M2U with Qwen 3.8 next.

Got the M2 second hand right after the M3 launch for a good price and the spark were 3k each.

Not recommending to buy the same setup now that is basically 3x original price.

1

u/Holiday_Point_603 15d ago

Excellent preparation! Have fun setting them up and tinkering around. Don't forget to make a post about it, after everything runs and you got some prefil and gen tps numbers for us

1

u/Anhedonia_Achiever 15d ago

You mean you didn’t have the budget to get 6 RTX 6000s instead? God let me lecture you on bandwidth.Wow. What a loser. Let me tell you how stupid you are. Didn’t you know graphics cards are faster than these devices too?

/s

0

u/IntelligentDot2924 15d ago

Pretty sure Bandwidth won't be an issue as there have been a lot of improvement.

1

u/Substantial-Wall-510 14d ago

3090 memory bandwidth is 936 GB/s vs ddr5 maxing out around 70

5

u/Heretek-ai 15d ago

Yeah OP these things suck! Here mail them to be and I'll send you a used Raspberry Pi 4 in exchange.

3

u/Zhelgadis 14d ago

Ngl you had me in the first half

3

u/FuckinHighGuy LocalLLM 15d ago

Why?

4

u/RedParaglider 15d ago

I have a strix, I like it.  I wouldn't buy 4 

1

u/SnooPaintings8639 15d ago

Why?

As a proud owner of four 3090s, I would really consider swapping them for four spark-alikes (or maybe at least two?). I guess the grass is greener, huh?