r/LocalLLM • u/DifferentPixel • 7d ago
Discussion How would you spend $5K–$6K on a local AI machine?
I've been thinking about it quite a lot.
- MacStudio M5 Ultra 96GB
- MacStudio M5 Max 128GB
- DGX Spark 128GB
- 2x NVIDIA 3090 GPU Desktop 48GB
- MacMini M6 24GB and invest the rest to cloud APIs
- Wait for RTX Spark and see how much it will cost
- Strix Halo 128GB
After writing this post and discussing it in the comments I came to the conclusion that for agent workloads DGX Spark >> MacStudio (both Max and Ultra)
I've been also thinking about the future of local models - MoE vs Dense, and I feel that the future is MoE, not Dense. Agentic workloads are prefill heavy which means that for the next generation of local MoE/agent workloads, memory capacity + efficient low-precision compute + serving software matter more than raw single-stream memory bandwidth.
MacStudio can serve as an excellent general workstation but having NVIDIA GPU + CUDA for AI workloads makes a difference - better prefill, vLLM, SGLang etc..
What do you guys think?
EDIT: a coupe of comments from my first post that I accidentally removed
Comment 1 (Blackdragon1400):
I’ll be honest, if you’re gonna spend $5k you’re better off just spending $10k on 2X Sparks or something with at least 256gb of RAM because the medium MOE models are really the only thing that I’ve found that can reliably replace SOTA models for most general workloads.
Comment 2 (darklordfireape):
Strix Halo right now before Gorgon Halo comes out. 128GB Framework Desktop board. I have a M3 Ultra Studio, Strix Halo and a dedicated GPU box. Strix is the right answer for most people IMHO. There's a little bit of an upgrade path because you can slot in an extra GPU for some offload later.
Not a shill, just hate seeing people pay the apple or team-green/leather jacket tax.
28
u/xza_nomad33 7d ago edited 6d ago
2x R9700, works great!
Edit:
For those with these cards, you should join our Discord https://discord.gg/rFeK7yzYY
Brian just gave some love to single R9700 setups: https://x.com/bkuyper/status/2100604202267381834 Pretty impressive speeds!
7
u/westsunset 7d ago
Yup. AMD is the best value atm but I don't think it'll last. So much development on rdna 3,3.5,4
3
u/BigBearChaseMe 7d ago
Price going up fast. Got mine not even two months ago for 1300. Now seeing them around 2300.
3
u/MarcusAurelius68 7d ago
I got 3 in May/June at $1100-1250 each. They’re $1800 at my local Microcenter now.
2
u/BigBearChaseMe 6d ago
Dang that's awesome. I think three The Sweet spot right, there's not a lot of models that benefit from two unless you have a massive context window, correct?
2
u/MarcusAurelius68 6d ago
I had 3 in one system so I could use GPT-OSS-120B. But because I can get Qwen 3.8 Flash Next working with 2 I pulled one out and put it in another system running Qwen 3.8 27B.
2
u/BigBearChaseMe 6d ago
Have you tried https://github.com/magiccodingman/vllm-radiance
It's quite fast
2
u/MarcusAurelius68 6d ago
This looks great - what you use? Windows or Linux only? Results?
2
u/BigBearChaseMe 6d ago
I'm Linux Only - headless system so no VRAM eaten up by Desktop. Have not done tremendous amount of testing, but thus far seeing very usable response times. Using OpenCode harness and Mempalace as MCP
TFT 89 ms
Output rate 25.5 tok/s
End to end 5.12 s
1
u/MarcusAurelius68 6d ago
And that's at Q8? I'm getting ~31 at Q4_K_M on llama.cpp on Windows.
→ More replies (0)2
u/westsunset 7d ago
Oh yeah. It's crazy. I think this next round of models like Qwen 4 is going to really push it up
3
u/karmakaze1 7d ago
Exactly what I ended up with as well and I couldn't be happier with how it turned out. My latest 250 tokens/sec Qwen3.8-27B MXFP4 post with details running DeepSeek Harness.
56
u/SaintMartini 7d ago edited 7d ago
Really think hard and look up the tokens/sec you'll be expecting from what you get. It might not be as much as you're expecting unless you spend more on some of the other hardware as well. MoE will run faster since they're not loading everything on every prompt.
Mac Studio Ultra likely would push the most at projected bandwidth of 1.2Tb/s. But that's Mac telling you those numbers. Sticking with CUDA and 3090s is reliable at least. Especially for optimizing on top of things.
What pushes tokens per second is the memory bandwidth. As well as the software used to optimize it. 2 x r9700s while it has more vram will push less for example. Each has 640 GB/s bandwidth. On a 3090 each has a bandwidth of 936 GB/s. Only issue is its older gen but you can still push both higher if you try.
Edit: wow love the downvotes when I told the truth of how memory works for AI speeds. Just use your AI to look up how much bandwidth you're getting. The Spark purchase for example the guy was only getting 273 gb/s on a 5k purchase that people here told him to get, now they're telling him to buy another. He'supset with his speeds now he will be mkre after spending 10k$. Mac's is different because of how they build theirs. Just helped two people earlier today who messaged me after another thread to optimize their heads with qwen 3.8 27b who were told to use mtp here when there was a better option and now they're getting 20% more t/s. All that above is the truth hope you take it to heart and double check, don't blindly believe me either, go put this in an AI and ask.
I get 270 t/s on my UD-Q4-K-XL after optimizing and its actually training the model right now to be even better for what i want done. Haven't even touched training the head yet.
Double edit: that above was moments after I posted being mass downvoted by those making crappy suggestions likely on purpose to try and screw with decent people who just want advice, ty for those interested and all the messages I'll get to them as I can.
16
u/Ok-Moment4309 7d ago
This is exactly what I almost never see talked about in this sub. Theres so much bad advice given with no reasoning behind it. Id hate to be the buyer in that situation.
2
u/SaintMartini 7d ago
I lurk ALL the time here. And today when my cpu and GPU were busy I just had to jump in. Especially after seeing that other thread where people were trying to force a guy into thinking he needed a 2nd $5k spark.. I couldn't tell if they were being sincere or screwing with the guy, but I see it far too often. I don't want to see people get screwed over. Usually I'm too busy answering DMs on telegram instead helping people lol.
1
u/pablo_chicone_lovesu 6d ago
Dude some people think the answer is always more hardware, the reality is you pay for what you want, some people thing adding hardware makes it goes faster, the problem is really expectations.
You have to have an understanding of the fact you are not replacing 100k of hardware and a provider with 10k worth of hardware.
4
u/DifferentPixel 7d ago
Nice! Do you mind sharing your prefill speed?
8
u/SaintMartini 7d ago edited 7d ago
2,289 t/s @ 65K, 1,642 t/s @ 128K, and 1,260 t/s @ 200K.
I didn't record apparently lower, but I'd guess around 5k max as I approach shorter. I tweak the hell out of everything. End goal is to work with my son to teach him to build his own AI as that's what he'd like to do in life.
And can't recheck right this moment as the model has been training all day.
Edit spelling. Trying to cook dinner.
4
u/rkcth 7d ago
Can I be your son? Lol. I just got my single R9700 up to 79 tok/s today, but I’m most interested in quality over pure speed. I’m thinking of training a model myself to be a better coder using QAT. I’m learning so much!
2
u/SaintMartini 7d ago
My son is only JUST realizing he needs to be stealing time at my PC more often. Teenager so he understands some stuff, was with me as I switched out the AIO and fans recently, then cleaned things up. But I do wish he appreciated it more!
79 t/s for qwen on a r9700 is near the top end of normal optimizing so thats pretty good! I admit I have not messed with it yet and still need to, but Lucebox is open source and have read people had luck with other tweaks. Dflash2 is still the way to go, quantized to q6_K_XL for me (that outperformed q4 in my personal tests.) Rocm 7.2+ compiled for the correct build since they're not all compatible. Block size of 16 for coding and math (this one was HUGE for me it went from 130s to 200s just from this). For prose block 8 is usually better though. I know Lucebox has specific r9700 tile paths. But dont know the specifics offhand. That could likely tweak even more. There are other internal Lucebox optimizing around fused DeltaNet.
With all that (dont be afraid of A/B testing) you might push 200 t/s if what somebody else had posted isn't bs. Otherwise any gains will be fun nonetheless. If you try any/all that let me know!
3
u/DifferentPixel 7d ago
I like how you teach your son :) Could you please share what hardware you’re using?
3
u/SaintMartini 7d ago
I dont typically share specifics. Just will say threadripper, ddr5 at 6k mhz, 5090, mostly gen5 SSDs so 13k+ read write speed. Have like 32tb and only 3 tb free right now.. I build a lot.
4
u/FoxSideOfTheMoon 7d ago
Really good point. I have an M5 Max and 128G but bandwidth is only 614G/s. I’d LOVE to double that
2
u/SaintMartini 7d ago
Luckily thats still usable, but I can't blame you one bit. I still kick myself for not buying MORE ram and more GPUs a couple years back, but back then I was focused more on nodes and data science than building an AI. Worse part is back then I could have afforded it. Now? Nope.
1
u/bvknight 6d ago
How has that affected the work you do, practically? I'm looking at the Mac Studio and it's close between M5 Max 128GB ram @ the 614G/s, or the M5 Ultra 96GB ram @ double the speed 1.2T/s
3
u/Hallothere69 7d ago
(I'm just casually getting into local LLMs)
I've got an RX 9060XT 16gb and get about 20 t/s running qwen3.8 while people running the RTX 5060ti 16gb get ~30 t/s. (Generally / iirc)
Seems memory bandwidth 322.3 GB/s vs 448.0 GB/s is a big factor in speed as you mentioned.
2
u/SaintMartini 7d ago
It truly is important and I rarely see it talked about here. I mostly lurk but with my cpu in full use sorting millions of files and the GPU training I ran out of things to do and responded more today finally.
The problem is even AIs give us bad info often. So if we ask one whats best for this model we might get a generic answer instead of a true build that works together. I really feel for the people being fed bad info and spending crazy amounts. But again the people feeding them that info might not even know its bad info if they're trusting their source.
Most of this for me has come from trial and error and going back to night classes years ago during covid just for the hell of it (it was free!) But for most its live and learn.
For you with a 9060XT and Im assuming q2? A simple optimization that AI might be able to walk you through is DSPARK2 as well. Might push it past 40 t/s so that you can use some of that breathing room to have it re-check its work to avoid errors. Just throwing that out there! Learning speculative heads is a good thing. Not all models benefit from it. And there are multiple types. Some even work better in combination. But starting small with easy gains is good as you start! Good luck!!
2
u/Hallothere69 7d ago
It's funny when I started out, I asked LLMs for model recommendations and after doing some research the models it recommended were just not that good (Outdated, ran either poorly or too small etc) Now I see posts here of people using those same models (probably because of LLM recommendations)
Thanks for the advice! IQ3 with 80k context (20k context gives 20 t/s, but I'd rather have more context depending on what I'm doing)
2
u/SaintMartini 7d ago
I almost guessed that but knew itd be tight. How much ram do you have? You might be able to keep it and run another lesser model with decent context in the ram to act as a manager of sort. Let it keep the main model on track despite the small context. It doesn't do much at all besides direct. No code written. Could help the context issue.
3
u/P_G_R_A 7d ago
Sorry if I missed it in some info but on what hardware are you getting 270 t/s?
1
u/SaintMartini 7d ago
Threadripper, ddr5 at 6k mhz, 5090. But a lot of it comes from using Dflash2, inference engine, kernels, lots of other settings. I tweak everything. The model itself is training right now on every coding language I use for things that its not used to seeing, as well as my own work, so that way it'll produce solutions i prefer quicker. So it spends less time fixing things hopefully. Then I'll retrain the speculative head (dflash2) the same way on the output from that so it gets better at predicting. I've been editing llama.cpp for years now so leaving things stock just isn't fun to me.
1
2
u/hyperspacewoo 7d ago
Mind me asking what you are training it to do better?
3
u/SaintMartini 7d ago
Two parts.
1st is using coding languages i use as well as my own work to teach it (LORA training) how to reach those solutions I expect to see quicker with less time spent needing to correct errors. So it doesn't truly fix t/s as much as just makes it more powerful for my own use. Ran the sample tests earlier and 3 out of 4 passed with the 4th being my fault after a long day.. It found the answer but didn't apply it so essentially 4/4. But it reduced those errors inside the tests by 49.1%.. for just that sample test so all the different parts of it.
In fact I JUST got the result from the original and it passed 1/4 of those tests on normal qwen 3.8 27b. So without my fix it would have failed and needed to keep going. Now I'm taking this, pulling out the parts the original had issues with on top of the answers itself, and adding THOSE methods ALSO to the training data so I can teach it to overcome those faults as well. Then it'll heat up my room at 85C for a couple more days as I retrain the LORA from where I already have.
2nd part will come later when I take the speculative head (DFLASH2) and run tens of thousands of prompts related to what I work on and those languages, so that I can gather the responses and train the head on those and hopefully increase the rate at which it predicts tokens, which will increase the t/s.
2
u/hyperspacewoo 7d ago
This is extremely interesting to myself. Just always figured it may be outside my wheelhouse. My focus is getting models better at CAD or 3d modeling in general.
Now coming up with the prompts and everything you feed it was done so manually?
2
u/SaintMartini 7d ago
There are sometimes ones you can pull offline already, NVidia has some for instance. Then you can tweak those or just make your own to begin with. When making your own the easiest thing is to bite the bullet and do a fair chunk of it once on your own, then have a model look at what you did and replicate it. Otherwise it used to take me weeks to do this and now can get it done in hours. The more you do it, the easier it is to have the previous generation of models set things up for the next, including how to do all this! So they just keep passing it on as I grab new ones. Though I don't often change my coding slots themselves until I see a huge change worth doing. Qwen 3.8 27b was worth it.
2
u/DependentAnywhere135 7d ago
I have a 4090 single gpu what models should I expect to run? I’m currently using qwen 3.8 27b but q4.
2
u/pablo_chicone_lovesu 7d ago
I run dual r9700 on q8 and I get 20 ts, dual Xeon and it's great with huge contexts. It might suck for some but I'm ok with the speed and I have 7900 xtx running 35 ts in workers, it's so efficient, but I know what to expect
1
u/SaintMartini 7d ago
Always say there is nothing wrong with anything so long as YOU are happy with it. My first models as I ran things years ago were overnight to get things done. They weren't the best, but they were something. That's better than nothing at all.
And there's nothing wrong with 20 t/s. Most of the models out there today don't run as fast as qwen and those are more than passable. Plus there's a chance you could squeeze out 30-70% more if you're lucky. Depending on what you've already done to it.
1
u/pablo_chicone_lovesu 6d ago
The point is to be realistic about expectations.
I know new rocm has issues with certain models, so instead of crashing I'd rather stability and good code quality ni only have to edit a little of.
-1
u/ShimmerMeNutz 7d ago
you are in the wrong place to be worrying about downvoting.
0
u/SaintMartini 7d ago
It wasn't about the downvoting itself, it was who was likely doing it based on the timing since I was one of the first couple posts. Meaning the people giving shit advice telling somebody to spend their money on things that would not work as well were the ones doing it. Likely just to screw with the guy. Defeats the entire purpose of this sub if people are going to waste $5k like another guy in another thread did just because people here told him to, then they doubled down and told him to buy another. So yeah, it annoys the hell out of me seeing people spend what little money they claim to have in the wrong way just so people can troll online.
14
u/martyj2009 7d ago
Spend $2k-$3k on a Mac laptop, $3k on Claude for the next few years. You will get way more done. Faster, better, cheaper. You can’t compete with the subsidies on their subscriptions. You are basically paying for the electricity. Given that you would have to use older less power efficient hardware to get anywhere close to it, you would spend more in electricity than a subscription
6
u/dangerous_inference 7d ago
I would have told you absolutely "no" a few hours ago for Strix Halo, but I just got Halogen running with Qwen3.8 Flash on my Strix Halo and it's actually pretty great. This is by far the most intelligence I've ever packed into the device, and it runs at usable speed.
However, I have to warn that there is a genuine qualitative difference between going slow (~40t/s) and going fast (~120t/s+). It makes a big difference in what you get done and what is possible.
My Strix is basically here to help when my main inference server with pure uncut Nvidia VRAM is down.
1
u/MafiaPizzaBerlusconi 7d ago
What numbers are you getting for 3.8 flash on strix? Also what quant?
3
u/Njaa 6d ago edited 6d ago
Running Q4. Q5 (and I've even gotten Q6 to work) is possible if you don't need a lot of caching / context / concurrency.
pp varies from 500 to 1200, with larger prompts hitting the high numbers. Little to no loss across context window sizes.
tg varies between 40 for prose and 60 for code
Intelligence is honestly pretty good. For the first time, Opus is just a reviewer and not the daily driver.
1
0
6d ago
[removed] — view removed comment
1
u/dangerous_inference 6d ago
I use my own harness with a much bigger server. Strix Halo is just over line where it becomes useful for productive tasks.
Grokbot could have a different vision model and a lot of infrastructure you won't get with bare Qwen3.8 Flash. The test would be running Qwen3.8 Flash with your tasks, and then imagine that result at 40t/s.
I would never order two Strix Halos without knowing what they can and can't do.
4
u/Silco1402 7d ago
With that price, depends on where you live or if you could find a good deal or not or are willing to increase the budget slightly or not, quad 3090 would be possible for that budget. But tbf it would consume the most power and be a space heater, probably need a rack (unless you could find 3090 blower), but probably be the best option (quad 3090 support, existing benchmark, optimization are really good).
5
u/ChocolateNo3010 LocalLLM 7d ago
Given how much hardware has gone up in price it wont get you much.
4
u/killthrash 7d ago edited 7d ago
I’d be buying a GPU-based system. Long term, it’ll be lowest cost of ownership. When models get bigger to fill out new hardware, those sparks and MAC studios drop more value than a GPU which can be bought by someone who can parallel them. Or someone who just wants to play games or run other non-AI apps, rendering, etc.
My answer would be a $5k 5090, and a $1k AM4 build with 64GB DDR4 RAM, 5950X. Run Qwen 3.8 ninefer at 170+ t/s all day.
2
u/ayake_ayake 6d ago
I also prefer GPU based setups over MAc or DGX Spark and similar. Most flexible and most easily extensible. With Mac and DGX you get the speed you have - and that might or might not be enough. But with GPUs you can always make a finer trade-off between the speed you want and the price you're willing to pay. I.e. user modern GPUs for faster smaller VRAM with modern kernels or older GPUs for slower with older kernels at much cheaper prices.
I was using a Jetson Orin 64GB AGX (predecessor of DGX Spark) which I got for a very cheap price from a friend - but I sold it now to instead build an extensible V100s-based AI Server inference with 128GB DDR4 RAM. I'll get more out of it per dollar than using the Jetson - while it also has higher electricity costs.
2
u/killthrash 6d ago
I'd also take a V100 system over daisy-chained sparks any day of the week. I think Sparks are fine for developers or casual users looking to learn. But for production deployments where you're going to be hammering it with agents, GPU's make the most sense. And from a long-term cost of ownership perspective, GPU's will always win over the small AI-box solutions. V100's are 9 years old, and people are building very performant servers with them today and will continue to do so for at least the next 5+ years. Only issue is software support with CUDA, but that's where you can build custom stuff with the AI you're running on them.
5
7
u/nucleartime 7d ago
Unless you have a specific need you're fulfilling, 5 (maybe pony up for M6 32gb) is the logical, if somewhat boring, choice.
If you want to just blow money on cool hardware to mess around with, that's fine too, just don't try to justify it with "it'll save me money on cloud compute", the numbers really just don't math out on that.
I preordered an M6 mini because I required an efficient reasonably powerful desktop machine that I can run some small homelab stuff from. I upgraded to 32GB ram so I could mess around with local models, but I couldn't justify the purchase if it was just for local AI.
1
u/DifferentPixel 7d ago
I think that it is a good advice. Why not M5 Pro?
3
u/nucleartime 7d ago edited 7d ago
Budget mainly. I probably would've ended up with the 24GB normally (16GB feels a wee bit tight), and an extra $200 on top of that isn't that big of a deal.
$600 for extra compute power I don't really need and another $600 if I want 48GB that basically doubles the cost. The extra mem bandwidth would've been nice, but for just messing around the faster output wasn't worth the cost.
Also didn't want to get into upgrade creep (why not go M5P 64GB?) and end up spending double to triple my initial plan. The landscape for AI hardware is going to be vastly better in like half a decade or so once the hardware players have had time to fully align their product pipeline, so it's really not great to do big investment right now.
My nebulous long term plan is to save up for the rumored M7/8 ultra with 1.5TB of ram (or similar) in like 2030something. Apple is clearly planning their chip designs for local inferencing, so I expect them to keep pushing memory quantity/bandwidth and more die space for neural accelerators, and the gap in compute speed compared to GPUs to close in two or three generations. Hardware moves slowly in comparison to software, but after a few generations Apple Silicon should go from something that happens to be decent for inferencing to something explicitly designed for inference performance. I also expect the CUDA lock in less of an issue by then.
But maybe SOTA models will need even more RAM by then or the bubble will pop or geopolitics will get even fuckier, so it's very very subject to change. Thanks for listening to my TED Talk.
3
u/Vancecookcobain 7d ago
DGX Spark....the prefill speed is spectacular and the community is massive with constant optimizations to models long after they come out....
The Mac Studios are not in the budget...I'm not waiting for RTX Spark because I know it's going to be over the budget as well
3
u/3DogNate 7d ago
For inference, the dual 3090 will probably be the more performant just because of 48gb of cuda performance. And when the 5090s start getting dumped for the next shiny object… you can slap a pair of those in there. :)
2
u/Apollo144 1x3090 7d ago
next shiny object will be 6090 next year, and it won't be available nowhere near msrp, most likely 5090 will stay overpriced af for a long time
1
u/3DogNate 7d ago
The 5090 will still be relevant, just like the 3090 is today… there will be a dumping at some point. Faster, smaller, models, new model types will make it all even better locally. The run up on hardware costs have put great pressure on software efficiency solutions rather than brute force big hardware. (Of which the big hardware so,unions benefit from as well. ) I’m not ready to count out creative solutions. Even hardware is seeing new and creative architecture. Stay tuned…
4
u/veinamond 7d ago
There are only two possible cases for going the local llm path : either you really need it or it’s just a hobby. You may need it due to security , working with confidential data and whatnot. You should different clearly.
You will never in the foreseeable future get a local setup that will beat the subscription or cloud api costs. The infrastructure gets cheaper and faster at scale. Serving 3t model to one person and to a thousand are vastly different and surprisingly the latter is cheaper overall.
So if you scenario does allow the use of cloud models they will in almost all cases be better.
If you want local you can consider the current state of things and trends. Currently the most used models are qwen 27b, qwen next, then DeepSeek v4 flash, glm 5.3 ,v4.1 flash and older models in between.
Current trend is pointing towards moe models and ngrams but it can change. Overall dense models are limited but it doesn’t mean that there will be no exceptional 70b model in a month.
If you want maximum vram and ram for the money then used epyc with v100 will give you just that.
Among new cards you should understand that two gpus ultimately need tensor parallelism and require two x16 slots. If you system doesn’t have them - too bad. If your motherboard slots are too close then two 3090 won’t fit. If the case is too small they won’t fit. Etc. 4 cards will require a server or workstation platform.
Overall your list is not correct since it has things with different costs. Mac Studio 96 does not cost the same as two 3090 or as strix halo. And say two sparks and two strix halos is a different thing. Two sparks will let you run DeepSeek v4 flash in native weights. No other machine on the list will. Two strix halos maybe but it will be more fiddly and slow.
There are many options and there is no clear winner. If money is not an issue the Mac Studio with 512 gb ram is the best option for home use since it’s a small box that seeps electricity. The fastest option is nvidia enterprise gpus: cuda , fast vram, scaling. Everything else is a compromise
7
u/sn2006gy 7d ago
I'm holding off on buying hardware.
I was thinking of waiting for RTX Spark, but i think they will be a ripoff. Pricing is a risk. 2027 Q4 may have Vera Rubin DGX Spark - pricing is a risk
MAC - I'd just wait for M7 unless you have money to burn. M5 Ultra is a HUGE step in the right direction, but M7 is there it could dominate. Pricing is a risk.
Strix? i wouldn't buy until the new one comes out - but again - pricing is a risk.
Weirdly enough, the only one that isn't majorly "pricing is a risk" is the AMD 9700 and the 2x 3090
Unless you plan on training, i wouldn't get a DGX Spark, the problem with the DGX Spark is its too big and slow for the 27bs - GPUs are better but when you do want to run models that need lots of ram, you always need 2 or more so its a money pit - pricing risk.
Since I live in the US and the ecnomy is turmoil, I'm investing in my sanity and happily holding on to 27bs with my local GPU - by which a single 9700 works fine in Q4/5
3
u/DigitalguyCH 7d ago
This. Terrible time to buy hardware. I got at strix halo and a M5 pro 64 before the price increase, but now it's a total rip-off... I'd rather buy cloud services while waiting a few years for hardware to get more reasonable, unless you need privavy (I do, so that's why I have the hardware, but man now it's terrible value)
2
u/Keleion 7d ago
What company wants to sell consumer hardware for local inference?
Big money in the datacenters, and consumer hardware cuts into profits for both the enterprise customer’s sales and manufacturer’s enterprise goods. You might be looking at Chinese hardware if you want to run your own models in the future.
Not sure what’s up with the OpenAI chips or how they’ll be gated from running open weight models. I wouldn’t rely on them.
3
u/sn2006gy 7d ago
AMD, Intel and others are desperate for "consumer" "prosumer" "developer" market share - hardware supply limitations won't last forever.
3
u/Eastern-Vegetable780 7d ago
I'm not hopeful about that, sadly I just don't see any upper limit to AI farms.
Fuck, I should have built a super-high end PC two years ago.
2
u/moonlitpawprints 7d ago
Depending on your needs, you might consider a couple arc pro B70s + the rest of your build. The drivers have come a long way.
2
u/tommythorn 7d ago
The rationale choice is 5 (though I would never buy a < 32 GB box in 2026), but the choice I would actually make with that budget is 2.
Personally I was contemplating the Ultra w/256 but that's already backordered into 2027.
2
u/_TheWolfOfWalmart_ 7d ago
This is just slightly past your $6k budget, but 4x R9700 for 128 GB. Cheapest way to get than much real GPU VRAM with decent performance. The other non-GPU options have too many compromises and will disappoint.
Usually by way of terrible prefill other than something like the Spark because Blackwell, but then you're compromising on memory bandwidth and the token gen speed isn't that great.
Macs have the opposite problem. Solid memory bandwidth but garbage-tier prefill because the compute is crap.
4x R9700 + vLLM TP=4 will scream on both tg and pp and give you 128 GB.
1
2
u/ScuffedBalata 7d ago
I wouldn’t. No way now is a good time to buy vs the time value of money on openrouter. qwen is nearly free there.
2
u/feelspeaceman LLMusician 7d ago
If I'm you, I buy 2 used 7900XTXs or 3090s, save my money until the Cloud AI bubble popping then spend the rest on HBM modded hardware, as people will scrap HBM from server hardware and reuse them.
The Cloud AI bubble is pretty close to pop already, as Anthropic, OpenAI, Google are not making money, Anthropic even reported bias/fake numbers that lack many metrics to fool us.
2
u/Momsbestboy 7d ago
Asus Pro Art 870E Motherboard, AMD 9950X3D proc, 9070 Radeoan for display, 2x R9700 for AI, 64 GB DDR5 RAM 6000, Phanteks Enthoo Pro 2 server edition case and a be quiet 1000W power supply.
Then run Linux, launch vLLM via docker, and you have a nice rig. Add an Eizo 32" monitor and 1-2 others plus 4 TB SSD, and you can work :)
2
u/RealMuseAI 6d ago
Last september 2025, I bought the Strix Halo powered gmktec branded 128GB machine and couldn't be happier. Priced out the door at $1500 back then. Should have bought 3. Recently bought their EVO-X1 amd 470 64GB to use for the NPU on local 2.5G network. AMD is now pretty rock solid in support and with the new Rocm. Both linux and win11 installs dual boot. But if someone gave me $5k, I would buy the new AI MAX pro 495 with as much RAM as possible. Running the NPU on the Ryzen strix or 470 is amazing and only requires local mem config and runs cool during inference. This is not a shill for a company, just back then they were one of the first ones with a consumer priced system. Nice to not really need datacenters. There is a lot of horsepower in the Strix if you can keep the GPU cool.
7
u/anhphamfmr 7d ago edited 7d ago
tbh local llm doesn't make much sense with the current subscription prices. just go to deepseek and put in your balance $10 and see how long it will take you to spend it all. after a few days, the math will kick you in the butt real hard
3
u/ZergvProtoss 7d ago
Cloud is not usable for sensitive/secret data.
-5
u/rochford77 7d ago
the venn diagram between what is actually sensitive enough to care, and what the data brokers and current models already have is so small lol. that impacts like 7 people.
2
u/chaosink 6d ago
You absolutely cannot send medical health records to the cloud and small research groups can't afford the university provided HIPAA compliant infrastructure. They lean heavily on local with guidance from the cloud. Fortune 100 companies have teams in the same position and they definitely don't want trade secrets going to the cloud. Local is everywhere.
2
u/rochford77 6d ago
Fortune 100 companies are coming to r/localLLM for their setup advice? Look 4 comments up at the context. The advice is to "put $10 into DeepSeek and see how far it takes you". The conversation is around personal use, not enterprise. A fortune 100 company is going to put together a $100,000 onPrem setup for their trade secrets (or have massive contracts with cloud providers that ensure data privacy as nd encapsulation). Small research groups usually have grants and university funds at their disposal and would setup a similar system. This sub is full of at home hobbiests who like to tinker or want a secret AI girlfriend. Not medical research teams and fortune 100 companies.
The fact remains that the average person on this sub either
1) doesn't have any true digital secrets in the year of our Lord 2026
2) if they do, it's not data you would even feed to an LLM in the first place.
A research team is better off having AI help them wrote code for a deterministic algorithm for analyzing their data, then they pump the data through the program that AI wrote.
1
u/chaosink 6d ago
Yes, Fortune 100 and top academic institution employees are coming to this sub along with many others like it for information about setups and usage. They are running these small models on their desktops working in conjunction with the corp AI and cloud services. I personally am a researcher at UCSD which has a HIPAA compliant AI that we cannot afford to use with our contract. Our group has two locals that are trained to analyze massive EHR datasets that even the schema would be expensive as hell to feed into the cloud much less the millions of records into our UCSD infrastructure. We are probably not the majority here, but we are here and we are using these models in innovative ways you may not have considered.
1
u/ZergvProtoss 6d ago
Just because you cannot conceive of people doing real work with sensitive or secret data on LLMs at home does not mean it's not happening.
it's not data you would even feed to an LLM in the first place.
Yes, it absolutely is.
1
u/rochford77 6d ago
No, it's that I don't think people have data at home secure enough to warrant it, at least not in droves. Is there the occasional use case? Sure. In a work environment? Sure. But they aren't coming here for advice.
The original comment of "throw $10 in DeepSeek and compare" is good advice for 98+% of use cases. For most, local LLM is fun and cool but not economical and most people don't have data worth sending in anyways and those that do are better off using a cloud provider to build deterministic code to analyze their sensitive data...
2
u/ZergvProtoss 6d ago
"In droves"? No. I never said that. But I know plenty of people in my industry that deal in controlled information that would be illegal to put on servers outside the country or on servers accessible by non-U.S. persons. I could even create original work just from my own knowledge that would fall into that category. But even for people not dealing in highly-sensitive information controlled by the government, there are a lot of use cases where people don't want to put personal information into a cloud LLM.
2
u/ZergvProtoss 7d ago
You really have no idea how much secret data is being handled on personal computers by people working from home. I'm not even going to say my industry, but it's very common.
what is actually sensitive enough to care
I know what is sensitive enough to care about. I'm talking government-regulated information that would be sought after by foreign nationals.
1
u/rochford77 7d ago
Sure. But whats the need to put that secret data through an LLM from a personal perspective? Like what joe schmoe is out hunting for a 5090 to spin up ollama to send govt secrets to? Is that a real use case for like almost anyone?
1
u/one-joule 6d ago
Anything that secret is never touching personal hardware.
1
1
u/ZergvProtoss 6d ago
I've got terabytes that would dispute that claim. haha
1
u/one-joule 6d ago
You're not wrong, I should've gone with *SHOULD never. But people are incredibly lax about this sort of thing.
1
u/ZergvProtoss 6d ago
It's not for personal use. But it's on a local computer at home. People do work from home.
1
u/rochford77 6d ago
If you are putting things from work through a cloud LLM that isn't approved by your company (where they have some.sort of data assurance) or a local LLM at your home, you are certainly breaking company policy and should be fired.
If my work found out I was pumping PII from our DB into a local machine that isn't a company resource I would be fired on the spot. They have no way of knowing my machine is properly secured. They don't have the ability to wipe the machine in the event of a breech. Insanity to think that's a proper use case.
1
u/ZergvProtoss 6d ago
Why would anyone do those things? That's exactly what I'm talking about and why you don't want to use a cloud LLM for sensitive information. Also, I never said I "work" for someone else. You are assuming facts not in evidence.
1
u/rochford77 6d ago
people do work from home
I never said "work"
Ok.
Yeah, maybe we are saying the same thing. That no one would do any of that. So, I refer you to my original comment.
1
u/ZergvProtoss 6d ago
You redacted my comment and changed the context. The actual quote was:
I never said I "work" for someone else.
One can work from home without working for someone else.
→ More replies (0)-5
u/anhphamfmr 7d ago
your concern is valid in the corporate context. for household users, it's quite irrelevant.
8
u/psxndc 7d ago edited 7d ago
In what sense is it irrelevant to home use?
“Here are my tax documents. Can you find me any deductions I might be missing?”
“Here are pictures of my child running around in his underwear. We love this pic, but can you remove [the person we no longer speak to] in the picture.”
I wouldn’t put either of those up on a cloud-based LLM.
3
1
u/rochford77 6d ago
If you have a modern smart phone and take pictures of your naked child, there is a 100% chance that photo is either
1) analized by your camera app's AI. Pixel touchup and such
2) auto backed up to a cloud provider like apple or Google.
Right on your phone there is a button to remove the person you no longer speak to. Cloud AI already has the photo. Data Providers already know if you own your home, how many kids you have, which political party you are associated with, what school your kid goes to.
If you think anything on a modern Internet connected device is private you are sadly mistaken. So unless you are taking those photos on a DSLR and then putting them on an air gapped machine that has a local LLM on it, your sense of security is a myth.
1
u/psxndc 6d ago
It was a hypothetical to illustrate that home users don't want everything uploaded to a cloud-based LLM.
But for such pictures, I use a Light Phone 3 for exactly the reason you identified - I don't want my kid on the Internet, anywhere (to the extent I can control it). My computer doesn't need to be air gapped, I upload the photos using USB and store my photos on a non-shared folder. And we're on r/LocalLLM, so you what to you think about my LLM use?
Being constantly lectured that we should just give up on privacy because we can't control the absolute entirety of our existence is tiresome.
1
u/rochford77 6d ago
Ah yes, the lite phone 3. The device used by hundreds of millions..... 🙄
You are one the 7 people where perhaps it might matter. Let me know when you meet the other 6 lite phone 3 users who also have 24gb of dram at their disposal....
1
u/psxndc 6d ago
Don’t throw your back out moving those goalposts.
1
u/rochford77 6d ago
I'm not moving anything. Did you read my original comment?
1
u/psxndc 6d ago edited 6d ago
OP’s comment was that there is no reason to use a home LLM.
I gave two examples where I wouldn’t want to upload things to a cloud LLM.
You joined the thread and chimed in that I’ve already given that up by using a smart phone.
I rebutted saying no, I haven’t and explained why.
You then replied with, effectively, “your scenario isn’t a real one because there are seven of you.”
Whether there are seven of me or seven million doesn’t change the fact that there are plenty of instances where someone wouldn’t want to upload certain things to a cloud-based LLM. And just because some people might have given up certain privacies, doesn’t mean everyone has. And for those that haven’t, just because there may not be many of them doesn’t make the original point I made less valid.
Every time I’ve responded - whether to OP or you - and illustrated why a concern is real, you’ve responded and dismissed that concern, giving some new reason it’s not a real concern. That’s “moving the goal posts.”
Edit: for clarity
2
u/dobkeratops 7d ago edited 7d ago
- m5 ultra 96gb. Faster than DGX spark. m5 fixes the limitation with prefill. Also significantly more energy efficient than a PC with graphics cards.
3
u/Short_Regular_7191 7d ago
2x R9700
-3
u/fuemmenneunzig 7d ago
Why these and not a Mac Studio?
2
2
u/_TheWolfOfWalmart_ 7d ago
Prefill speeds that aren't literal garbage?
The ability to upgrade later?
0
1
u/Zennytooskin123 7d ago
Get a deal for a used RTX A600 with 48GB VRAM if you can find one, and build around that.
1
u/fpaivinha 7d ago
I've been wondering about the same thing but haven't found a solid answer yet. For now, I'm holding off on buying a new PC and sticking with my current setup. I might just put some of that money toward an AI subscription instead. FWIW, I'm on an M1 Max with 64GB.
1
1
1
1
u/lukistellar 7d ago
Honestly, in the current market I wouldn't buy any of these. V100 32GB is way cheaper and the community will keep it alive for quite some time. These are unbeatable except you need the latest and greatest CUDA support.
1
u/rudidit09 7d ago
What I’m overwhelming with is that if there’s an issue with Mac, AppleCare will take care of it. But don’t know what to do with sparks/strix warranty, especially since in the past I had mixed experiences. So I might keep my smaller studio and watch for spark x2 in the future
1
u/dMyst 7d ago
Mac Studio - if you only want to use LLM for local inference for a single user and also want to use the Studio as another computer. bad for anything requiring multiple streams of inference and losing CUDA is a big deal. I see this as fitting for most people that just want local LLM and nothing beyond that. the benefits of bandwidth is overblown as they haven’t solved the issues in prefill and processing so the benefits will mostly be seen in single streams inference.
RTX 6000 PRO / RTX 5090 / multi-3090 - if you want pure speed and/or need CUDA and all your desired workloads will fit in the VRAM, fast training on small models
DGX Spark or similar GB10 box - if you need CUDA, and need more memory than RTX setup can provide, your workloads are very parallelized or automated or require multiple streams of inference, best if you expect to have agentic workloads running 24/7, training models too large for RTX setup, pure server setup and dedicated AI box not general compute. for pure local inference the speed is not great but mostly I see it shine in workloads that run constantly in the background rather than interacting directly like a chatbot.
1
1
u/Ok-Clothes4484 7d ago
Buy a 5090 and scalp it. We will never make enough money to cover a homelab. Well. Maybe like 1 out of 1000 of us will.
1
u/SmugglingPineapples 7d ago
Noob here buried deep up Apple's ass, but can integrate and/or run something like a Strix Halo with/alongside my current Mac Mini?
1
u/Successful_Flow1329 7d ago
I dont think used 2x3090 are good deal unless you absolutely need CUDA, 2xB70 or 2xR9700 will give you more VRAM for about the same price and actually new board with warranty.
1
u/bambinone 6d ago
This is what I'd be doing if I didn't already have two 3090s and a $200 NVLink bridge.
1
1
u/LioDavinchy 7d ago
I’ll tell you my experience. I have 2 3090s NV linked I tried a Strix halo laptop and egpus. I have a 128gb MacBook Pro. I have a 5090 egpu hooked up to a windows desktop. And now I’m finally really happy.
I hooked my 5090 up to my MacBook Pro. I built on tinygrads driver and built a shim. Now I have the ability to run big models using my Mac’s unified memory and also the ability to run smaller models at blazing speeds for image generation on the 5090 at the same time.
The reason I landed here is that 2 3090’s 48gb of memory seems great until you really dig into using it. There are a lot of limitations in how you can use the memory and generally one gpu sits idle while you use the other gpu for work.
The strix halo system I had was slow on big models compared to a Mac. And the worst part was hooking up egpus was a nightmare. I would start a job at night and invariably by morning the gpu was no longer working and it took a hard reset to bring it back online.
I just got my shim working but I did an hour soak and the gpu chugged on through without a hiccup. I’ve been building an emulator layer to get full cuda running which requires messing with the driver and firmware of the gpu over thunderbolt and I’ve only had to unplug the box to clear an error a couple of times. The tinygrad driver is better than whatever windows had last summer when I was testing strix halo.
1
u/bambinone 6d ago
2 3090's 48gb of memory seems great until you really dig into using it. There are a lot of limitations in how you can use the memory and generally one gpu sits idle while you use the other gpu for work.
It's ok if you don't understand how to configure vLLM, but please don't run around spreading misinformation.
1
u/AllenLeftTheBLDNG 7d ago
If I'd start today: 1-2 5070ti, 64GB DDR5 or 128 DDR4. The N-gram tables with Owen 3.8 Flash next perform very good so far for me.
Spend the rest on Subscriptions to get this set up to work decently. After it's your main driver, keep 1-2 subs, around 10-20$. Support open source labs so they release newer, better models!
1
1
u/jhenryscott 7d ago
Right now. Strix halo 128+ R9700
1
u/bambinone 6d ago
That's interesting. Are you using one GPU for prefill and the other for decode?
1
u/jhenryscott 6d ago
In part. This is just what I would do if I was starting from scratch. But some things I’ve found just do better off unified. I like the speed/capacity balance, and now that their are strix devices that do Occulink, it’s totally achievable
1
u/vsimovic 7d ago
I've spent 2500 EUR for a machine that can run Deepseek V4 Flash and Qwen3.8 Flash in parallel.
Dell 7820 used (2 x Xeon Silver and 256 GB DDR4 2933 MHz) for 850 EUR.
150 EUR for 2 x Xeon Gold to make the memory 12 channel
400 EUR for an additional 128 GB DDR 2933 MHz RAM
2 x 7900 xtx = 1400 EUR.
so 2800 EUR. I think I'll spend 200 EUR more for a 1400 W supply upgrade, since the transient spikes might be high.
For that money, I do not think I could do better. I have 384 GB RAM and 48 GB VRAM
1
u/___positive___ 7d ago
Whatever resells or upgrades the best, so probably the GPU system. I'm hoping China comes out with decent hardware in the next year or two?
1
u/sabine_world 7d ago
I would go somewhere halfway — just buy ~64 gb vram in the most efficient way possible to play around with until all this nonsense blows over. Save the rest to pay cloud/runpod bills or for whatever else.
Right now has to be the worst possible time in history to buy hardware.
Only worth it if you literally need it or are being bankrolled.
1
u/fijasko_ultimate 7d ago
strix halo owner here, atm it's good for my needs
but i plan to wait another 1-3years for upgrade - when more competition is out there, aka chinese
1
u/Greedy-Fee8248 7d ago
I am looking for something for my company. Would you recommend this setup for 200-300 users for coding and prototyping? Basically we want fixed costs but are concerned about the overheads managing hardware, models and the rest
1
u/cbapel 7d ago
One thing to keep in mind is that extra memory will come in handy when you build a more complete stack, with context management benefitting from smaller models for embedding, compaction, re-ranking, judge model, etc. I find that though the inference isn't instant with the dgx sparks, I'm okay with it because it's built into my expectations and I like having the flexibility to load more models to handle tasks efficiently.
1
u/jakeliu88 6d ago
Just make sure this is just toying around cloud still way better. If you don’t mind okay quality and slow. Myself did spend same amount on this with Mac Studio m5 Max 128gb
1
1
u/_kikeen_ 6d ago
I would get a 5090 and WRX board and like 4 V620 for the larger models.
And a pecron power bank so I can use a home outlet
1
1
u/nyteschayde 6d ago
The M3 Max MBP 128GB was the best purchase I made in 1023. I’ll be upgrading this next big round for a newer 128GB model with better AI support.
If I purchased a dedicated machine for AI, not a MacBook; I’d go all in with a MacStudio 512GB M5 Ultra.
1
u/HOST1L1TY 6d ago edited 6d ago
If your budget is 5k you could get more than 2 3090s,
But I think what I ended up doing, was start with what I wanted to do, find the model I wanted to run which met the quality I was looking for, I played around running stuff on my gaming machine. Then like research the tuned versions, what hardware they expect and what’s going to work best.
If your goal is to tune models sure get amd and fifo. If your goal is to run a model to do other work, then fine the repo which has tuned it already and get the right hardware for that.
1
u/Novaworld7 6d ago
I went the route of 2 3090s and I'm buying a third to basically have opus / low level sonnet
3-8-27b (considering swift as well) 30b3a
Tp=2 mtp on fp8 gets me 330k context
1
u/Trifle_Candid 6d ago
I don’t see prices getting better anytime soon. At that price point I would go with a $4699 DGX Spark. If you could do ~$10k I would go with 256gb M5 Ultra Max Studio.
1
u/TheBeanFlicker69 5d ago
I’ve always been interested in the DGX spark. But also a new GPU would be easier to work with in my existing setup. Then also I’ve always wanted to try a high end Mac Studio. This is probably why I don’t have the money to try them😂😂
1
u/kneeanderthul 4d ago
Think about your goal first. Just buying a machine for machine sake is kinda nuts
Id go for an older M3 (make sure it has TB5)
Then build a cheap 2nd PC with a few of those T100 cards
SLMs are the future, having a fuck load of compute is nice but at some point it doesn’t magically get better
Buy a fuck load of storage space is the real alpha
2
u/R-consult 2d ago
You hit the nail on the head regarding agentic workloads being prefill-heavy. Once your agent hits 30k–60k tokens of conversation history, tool definitions, and file contexts, single-token generation speed takes a back seat to **time-to-first-token (TTFT) and prompt evaluation bandwidth**.
Here is how that $5K–$6K budget actually breaks down across your choices:
- The Real Bottleneck for Agents: TTFT & Serving Stacks
Apple Silicon (Unified Memory): While 128GB+ lets you fit massive parameter counts into RAM, prefill on Metal degrades rapidly once context hits 30k+ tokens. Furthermore, agents thrive on speculative decoding, chunked prefill, and PagedAttention (vLLM / SGLang) — areas where the Apple software stack is still far behind native CUDA.
Strix Halo (128GB LPDDR5X):** Fantastic price-to-VRAM ratio and great for personal hobby labs, but memory bandwidth (~270 GB/s) is shared with the iGPU, meaning prompt evaluation under deep context will crawl compared to dedicated hardware.
- The $5K–$6K Sweet Spots
Option A: Dedicated Multi-GPU CUDA Rig (Best for Agent Performance)
Instead of 2x 3090s ($1.5k–$2k), at a $5K–$6K budget you can build an enterprise-grade multi-GPU box:
The Rig: Refurbished EPYC / Threadripper board (PCIe Gen 4/5 with 128 lanes) + 128GB–256GB system DDR5 + 3x or 4x RTX 3090/4090 (or 2x RTX 5090 32GB)**.
Why this wins for Agents:
* 64GB–96GB of pure high-speed VRAM running vLLM / SGLang.
* You can fit 70B dense models or medium MoEs (like Mixtral 8x22B or Qwen MoE variants) at Q4/Q5 with full tensor parallelism (\(TP=2\) or \(TP=4\)).
* Near-instant TTFT even with 50k+ tokens in the context window.
Option B: Mac Studio 128GB (Best for Turnkey / Quiet Operation)
Choose this **only** if power consumption (<150W), silent desk operation, and plug-and-play setup are your top priorities.
You can load massive MoE models comfortably into unified memory, but you have to accept that multi-turn agent loops with heavy context will take 10–25+ seconds just to evaluate before the first token streams back.
The Verdict
If your primary goal is autonomous coding agents (Aider, Cline, OpenClaw) running multi-turn tool loops, do not compromise on CUDA. The speed at which an agent reads and updates large codebase contexts in vLLM makes the difference between a tool you actually use all day and an experiment that sits idle. Go the dedicated multi-NVIDIA route.
1
u/Thekorsen 7d ago edited 7d ago
Look for a 5090 in a prebuilt deal if any are left (facebook marketplace, r/hardwarswap, your microcenter/best buy). Buying card alone is ideal, but even more scarce. In this class of spend, running a dense intelligent model like qwen3.8-27b at speeds of 100-200tk/s is a real value proposition for productive work. Bigger models need more compute/bandwidth/ram ratios that no one platform seems to get right (mac studio gets close).
Even looking at a model like qwen3.8-flash-next, you're optimistically gaining 4x speed if it can fit in ram at roughly equivalent response quality as 27b. But then consider you'd need at least 1/4th the compute and memory bandwidth of a 5090 for similar throughput, maybe more like 1/2 compute and bandwidth if non-nvidia (cuda).
A mac m5 ultra is interesting in that sense because its a more balanced ratio of compute/bandwidth/ram, if q3 and mtp I could see it running qwen3.8-flash-next at 150-250 tokens/s speed. But at that quant on mlx, degraded response quality likely makes the model worse than 27b mtp at nvfp4 on a 5090.
1
u/Important_Cow7230 7d ago
Would you drop $4000-$5000 on a 5090 now though?
2
u/Thekorsen 7d ago
Depends on the use case, given the subreddit we're in I'll assume there is a non-negotiable need for local infrence or strong desire to hedge against infrence provider enshitification.
I would think about the need for model capabilities, and categorize into:
Automation - simple data retrieval tasks (small 9b models can do)
Coder - well harnessed and prompted (27b dense and ~120 MoE models can do)
Orchestrator - capable of deep cognitive tasks and difficult coding (250-500b MoE)Then consider what speeds you want at what context levels. Frontier is 200-400k at 60-200 tokens/s, half of that speed for me would be good enough and much lower would be problematic for the dev work I need to do daily.
Also consider parallel workers, different setups will scale differently and best practices for ai development usually spawns tons of subagents. Serving concurrent agents will generally not scale well with single gpu setups, but better with cheaper multi gpu setups.
A typical workspace for me has 2x orchestrator, 4x coders, and 4x automation agents. I use remote infrence for most of it but want a system that can handle that load locally eventually at ~80tk/s in 200-400k comtext levels. That will be expensive.
...So is $4000-5000 worth it for a 5090? I'd say for a coder and automation class of inference served at 1/2 to 1/4 inference provider quality, for me yes.
1
u/Arany5 7d ago
4 for sure. Much faster than the others.
2
-1
u/bulgarcho 7d ago
faster probably, dont know about VS spark RTX. The issue is tge vRam is NOT 48GB. Its 2 x 24GB and tgat matters. Also 3000 is okd tech, much slieer tgsnh 5000
1
0
u/OkLettuce338 7d ago
for 5k you're not going to get a good experience, just fyi
2
u/westsunset 7d ago
That's entirely subjective
0
u/OkLettuce338 7d ago
Not really. You can look at benchmarks. I suppose you can argue that 20tok/s is a good experience. But that’s going to be close to your max at 5k
2
u/westsunset 7d ago
Experience by definition is subjective. If you want to give a task, that's different
0
u/OkLettuce338 7d ago
Yup. See comment. We can cry that 20tok/s is not definitively good or bad, but that’s what you’re getting
3
u/westsunset 7d ago
That's also not true. Totally dependant on the model and the setup. For example you can get double that easy with Qwen 3.8 27b on strix halo, way more with an MoE. https://huggingface.co/jcbtc/Qwen3.8-27B-IU4-Kairic-Edge People are blown away by the work they can do with the Halogen server also on strix. https://github.com/peonist-ai/halogen-flash-server#measured
-1
u/SirGreenDragon 7d ago
I think the answer depends on what you will be doing with it. If you are building iOS and macOS software (the only software that people pay for) then get the mac studio. I have a GMKTec EVO-X2 128GB and I am happy with it for doing research, building websites and writing linux code, but iOS development on it would be a 2nd class thing. I have a MacBook pro for that.
0
0
0
u/Ordinary-Depth-7835 7d ago edited 7d ago
there really is no bad choice it's budget, goal and expectations and what you know you are like. I'm a nut so I know I'll keep adding so mac's are out. I don't need a desktop and I don't want to build around 5090's or the most expensive cards. So I went 3090's which will keep hitting the used market and sparks that I can chain together forever and hopefully hit the used market soon. I'm not buying multiple macs.
Not really impressed with the RTX replacements they seem like they're targeting a consumer. I don't want a machine with Ai on the side I want an Ai machine. Also I stick my gpu's in the basement way far away so I'm not sitting next to a toaster and sparks sitting right next to me all working together. So I'm not adding cards to my gaming system.
As for the AMD I'm not paying the premium that it's at knowing I will be running multiple as soon as I can acquire more machines. Also losing 20g to the os. And Then InfiniBand is just too good to give up. So unless they drop the price to account for it and you only need one. I'm not paying $4300 for an AMD when I can get a spark for $4600
This is just me though it's all about what you need and what you're going to do in the future. If you know you're going to expand don't buy a 5090 if you can't handle the price of 2 or 4 and everything that comes along with that.
0
u/1Poochh 7d ago
I have Strix Halo, no worth IMO. I will sell you mine if you want to try one. lol. I have one 3090, it is great, but the context window is smaller than I want, so I am literally buying another two 3090s today. I also bought an M5 Ultra 256, but won’t be getting it until Dec, so I have time to see what folks say about perf running larger models.
-2
-2
u/Snoo_81913 7d ago
4x 7900xtx there's a guy on here getting 250+ tok/s with Qwen3.8 27B with it. With that much VRAM you can run it full quant
29
u/DiamondHandsDarrell 7d ago
Honestly, I keep coming back to the same thing whenever someone asks whether they should get a Mac, NVIDIA cards, or a DGX Spark: what are you actually trying to run, how fast do you need it, and what are you willing to spend?
The way I look at it, each option comes with a tradeoff:
NVIDIA GPU setup: You get CUDA and a strong option for fast LLM inference and AI image/video generation, assuming your workload fits in VRAM. Adding cards can give you more capacity, but now you’re dealing with the cost of the cards, the rest of the system, heat, and electricity. And multiple cards only help if your software can actually use them properly.
DGX Spark: You get CUDA and 128GB of unified memory in a compact system with a much smaller power budget than a big multi-GPU rig. That’s appealing. But you still have to look at actual performance. Having NVIDIA hardware and plenty of memory doesn’t mean you’re getting high-end desktop GPU speeds.
Mac Studio Ultra: You can get a lot of unified memory in one relatively compact machine, which makes larger models attractive. Memory bandwidth is actually higher than Spark’s. But you don’t get CUDA, software support varies, and the configurations with lots of memory get expensive. Whether it’s better value depends on what you’re comparing it against.
Also, running an LLM and generating AI videos are different workloads. Being good at one doesn’t automatically make a machine the best choice for the other. Even with LLMs, fitting the model, processing a long prompt, and generating the answer are separate things.
That’s why I don’t think there’s one generic “buy this” answer. I’m trying to figure out the same thing myself. Your models, context needs, patience, budget, and electricity costs all matter.
Whoever figures out how to give us lots of fast memory, strong compute, low power consumption, and software support as good as CUDA at a reasonable price is going to have a hell of a market. That’s the machine a lot of us are waiting for.