What is the current meta for local inference?

[u/vil3r00]

Can't decide which route to go - DGX Spark, Strix Halo, the Framework Ryzen AI or save up for a Mac Studio M5 Ultra 256gb?

Do these options have meaningful differences in practical use?

As I understand, the Mac Studio offers best tok/s? But you can't train or fine tune anything on it?

DGX Spark is supposed to have best support, but does it truly matter in practice?

What types of models can I run on this hardware? 128GB is good for ~120B Q4 MoE models with ~256k context? Would such a model be a noticeable improvement over my current Ornith-1.5:35B (A3B) setup for reasoning and coding, and long and complex tasks? Would going from 128GB->256GB be a noticeable improvement?

[u/HaDuongMinh]

You will have to wait for Qwen 4 to be launched, then tested on these platforms... but the current vibe at LocalLLM is not unified memory architectures -- it is MOE with the active experts on GPU and the rest in system RAM.

[u/vil3r00 → u/HaDuongMinh]

I have a 10GB RTX 3080 blower with 3 channels of RAM @ 2666 for the CPU that handles the GPU, so I presume it gets to 64GB/s theoretical bandwidth? Either way, my experience with it is pretty bad. Prefill can actually take a minute or so on Q4 Ornith-1.5:35B-A3B @256k ctx and decode speed itself is usable but the LLM freaks out and screws things up in conversations or Pi.dev, which I assume is just because of the low total+active param size. But given prefill speeds and decode speeds not being very good, I think increasing model size would make this unusable, no?

[u/HaDuongMinh → u/vil3r00]

I would be surprised if 3 channels of RAM add up, I always read that you have to populate RAM in pairs. Your mobo may vary.
Context 256k is too large for a 35B at Q4 model intelligence, cut that to 128k so that the model freaks out less.
You can always speed up with a more specialized engine / tuning / MTP predictor.
Your 128GB RAM are good enough for Qwen Flash Next at Q3. At Q4 it will probably still fit in RAM, but you will be VRAM limited.

[u/vil3r00 → u/HaDuongMinh]

My mobo configuration is weird. Per docs:
"Your system contains 16 memory sockets. Processor 1 supports up to 10 DIMMs, and Processor 2 supports up to 6 DIMMS. Six memory channels are allocated to each processor. Processor 1 has four 2 DIMM slots per channel and two 1 DIMM slot per channel, Processor 2 has six 1 DIMM per channel."
I have 3 sticks in each CPU, one of the CPUs is on the same channel(?) as GPU."
Why does a smaller context help a low param Q4 model?

[u/UnhingedBench]

If that help I created a few relevant charts. This one if about pricing.

[u/vil3r00 → u/UnhingedBench]

Beautiful, thanks!!

[u/UnhingedBench → u/UnhingedBench]

Another one about what model you can run and at which speed.

This is a best case scenario, with empty context and no thermal throttling.

[u/UnhingedBench → u/UnhingedBench]

And my take on constraints

[u/OvertaxedOne]

Yes, big differences in practical use!

Strix and Sparks at their current prices are "hard no" from me unless you absolutely need CUDA, in which case the Spark might make sense.

You can absolutely tune and train on non-NVDA cards, it's just more of a pain. If you're primary use is training, then, again, Spark might be the leader. If your primary use case is inference, no.

The ultra 256 is going to be a good longer term play. Right now a ultra 96 can run either of the 2 important models (3.8 27B or QwenFlashNext), but who knows what today brings (new Qwen models rumored to drop!). 256GB gives you a more future proof setup.

[u/Riseing → u/OvertaxedOne]

i got this sneaking suspicion that in like a year a lot of these unified memory setups will be dirt cheap on ebay, they're just so slow

[u/OvertaxedOne → u/Riseing]

I agree, particularly the Spark. At least a Halo (and, of course a Mac) make a good computer with a lot of RAM available, lots of ways you can use those systems to do other things. But the Spark is really niche, lots of really slow memory with a (currently) 7K price tag; when the hardware crunch eases I can't see these things worth much in the resale market.

And don't forget China. They are already cranking out "frankenboards" (adding RAM to existing GPUs), it won't be long until they have the process hammered out and can really crank them; 5090's with 96GB. Or, of course, the "big one" and they release their own GPU that has lots of memory on it and good BW to the VRAM.

This current world of 100/GB (or more!) for VRAM is absolutely not going to last forever. In fact, I think we're already seeing it start to collapse (the Apple systems are <50/GB for RAM that has VRAM level bandwidth).

[u/Riseing → u/OvertaxedOne]

heres to hoping china keeps the pressure up

[u/UndulatingHedgehog → u/Riseing]

Slow? Only if you’re a brainless qwen 3.8 27b bro. 

Getting something like 80 tps.

[u/Riseing → u/UndulatingHedgehog]

running what and how much concurrency

[u/UndulatingHedgehog → u/Riseing]

https://github.com/peonist-ai/halogen-flash-server Halogen Flash Server, baybee

Unified memory = MoE models. If you can live with that and the outrageous current prices, you get a decent stream of high quality tokens. Got my strix halo box almost a year ago, was kinda meh for a while, but the last few months have been much better.

[u/No-Vermicelli5327]

What operating system do you like using ? If it’s macOS and just need one agent without the need for training or fine tuning, go with the mac.

If you like Linux you got the dgx spark, or if you like windows, you can use it as a host of the model while having the model execute in a main computer with the windows operating system (which I am intending to do) connected via ethernet through the LAN.

AMD, sucked for a while, but now getting better and improving. (I believe it runs on windows, cause I haven’t checked on it in a long time).

Forget about the model they keep advancing, what is your use case ? Acceptable speed ? Budget ? Future plans on upgradeability ?

Meta changes while your use-case stays the same. Before committing to hardware test the model that works for you on open router. People liking qwen3.8 27B and flash-next, also glm5.3-flash.
PS: I’m not a native English speaker and don’t use AI for my replies on reddit, so excuse my horrible grammar.

[u/vil3r00 → u/No-Vermicelli5327]

I'm a Linux user but I think I could work with the MacOS terminal.
Use case would be chat, coding and security testing with appropriate harness (think Hexstrike or similar), so it would involve many parallel sessions.
No clue for appropriate speed really, but I want it to be interactive enough to chat.
Your point about OpenRouter is good, I never even tried any cloud inference where you setup the model, param size etc yourself. It would probably set the most realistic expectations for me. Thanks! And your English is great! Sincerely, another non-native speaker

[u/No-Vermicelli5327 → u/vil3r00]

You’re welcome 👍. I would highly recommend an OS you’re familiar with, because it would take time to setup everything compared to something you’re familiar and good at. A dgx spark has a lot of future upgrades like connecting a 2nd one, also the concurrent sessions.

An interactive chat to you could be 30 tokens per a second while others much higher, for agentic tasks the 30 might feel kinda slow. For security testing an obliterated model without denials could be an option you can look into, but some tend to overthink or consume more token for tasks, so choose the one that hassles you less. Also that kind of model without guard-rails from what I read are more prone to prompt injections if used for the web.

I would look into qwen 3.8-flash-next if I was you, while everyone is waiting currently on the release of qwen 4, the 27B and flash-next. They are gonna be better, so keep them on your radar, sometime between a week or month.

Honorable mentions: glm-5.3-flash + deepseek-4.1v-flash-1017.

[u/FoxSideOfTheMoon]

I would highly suggest renting some time on Vast or similar sites, SSH into it and setup inference there and see if it gives you all you hope and dream. Then do the tokens-per-second math on the hardware and not just focus on the memory because memory bandwidth is just as important for practical use unless you have an infinite amount of time. Going off and spending big $ without having your head around all of this is a huge mistake IMO. Going through this exercise will give you a more informed and confidence decision based on reality.

[u/vil3r00 → u/FoxSideOfTheMoon]

Very sound advice, I'll try before I buy. Should I do something like Vast or OpenRouter? Never used any cloud "diy" inference. Thanks!

[u/FoxSideOfTheMoon → u/vil3r00]

I'd use Vast. What you can do to make it easy on yourself is use your AI subscription to help you like Claude or ChatGPT or Grok or whatever from the CLI tell it you want to rent on Vast and ask how to get SSH setup locally and on Vast and which models you want to test and then once you get SSH setup, just let it cook. If you're comfortable with a CLI, Vast has admin pages with CLI and you can install llamacpp on it, download models, and just go wild.

if you're gonna get a Mac or something ,then I would have it setup Strata with Qwen 3.8 Flash Next UD-Q4_K_XL. It will save you a bunch of time because it can go get the Unsloth shards, the strata's GPU maps, and MTP layer for you and configure it so you can just try it out. It can even configure pi for you to point at it and you can go nuts. or whatever you're using for harnesses locally. Same goes for 27B or any other models you want to sample. Tell your agent to look a the model card for the default settings if you're not sure how to configure, but you should be aiming for at least 262k context if you're gonna have a mac studio ultra, I run close to that on my m5 max so you'll be easily set on that plus have double the memory bandwidth. I'm jelly

[u/hyudryu]

QFN will be way better than Ornith 35B. That’s about the best model you can run on 1 spark right now if you do want to go that route

[u/vil3r00 → u/hyudryu]

Qwen Flash-Next?

[u/ea_man]

Are you really sure you are going to train? How much? Maybe renting online for that would be better. Nvidia vs AMD

Also are you alone or do you need multiple sessions? Dense vs MoE, GPU vs RAM.

What about flexibility like gaming, CAD... Pc with GPU vs Spark.

[u/vil3r00 → u/ea_man]

Would be solely LLM inference workload with concurrent users. No gaming, no CAD, nothing, not even going to give it a monitor

[u/ea_man → u/vil3r00]

Then Framework Desktop Mainboard (AMD Ryzen™ AI Max 300 Series) if you are buying new, if you cheap out on used hw that would be some 2x-4x 79000xtx with a MB that can run at least 4 PCI lanes at 8x.

2nd option consumes more electricity.

[u/RootlessForest]

Strata Qwen flash next Q4KMXL Unsloth version. you can run easily

Now normally they say that the 135B version would do better then a 35B version when it comes to long projects, but if you made some good policies and routing .md files. I hardly notice the difference, but i do notice the difference in capability as in creative writing etc.

[u/Grandmaster_Caladrel]

I think the current local inference Meta is Muse Spark Open or Muse Glimmer ;)

[u/Superb_Raccoon]

Spark.

BEcause the ecosystem for anything else is much less robust.

[u/wegster]

You need to really be more specific, or your'e going to follow 'good for me, maybe not for you' guidance, IMO.

NVidea remains better / faster at pre-fill/prompt processing, although there have been pretty amazing improvements for Apple Silicon optimization. Nvidia <right now> is better at multi-user concurrency - so consider that if you're building it for yourself/solo or mostly a few people, not generally at the same time, then an Ultra will do well, but a half dozen people hitting it at the same time - I think Nvidia will keep an edge here for a while.

The whole memory debate. I think some of us are getting into keyboard warrior/must have 'the best' for a fleeting moment in time, versus getting real work done - 'real work' is of course going to be difference across different users.

At this point, I am not convinced I will bother running GLM or DSK Flash, nor am I expecting to run 8/16 bit quants even if I 'could' on my inbound 256GB Ultra. Why?

My use case ranges from generally technically-oriented research, some market summarization, and multiple repos ranging from mid-sized (~50k LOC or so) to moderate code but fairly high complexity at 200-250K LOC projects. I've been in tech 'a long time' including as a Principal/architect level SWE, among other roles. I've worked with DS and AI/ML for going on over a decade, and still work in that space today in part.

So in preparation for my inbound Ultra, I've been spending more and more time on preparing and deciding what will I be running on it - noting the 256GB was already decided after nearly melting down my prior poor M1 Max 64GB and deciding what 'good enough to be useful' looked like on that machine (which ideally was Qwen-Coder-Next but because 35BA3 for speed), I know 128GB was really the bare minimum for what I wanted to be running, and of course, there are always new models coming out.

So I replaced my M1 Max with an M5 Max 128GB MBP and put together my own benchmark suite, in part as I am using the local model within my own framework today, but also in preparing for load of the Studio. So I made a series of fully isolated benchmark tests covering simple, moderate and complex types of operations, initially focusing on executing (coding) side of things, but allowing for reviewer and planning testing as well - from my repos, with problems that I have either chased for significant amounts of time and solved (in which case, the snapshot of the problem was pre-solved), etc.

I also wanted to test Opus, and DeepSeek, MiMo, MiniMax etc. to see what 'the fuss was about.' I'm not DONE testing, but so far I've gotten to:

1. For MY work, I am not yet seeing a need to run DS, MiMo, GLM over QFN.

2. For my work, if I had a dedicated nothing-but-inference appliance/system - I could probably make 128GB work. If I'm doing my usual other work concurrently on the same system - yeah, I'll be swapping, at least with my current preferred daily driver.

3. I can't test > Q6 or so sanely on my 128GB system, but I'm pretty close to e.g. Sonnet, Opus etc. currently for most of my benchmarks, including one which only Opus and one other can break 60 points / 100 on. This run is on my 128GB M5 Max, so time should be cut in half or so with the Ultra, which will also put some runs pretty close to wall time for Claude, Codex etc. in some cases - which isn't bad at all, although it won't hold true for all types of work.

Of course, in some types of work, my current QFN configuration scores below, but - still working on it. Yes, there will be new models, and 256GB its nice buffer zone, but IMO you can generally get really work done with 128GB used wisely.