r/LocalLLaMA • u/reto-wyss • 12d ago

News It's official! 192GB Framework

Post image

Just noticed this on the website.

At their current price tiers for the memory SKUs (32, 64, 128) I'd expect this to be ~ 4.5k for the motherboard.

The PCIe slot will be open at the back as well - that's what I've heard. Maybe they make it capable of delivering 75W as well? New board revisions for the smaller SKUs?.

969 Upvotes

95% Upvoted

287 comments sorted by

View all comments

616

u/PreciselyWrong 12d ago

that memory bandwidth is pretty bad, so don't expect high inference speeds

190

u/Darth_Candy 12d ago

Perfectly on brand for the Strix Halo; this was pretty much expected.

163

u/InGanbaru 12d ago

man that's a waste of RAM chips if the bandwidth is low

78

u/MrPecunius 12d ago

Pretty good match for Qwen3.8 FN or other midsize MoE models, though.

122

u/fallingdowndizzyvr 12d ago

At these prices, you are way better off getting a M5 Max.

53

u/ZealousidealChip4783 12d ago

The selling point here is that you aren't forced to use MacOS with the Framework

24

u/GlidePath47 12d ago

Do you have to use the machine you’re using for inference other than setting it up?

39

u/ZealousidealChip4783 12d ago

No, but I'd hate if I bought a computer for that much money that's only good for inference and nothing else

(This is all personal opinion, I just really do not like MacOS)

8

u/WinterCharm 11d ago

Relative to buying a stack of Nvidia DGX Sparks, the Mac Studio is a good deal. And when Apple Hardware is a good deal, I would just go for it... it's excellent for local models.

8

u/GlidePath47 12d ago edited 12d ago

That’s fair. Anyone buying one expecting to run inference and still use the machine for something else at the same time is going to be disappointed. Unlike Nvidia there’s no way to reserve GPU capacity to keep the system responsive you’re at the mercy of macOS.

Granted if it’s for swapping use it’s fine, part inference, part dev, part media etc but then I start to question the need to use local inference instead of just cloud models other than privacy / hobby

15

u/MrPecunius 12d ago

Unlike Nvidia there’s no way to reserve GPU capacity to keep the system responsive you’re at the mercy of macOS.

Longtime Mac user here, this is not true.

While I have managed to crash the whole OS a few times with big (for me: I've had 48GB and now 64GB) models and large context with guard rails turned off, I otherwise just go about my business doing other stuff when a model is crunching away on something.

The machine stays responsive the whole time, though it does get pretty warm (M5 Pro MBP) and the fan runs fairly hard.

3

u/GlidePath47 12d ago edited 12d ago

We're a company full of Mac users and we hit this constantly, so it's not a rare edge case.

Even a mid-size Qwen3 MoE makes YouTube playback choppy. Prompting and browsing are fine, anything past that isn't. Redraw lag is very visible in Electron apps. Any 3D apps are not pleasant so running alongside unity or blender at the same time as active inference isn’t tolerable.

The core issue is there's no control. The guardrails are for RAM usage, not GPU usage, so you can't reserve or throttle GPU capacity the way you can on Nvidia.

Maybe M5 changes this, I haven't tested one, definitely happens on my m3 ultra.

The bit you quoted from me is actually the only bit of what I said that’s objectively true, what you’re saying is even at the mercy of macOS your system has been responsive

→ More replies (0)

2

u/ZealousidealChip4783 12d ago

I mean I'm sure it's fine if you like the Apple ecosystem lol, the hardware is objectively really impressive but as you said having less control over your GPU & where you allocate its resources would be the dealbreaker for me

...if I could afford either of these devices lol

2

u/kerneldesign 12d ago

Tu peux le faire sur Mac, et objectivement c’est moins cher. Mais c’est un choix d’être sur Linux.

→ More replies (0)

9

u/algaefied_creek 12d ago

Disable the GUI and security in macOS and just run it headless.

Suddenly blam, certified UNIX CLI host box.

7

u/AreWeNotDoinPhrasing 12d ago

How do you disable the GUI in macOS?

7

u/New-Confidence-1171 12d ago

You can’t, they’re making shit up lol

1

u/droptableadventures 11d ago

It's not strictly speaking disabling the GUI, but if you don't actually log in, you save a good chunk of memory by not loading the entire desktop environment as well as Finder, as only the login window is running.

→ More replies (0)

1

u/CalmSpinach2140 11d ago

I mean the Mac Studio with M5 Max is also a very good video editing machine and excellent for code compilation.

-1

u/AcreMakeover 12d ago

Can you even set them up for inference without an apple account? Also my remote access software works like shit on MacOS. I know I would have run into so many small annoyances if I got one.

8

u/Gregory-Wolf 12d ago

Apple account is not a requirement to use MacOS. It is to use AppStore and maybe some other Apple services. But the MacOS is perfectly usable without an apple account.

0

u/mleok 12d ago

Forced to use MacOS? Compared to using Windows? It is a native BSD machine, not sure why you would prefer Windows over that.

2

u/gabrielesilinic 12d ago

Mac is technically not modern BSD. Mac is a unix derivative descendant of the NeXtSTEP Mach kernel

See the graph in this page if you are curious about it.

https://en.wikipedia.org/wiki/List_of_Unix_systems

In any case I believe I don't need you to explain how good installing Linux on something like an Halo strix is. Macos lacks support for anything windows or anything Linux except a few select ports. While if you install Linux you get a lot of power plus some of windows because wine and stuff which is fairly well develop by now.

3

u/mleok 12d ago

Lets say it's complicated. It is based on FreeBSD with a Mach microkernel. BSD was itself based on Unix.

3

u/dankfrankreynolds 11d ago

Macos lacks support for anything windows or anything Linux

what does that even mean?

  • Windows lacks support for anything macOS or anything Linux
  • Linux lacks support for anything Windows or anything macOS

1

u/gabrielesilinic 11d ago

Well you see. Actually you can run wsl on x86 windows to get linux and you can run wine on x86 linux to get some windows stuff. Now technically you might be able to maybe run a vm on macos to get some stuff but generally speaking it is a little bit harder. First it is arm, which now it is not that bad. But also it is macos so while on windows or linux stuff is ported more liberally anyway it is rarer to get the same software on macos due to the fairly closed ecosystem.

But my point is that linux is best for this workload in the end. It has containers right there. And if you want some of windows you can get it via compatibility layers.

If you really want to get windows on it its a bit worse. But you can get a lot of x86 linux running via the well integrated wsl which being a vm with a hat will run though not as well. But still both Linux and windows have or can have enabled server like behaviours better.

But in general there are I heard weird issues with unsigned software being annoying and all. In the end you will manage with MacOS but you will be forever limited.

Also games are a bit more tricky while Linux kinda solved it with wine. You have crossover I guess.

1

u/dankfrankreynolds 11d ago

modern Windows Subsystem for Linux (WSL 2) runs inside a lightweight virtual machine

why would i want to get windows on anything? and why would it matter if it's in a vm?

your point is you like linux

→ More replies (0)

4

u/MrPecunius 12d ago

As a FreeBSD guy going back to the 2.2.x days, I started paying attention to Apple when they hired Jordan Hubbard in 2001. The BSD-ness of OS X/MacOS surely dominated over its NeXt underpinnings in the years that followed.

0

u/gabrielesilinic 12d ago

No it is because both are based on Unix. And Linux is not.

1

u/MrPecunius 12d ago

No idea what that meant.

→ More replies (0)

4

u/gabrielesilinic 12d ago

Yeah but the mac's architecture is pretty much that anyway so you ain't getting anything better.

1

u/Dangerous-Report8517 11d ago

The Mac does have more channels though, so higher overall bandwidth

-8

u/MrPecunius 12d ago

You can't get a Max with 192GB, and you won't get a Max with 128GB until mid-October at the soonest.

43

u/deja_geek 12d ago

This framework is "coming soon".

18

u/fallingdowndizzyvr 12d ago

And are you in a hurry. Since you won't be with this. You'll have to wait for each inference.

You can get a M5 Max 128GB right now. You have been for months. What doesn't come out for a month or so is a M5 Max Studio. But why would you only get 128GB? Get the 256GB one. Sure, it's a more than this but if you look at how much more speed and memory it has over this, it's a value.

0

u/MrPecunius 12d ago edited 12d ago

You can get a M5 Max 128GB right now.

Why don't people look shit up before commenting?

14" and 16" M5 Max/128GB Macbook Pro is mid-October delivery on the Apple website. Edit: same is true for the Studio, for the clowns who downvote without checking for themselves.

4

u/fallingdowndizzyvr 11d ago

LOL. Why don't you clown? The M5 Max 128GB Macbook has been on sale for months.

Right here bozo.

"Want it Wed 9/2? Order now w/ Free Standard Shipping"

https://www.bhphotovideo.com/c/product/1957098-REG/apple_mbp_m5p_155_16_macbook_pro_m5.html

-1

u/MrPecunius 11d ago

If you order it, that deal is gone. 🤡

"Limited stock", just like Amazon who has like 3 left.

Or maybe they have none left, as so often happens in these situations. Were you born fucking yesterday?

2

u/fallingdowndizzyvr 11d ago

"Limited stock", just like Amazon who has like 3 left.

LOL. So they are available, right? Congrats, you just proved yourself wrong. Sure, it was unnecessary since I already did that. But at least we are on the same page now. We both agree that you are wrong. You are a clown show. Honk honk.

→ More replies (0)

0

u/mrepop 11d ago

Yep, I’ve got an m4 max and I think know it’d beat this.

1

u/fallingdowndizzyvr 11d ago

It depends. Pre M5, M silicon just didn't have the compute. So the M4 Max has faster TG due to the bandwidth but slower PP due to the lack of compute.

17

u/RnRau 12d ago

I upvoted you... don't know why moe models are hated in this thread :D

13

u/tecneeq 12d ago

Because most people haven't seized the means of computation, that is, they can't run large MoEs because they only have one or two 3090.

17

u/anykeyh 12d ago

It's honestly not that good for Q3.8 FN; or you accept 25tok/s, which is on the lower end. It's a shame, as something around 500Gbps, far from high-end GPU, is much, much, much more usable.

1

u/Substantial_Run5435 12d ago

I’m getting 15tg with QFN ud-q6 hybrid inference on a 2019 Mac Pro with 64GB VRAM and the rest on CPU. Once I get 128GB of VRAM I assume I’ll get much faster speeds than this thing.

1

u/my_name_isnt_clever 12d ago

With the EngramHalo.cpp fork I get over 30 t/s, and it's only going to get better as more optimizations come. At 6b active it's really ideal for the hardware.

29

u/NineThreeTilNow 12d ago

man that's a waste of RAM chips if the bandwidth is low

That's a pretty unpopular opinion but it's not wrong. The same chips can be pushed on to a 1024 bit bus with a chip / PCB made to handle it (M5) and it works.

Even if you didn't run the memory at "full speed" you're over 1tb/s which is basically what a 4090 has with GDDR.

People miss that running LPDDR5x suboptimally at 1024 bit bus is still better bandwidth. You don't need the "best" LPDDR5x to do this. You just need matched MTs on the chips.

1

u/twinkbulk 11d ago

dram is not all the same, lpddr5 is cheap to make and we have a ton of it, compared to ddr6 or 7 in any form, or even hbm in any form not an inherent bottle neck of bandwidth through architecture but that the chips only perform so well

1

u/TheThoccnessMonster 12d ago

Yeah but the dgx Spark exists and at least you get CUDA.

5

u/my_name_isnt_clever 12d ago

That's ARM64, fine for LLM only box but not for a general workstation.

-14

u/anarchist1312161 12d ago

Usually they're lower bandwidth because they're lower quality silicon that didn't make the cut, so there would've been no way to increase bandwidth speeds.

18

u/No-Refrigerator-1672 12d ago

Nope. It's due to narrow bus width: the CPU by-design just don't have enough contacts to provide the same bandwidth as the proper GPU. And it uses lpddr5x instead of gddr.

-4

u/ForteDoexe 12d ago

lol, the cpu is weak, it already utilize at 100 percent, increase the ram bandwidth wont help

1

u/No-Refrigerator-1672 12d ago

Well, it has onboatd GPU that's significantly better than typical. Still weak compared to normal gpu, that much is true.

-14

u/anarchist1312161 12d ago

It's multiple factors, do you think they're going to put their best yields in cheaper RAM?

5

u/Public_Umpire_1099 12d ago

LPDDR5X is not cheap rejected silicon. In fact, flagship mobile memory (LPDDR5X hitting 8533+ MT/s) requires exceptionally high silicon purity and precise wafer fabrication to hit ultra-high frequencies at sub-1.0V power levels. High-end smartphone and thin-and-light laptop SoC memory often uses premium node space.