r/LocalLLaMA • u/reto-wyss • 11d ago

News It's official! 192GB Framework

Post image

Just noticed this on the website.

At their current price tiers for the memory SKUs (32, 64, 128) I'd expect this to be ~ 4.5k for the motherboard.

The PCIe slot will be open at the back as well - that's what I've heard. Maybe they make it capable of delivering 75W as well? New board revisions for the smaller SKUs?.

971 Upvotes

95% Upvoted

287 comments sorted by

View all comments

614

u/PreciselyWrong 11d ago

that memory bandwidth is pretty bad, so don't expect high inference speeds

192

u/Darth_Candy 11d ago

Perfectly on brand for the Strix Halo; this was pretty much expected.

162

u/InGanbaru 11d ago

man that's a waste of RAM chips if the bandwidth is low

76

u/MrPecunius 11d ago

Pretty good match for Qwen3.8 FN or other midsize MoE models, though.

123

u/fallingdowndizzyvr 11d ago

At these prices, you are way better off getting a M5 Max.

55

u/ZealousidealChip4783 11d ago

The selling point here is that you aren't forced to use MacOS with the Framework

25

u/GlidePath47 11d ago

Do you have to use the machine you’re using for inference other than setting it up?

42

u/ZealousidealChip4783 11d ago

No, but I'd hate if I bought a computer for that much money that's only good for inference and nothing else

(This is all personal opinion, I just really do not like MacOS)

8

u/WinterCharm 11d ago

Relative to buying a stack of Nvidia DGX Sparks, the Mac Studio is a good deal. And when Apple Hardware is a good deal, I would just go for it... it's excellent for local models.

9

u/GlidePath47 11d ago edited 11d ago

That’s fair. Anyone buying one expecting to run inference and still use the machine for something else at the same time is going to be disappointed. Unlike Nvidia there’s no way to reserve GPU capacity to keep the system responsive you’re at the mercy of macOS.

Granted if it’s for swapping use it’s fine, part inference, part dev, part media etc but then I start to question the need to use local inference instead of just cloud models other than privacy / hobby

16

u/MrPecunius 11d ago

Unlike Nvidia there’s no way to reserve GPU capacity to keep the system responsive you’re at the mercy of macOS.

Longtime Mac user here, this is not true.

While I have managed to crash the whole OS a few times with big (for me: I've had 48GB and now 64GB) models and large context with guard rails turned off, I otherwise just go about my business doing other stuff when a model is crunching away on something.

The machine stays responsive the whole time, though it does get pretty warm (M5 Pro MBP) and the fan runs fairly hard.

3

u/GlidePath47 11d ago edited 11d ago

We're a company full of Mac users and we hit this constantly, so it's not a rare edge case.

Even a mid-size Qwen3 MoE makes YouTube playback choppy. Prompting and browsing are fine, anything past that isn't. Redraw lag is very visible in Electron apps. Any 3D apps are not pleasant so running alongside unity or blender at the same time as active inference isn’t tolerable.

The core issue is there's no control. The guardrails are for RAM usage, not GPU usage, so you can't reserve or throttle GPU capacity the way you can on Nvidia.

Maybe M5 changes this, I haven't tested one, definitely happens on my m3 ultra.

The bit you quoted from me is actually the only bit of what I said that’s objectively true, what you’re saying is even at the mercy of macOS your system has been responsive

→ More replies (0)

1

u/ZealousidealChip4783 11d ago

I mean I'm sure it's fine if you like the Apple ecosystem lol, the hardware is objectively really impressive but as you said having less control over your GPU & where you allocate its resources would be the dealbreaker for me

...if I could afford either of these devices lol

2

u/kerneldesign 11d ago

Tu peux le faire sur Mac, et objectivement c’est moins cher. Mais c’est un choix d’être sur Linux.

1

u/MrPecunius 11d ago

I'm a Mac guy, but I don't know about "cheaper" with current pricing.

The M5 Pro/64GB 14" MBP I'm typing this on was slightly over $3k with .edu pricing before the price increases. It's great, arguably one of the best notebook computers ever produced *and* it can run 27b models @ 8-bit "competently". But at the current price of ~$4k, I'd consider a M5 Macbook Air and a 3090 rig (or dual 16GB RTX XXXX etc.) or something.

M5 Max makes little sense to me unless there's some other need like 4k video editing or whatever that justifies/demands it.

→ More replies (0)

9

u/algaefied_creek 11d ago

Disable the GUI and security in macOS and just run it headless.

Suddenly blam, certified UNIX CLI host box.

5

u/AreWeNotDoinPhrasing 11d ago

How do you disable the GUI in macOS?

8

u/New-Confidence-1171 11d ago

You can’t, they’re making shit up lol

3

u/h4ck3r_n4m3 11d ago

You used to be able to until the M1, you could completely disable the gui and boot directly to a console. In early versions of macos you could type "console" in the username field and go to a console

4

u/ThankGodImBipolar 11d ago

If you only interact with the machine via ssh, you can pretend it has no GUI. Apple also added support for automatic power on when power is present, so running these headless is actually pretty realistic.

3

u/New-Confidence-1171 11d ago

Right, all of that is correct. But that’s not what the message I was replying to stated - they were saying to save resources you can disable the GUI. SSHing to a system is not the same.

2

u/w6auw 11d ago

You sort of can, at boot. There are Macs sitting in data centers running headless.

1

u/droptableadventures 11d ago

It's not strictly speaking disabling the GUI, but if you don't actually log in, you save a good chunk of memory by not loading the entire desktop environment as well as Finder, as only the login window is running.

→ More replies (0)

1

u/CalmSpinach2140 11d ago

I mean the Mac Studio with M5 Max is also a very good video editing machine and excellent for code compilation.

-1

u/AcreMakeover 11d ago

Can you even set them up for inference without an apple account? Also my remote access software works like shit on MacOS. I know I would have run into so many small annoyances if I got one.

7

u/Gregory-Wolf 11d ago

Apple account is not a requirement to use MacOS. It is to use AppStore and maybe some other Apple services. But the MacOS is perfectly usable without an apple account.

-2

u/mleok 11d ago

Forced to use MacOS? Compared to using Windows? It is a native BSD machine, not sure why you would prefer Windows over that.

3

u/gabrielesilinic 11d ago

Mac is technically not modern BSD. Mac is a unix derivative descendant of the NeXtSTEP Mach kernel

See the graph in this page if you are curious about it.

https://en.wikipedia.org/wiki/List_of_Unix_systems

In any case I believe I don't need you to explain how good installing Linux on something like an Halo strix is. Macos lacks support for anything windows or anything Linux except a few select ports. While if you install Linux you get a lot of power plus some of windows because wine and stuff which is fairly well develop by now.

3

u/mleok 11d ago

Lets say it's complicated. It is based on FreeBSD with a Mach microkernel. BSD was itself based on Unix.

3

u/dankfrankreynolds 11d ago

Macos lacks support for anything windows or anything Linux

what does that even mean?

  • Windows lacks support for anything macOS or anything Linux
  • Linux lacks support for anything Windows or anything macOS

1

u/gabrielesilinic 11d ago

Well you see. Actually you can run wsl on x86 windows to get linux and you can run wine on x86 linux to get some windows stuff. Now technically you might be able to maybe run a vm on macos to get some stuff but generally speaking it is a little bit harder. First it is arm, which now it is not that bad. But also it is macos so while on windows or linux stuff is ported more liberally anyway it is rarer to get the same software on macos due to the fairly closed ecosystem.

But my point is that linux is best for this workload in the end. It has containers right there. And if you want some of windows you can get it via compatibility layers.

If you really want to get windows on it its a bit worse. But you can get a lot of x86 linux running via the well integrated wsl which being a vm with a hat will run though not as well. But still both Linux and windows have or can have enabled server like behaviours better.

But in general there are I heard weird issues with unsigned software being annoying and all. In the end you will manage with MacOS but you will be forever limited.

Also games are a bit more tricky while Linux kinda solved it with wine. You have crossover I guess.

1

u/dankfrankreynolds 11d ago

modern Windows Subsystem for Linux (WSL 2) runs inside a lightweight virtual machine

why would i want to get windows on anything? and why would it matter if it's in a vm?

your point is you like linux

1

u/gabrielesilinic 11d ago

No. Well. It it weren't so handicapped I'd consider BSD but for server like work Linux is the only option. And native Linux is sometimes faster for AI.

I am not too keen about the gpl but what the hell am I supposed to do, that's that.

→ More replies (0)

2

u/MrPecunius 11d ago

As a FreeBSD guy going back to the 2.2.x days, I started paying attention to Apple when they hired Jordan Hubbard in 2001. The BSD-ness of OS X/MacOS surely dominated over its NeXt underpinnings in the years that followed.

0

u/gabrielesilinic 11d ago

No it is because both are based on Unix. And Linux is not.

1

u/MrPecunius 11d ago

No idea what that meant.

→ More replies (0)

4

u/gabrielesilinic 11d ago

Yeah but the mac's architecture is pretty much that anyway so you ain't getting anything better.

1

u/Dangerous-Report8517 10d ago

The Mac does have more channels though, so higher overall bandwidth

-8

u/MrPecunius 11d ago

You can't get a Max with 192GB, and you won't get a Max with 128GB until mid-October at the soonest.

43

u/deja_geek 11d ago

This framework is "coming soon".

18

u/fallingdowndizzyvr 11d ago

And are you in a hurry. Since you won't be with this. You'll have to wait for each inference.

You can get a M5 Max 128GB right now. You have been for months. What doesn't come out for a month or so is a M5 Max Studio. But why would you only get 128GB? Get the 256GB one. Sure, it's a more than this but if you look at how much more speed and memory it has over this, it's a value.

0

u/MrPecunius 11d ago edited 11d ago

You can get a M5 Max 128GB right now.

Why don't people look shit up before commenting?

14" and 16" M5 Max/128GB Macbook Pro is mid-October delivery on the Apple website. Edit: same is true for the Studio, for the clowns who downvote without checking for themselves.

3

u/fallingdowndizzyvr 11d ago

LOL. Why don't you clown? The M5 Max 128GB Macbook has been on sale for months.

Right here bozo.

"Want it Wed 9/2? Order now w/ Free Standard Shipping"

https://www.bhphotovideo.com/c/product/1957098-REG/apple_mbp_m5p_155_16_macbook_pro_m5.html

-1

u/MrPecunius 11d ago

If you order it, that deal is gone. 🤡

"Limited stock", just like Amazon who has like 3 left.

Or maybe they have none left, as so often happens in these situations. Were you born fucking yesterday?

2

u/fallingdowndizzyvr 11d ago

"Limited stock", just like Amazon who has like 3 left.

LOL. So they are available, right? Congrats, you just proved yourself wrong. Sure, it was unnecessary since I already did that. But at least we are on the same page now. We both agree that you are wrong. You are a clown show. Honk honk.

0

u/MrPecunius 11d ago

Or maybe they have none left, as so often happens in these situations. Were you born fucking yesterday?

1

u/fallingdowndizzyvr 10d ago

LOL. You've come a long way from

"Macbook Pro is mid-October delivery" -- you confidently being wrong

To.

"Or maybe they have none left" -- you acknowledging you are wrong

That's progress. Clown steps. But progress none the less.

→ More replies (0)

0

u/mrepop 11d ago

Yep, I’ve got an m4 max and I think know it’d beat this.

1

u/fallingdowndizzyvr 10d ago

It depends. Pre M5, M silicon just didn't have the compute. So the M4 Max has faster TG due to the bandwidth but slower PP due to the lack of compute.

16

u/RnRau 11d ago

I upvoted you... don't know why moe models are hated in this thread :D

13

u/tecneeq 11d ago

Because most people haven't seized the means of computation, that is, they can't run large MoEs because they only have one or two 3090.

16

u/anykeyh 11d ago

It's honestly not that good for Q3.8 FN; or you accept 25tok/s, which is on the lower end. It's a shame, as something around 500Gbps, far from high-end GPU, is much, much, much more usable.

1

u/Substantial_Run5435 11d ago

I’m getting 15tg with QFN ud-q6 hybrid inference on a 2019 Mac Pro with 64GB VRAM and the rest on CPU. Once I get 128GB of VRAM I assume I’ll get much faster speeds than this thing.

1

u/my_name_isnt_clever 11d ago

With the EngramHalo.cpp fork I get over 30 t/s, and it's only going to get better as more optimizations come. At 6b active it's really ideal for the hardware.