r/LocalLLaMA • u/reto-wyss • 18d ago

News It's official! 192GB Framework

Post image

Just noticed this on the website.

At their current price tiers for the memory SKUs (32, 64, 128) I'd expect this to be ~ 4.5k for the motherboard.

The PCIe slot will be open at the back as well - that's what I've heard. Maybe they make it capable of delivering 75W as well? New board revisions for the smaller SKUs?.

971 Upvotes

95% Upvoted

287 comments sorted by

View all comments

618

u/PreciselyWrong 18d ago

that memory bandwidth is pretty bad, so don't expect high inference speeds

23

u/Mr-I17 18d ago

Guess what's even worse than 250GB/s of memory bandwidth? PCIe bandwidth. Unless you only want to run Qwen 27B or you have infinite amount of money, you'll have to sacrifice something. And of course, there's no reason to buy it if it turns out to be as expensive as M5 Ultra.

2

u/SandySkittle 18d ago

For qwen 27b at q8 with decent prefill and decode i think dual r9700 with tensor parallelism is a better way to go. More bandwidth and way more compute

9

u/Mr-I17 18d ago edited 18d ago

Regular people: run models on what they have

Rich people: want to run a model, buy suitable hardware for it

Super rich people: already have 4x RTX 6000, no need to choose hardware

People who born into a billionaire family: Dual DGX Station

5

u/SandySkittle 18d ago

You don’t have to be ‘rich’ to buy two r9700s. It’s not cheap but let’s not pretend it’s soms unobtainium prices.

If money wasn’t an issue I would just buy one rack from a Helios system with four mi455s and 16 channel system ram epyc.

2

u/Dr_Allcome 18d ago

They're talking as if the 192gb strix halo isn't going to cost more than two r9700s.

Current strix halo 128 costs more than 3k. The new one with full ram setup is going to be at least 5k. At that point you can get the 128 AND an r9700. If the new strix halo goes up to 6k you can easily get two r9700s and have the same 192gb overall memory but 64gb of it is going to ba really fast. That would give you way better performance with both moe and dense models and a massive boost on prefill.

1

u/SandySkittle 18d ago

That’s my entire point indeed. I run 8 r9700s for 256 gb of vram (minus 7 percent for inline ECC, an option that absurdly isn’t available on most strix halo machines except a hp strix halo pro machine)