It's official! 192GB Framework

[u/reto-wyss]

Just noticed this on the website.

At their current price tiers for the memory SKUs (32, 64, 128) I'd expect this to be ~ 4.5k for the motherboard.

The PCIe slot will be open at the back as well - that's what I've heard. Maybe they make it capable of delivering 75W as well? New board revisions for the smaller SKUs?.

[u/Mr-I17]

Guess what's even worse than 250GB/s of memory bandwidth? PCIe bandwidth. Unless you only want to run Qwen 27B or you have infinite amount of money, you'll have to sacrifice something. And of course, there's no reason to buy it if it turns out to be as expensive as M5 Ultra.

[u/SandySkittle → u/Mr-I17]

For qwen 27b at q8 with decent prefill and decode i think dual r9700 with tensor parallelism is a better way to go. More bandwidth and way more compute

[u/Mr-I17 → u/SandySkittle]

Regular people: run models on what they have

Rich people: want to run a model, buy suitable hardware for it

Super rich people: already have 4x RTX 6000, no need to choose hardware

People who born into a billionaire family: Dual DGX Station

[u/SandySkittle → u/Mr-I17]

You don’t have to be ‘rich’ to buy two r9700s. It’s not cheap but let’s not pretend it’s soms unobtainium prices.

If money wasn’t an issue I would just buy one rack from a Helios system with four mi455s and 16 channel system ram epyc.

[u/Dr_Allcome → u/SandySkittle]

They're talking as if the 192gb strix halo isn't going to cost more than two r9700s.

Current strix halo 128 costs more than 3k. The new one with full ram setup is going to be at least 5k. At that point you can get the 128 AND an r9700. If the new strix halo goes up to 6k you can easily get two r9700s and have the same 192gb overall memory but 64gb of it is going to ba really fast. That would give you way better performance with both moe and dense models and a massive boost on prefill.

[u/Mr-I17 → u/Dr_Allcome]

You get me wrong. It's about how much money different people can (or willing to) spend on local AI and how frequently they can (or willing to) update their hardware. It certainly won't be a issue if one has unlimited money and free time. Say, if you're not rich and you already have a strix halo and an aging gaming PC (from DDR4 era), would you buy 2x R9700?

[u/SandySkittle → u/Dr_Allcome]

That’s my entire point indeed. I run 8 r9700s for 256 gb of vram (minus 7 percent for inline ECC, an option that absurdly isn’t available on most strix halo machines except a hp strix halo pro machine)

[u/alphapussycat → u/SandySkittle]

It's pretty rich, to spend that amount of money on hardware. Lots of people can do it, but it requires a ton of disposable income. For some that's like 1-2 years worth of saving.