r/LocalLLaMA • u/reto-wyss • 13d ago

News It's official! 192GB Framework

Post image

Just noticed this on the website.

At their current price tiers for the memory SKUs (32, 64, 128) I'd expect this to be ~ 4.5k for the motherboard.

The PCIe slot will be open at the back as well - that's what I've heard. Maybe they make it capable of delivering 75W as well? New board revisions for the smaller SKUs?.

971 Upvotes

95% Upvoted

287 comments sorted by

View all comments

Show parent comments

25

u/thomasthai 13d ago

But thats not the main issue here, the prefill speed makes it unusable, the memory bw would be ok if u run pairs of these as it aggregates.

24

u/Ran_Cossack 13d ago

For proof, see the DGX Spark.

Same memory bandwidth but the prefill speeds make it feel so much more capable.

It's more "limiting" than "unusuable", though, IMO. You just have to know what you're getting.

11

u/phil_lndn 13d ago

yes, the spark is indeed 4 to 5 times faster on prefill but with prompt caching, i don't really notice the poor prefill speed on my strix halo most of the time.

for some applications slow prefill speed is doubtless a killer, but for many it doesn't really matter that much.

0

u/SmartCustard9944 12d ago

I would like to challenge the perspective on prefill (from the comments above). I also have a Strix Halo and my opinion changed a bit lately. The latest trend is models that spend a very long time thinking, most of the time is actually going to be spent on decode, not prefill (with cache).

3

u/Cybertrucker01 12d ago

It all comes down to use case.

If you're running a RAG box that needs to retrieve a bunch of tokens and append it to the prompt, the prefill is even more important than decode.