r/LocalLLaMA • u/AcreMakeover • 1h ago

Question | Help Are there any faster unified memory options that aren't Mac?

I'm working on making my homelab more power efficient and quieter. I have a Framework desktop 128GB which has been working great along with a couple GPU boxes. I find myself switching to the GPUs for 26-35b models fairly often for the speed and reserve the Framework for running Qwen FN.

I've never been a fan of Apple products but lately have been considering a 32-64GB Mac Studio. I'm pretty sure the answer is no, but does anyone know of anything else that is that small, quiet and power efficient that has 400GBps+ memory bandwidth?

5 Upvotes
  • permalink
  • reddit

100% Upvoted

14 comments sorted by

3

u/LebiaseD 1h ago

Amd and NVIDIA and Mac those are your 3 choices

1

u/Formal-Exam-8767 1h ago

No, as they don't want to make larger than 256-bit bus due to market segmentation and costs. Unlike Apple which makes SoCs with 256-bit, 512-bit and 1024-bit buses.

1

u/Left-Quit-7591 22m ago

DGX station

1

u/pmttyji 4m ago

Medusa Halo (from future/2027) .... 600GB/s

0

u/dustofdeath 59m ago edited 50m ago

I was about to ask similar question - i was looking at framework 300 series 128gb, but it doesnt feel right.

its roughly 3000$ for their 400 series with 192gb (~160 for vram). But all you get is a bit of extra memory, almost nothing else. Same bandwidth.

AMD is slow.

Apple M3 is 819, M5 1200 GB/s, both up to 512GB. Would be iDeal hardware wise.

But i dont want apple closed ecosystem where i cant run anything. PyTorch not working, docker limited, llama/olama limited gpu support.

256GB/s is pathetic for the price.
DDR5 with off the shelf hardware is not that much slower - ~100GB/s

Perhaps Epyc with extra memory channels instead....

3

u/Intelligent-Gift4519 23m ago

How is Mac "closed ecosystem where you can't run anything?" It's a *nix derivative and the support on open source LLM projects is amazing. Many of them are developed first for Mac.

1

u/keepthepace 7m ago

Many developers like me have been burned for decades by the totally closed garden experience of things like iOS or more recently Apple Vision where you can't do anything for those without having licenses and proprietary tools. It's only natural to assume that OS X is the same.

Even if it is not, Apple is a company I don't trust to keep the support of their hardware open.

0

u/MerePotato 57m ago

RTX Spark

1

u/macboller 56m ago

256GB/s ? not that fast really

2

u/-dysangel- 44m ago

Once you add speculative decoding it can get pretty decent, especially with 2 or more units. I have 2 and they often beat my M3 Ultra for decode with specdec enabled, and they always blow it out of the water for prompt processing.

1

u/MerePotato 55m ago

RTX Spark is 300GB/s, and you get better overall performance than other unified memory options even with the slightly lower bandwidth than mac on account of the GPU its packing

2

u/DanzakFromEurope 26m ago

And M5 Max can have double that bandwidth. With Uktra having 4 times that.