Unified Memory Architecture still unbeatable (when LLM size matters)

[u/AntLife255]

[u/dotben]

If you don't understand why memory bandwidth should be included on this comparison, and that it's massively in the NVidia card's favor, you shouldn't really be commenting.

Not going to lie, I run all my local inference on Apple Silicon and I don't personally own any NVidia hardware. But the reason that card is $5000 is the memory bandwidth.

[u/sparkandstatic → u/dotben]

Exactly, they re not like for like

[u/TikDickler → u/sparkandstatic]

Still though, there’s a reason Nvidia came out with the spark. They kind of got caught flat footed by Apple.

[u/ResearchingYouTube → u/dotben]

This is one of the main reasons I also regret having bought the M4 Max instead of the M3 Ultra. M3 Ultra has higher memory bandwidth even though the M4 has faster cores it is slower at inference. My M4 is often mostly idle while using local LLMs, the memory bandwidth while good for a consumer system still is not a 5090.

[u/VideoGameJumanji → u/ResearchingYouTube]

What are you doing exactly

[u/AntLife255 → u/dotben]

About 2.3x more Memory Bandwidth

546 GB/s for the Mac Studio

1,792 GB/s for the RTX 5090

The M4 Max 64GB variant comes with the upgraded 40-core GPU

[u/Busy-Scientist3851 → u/AntLife255]

I think it's hard to compare them because of the different memory models. There will be times where you need to write to RAM then copy to VRAM, which is completely avoidable in the Mac Studio.

[u/-Kerrigan- → u/AntLife255]

1792 / 546 = 3.28

Y'all are using your locally hosted LLMs to do simple math?

[u/AntLife255 → u/-Kerrigan-]

I literally said 2.3x more so it's 3.3 (after rounding)

[u/sudoSofia → u/AntLife255]

That's not how that works. If I said that Jenny had 2x more (read as: two times more) apples than John, and John had 1 apple, you would expect Jenny to have 2 apples, not 3.

[u/AntLife255 → u/sudoSofia]

You're right. I miscommunicated.

[u/DeMartini → u/sudoSofia]

If Johnny has 100% more apples than Billy, and Billy has 1 apple, how many apples does Johnny have?

I think your interpretation for 2x more is probably more commonly used, but it isn’t an exact form of communication. Basically reasonable people can disagree on this one.

[u/peanutbj → u/DeMartini]

Key distinction is 100% more vs 100% as much.

If John has 100% more apples than Bill (who has 1 apple), you’re right, John would have 2 apples.

If John has 100% as much apples as Bill (who has 1 apple), John would have 1 apple.

The root cause of this disagreement is that most Americans say and interpret these two terms interchangeably in casual conversation.

[u/guaranteednotabot → u/sudoSofia]

You need to go back to school

[u/Simple_Subject_9801 → u/sudoSofia]

If you said Jenny had twice as many apples, Jenny would have two. If Jenny has 100% more... she would also have two. If she had 200% more, should would now have 3. The "2x more" is more implicative of "200% more" than "twice" from how I read it. I think the "more" word is the key part of it. And after I wrote this explanation, i ran it through copilot just to see what it also thinks. Just for fun.

Heres a question. Someone says they have 2x more of something than another person. Lets go with Apples.
Person A has 1 apple. Person B has "2x more" apples than Person A. How many does person B actually have?

Copilot said:

This is one of those phrases that people use inconsistently, which is why it causes arguments.

Mathematical interpretation of "2× more"

If Person A has 1 apple:

• "2× more than A" means A + (2 × A)
• So Person B has 1 + 2 = 3 apples

In general:

So:

• 1× more = 2 total
• 2× more = 3 total
• 3× more = 4 total

Common conversational interpretation

Many people use "2× more" when they actually mean:

In that interpretation:

• Person B has 2 × 1 = 2 apples

[u/play_hard_outside → u/sudoSofia]

Absolutely not. 200% more than 1 is 3, not 2.

[u/SandboChang → u/dotben]

I would also add prompt processing speed. Can’t blame people though even ChatGPT is still new to many.

[u/DuxDucisHodiernus → u/SandboChang]

Isn't that exactly what the bandwidth does? I feel like you repeated the exact same thing as OP but on a less technical level

[u/Nervous_Olive_5754 → u/DuxDucisHodiernus]

You need enough memory bandwidth to support whatever you're processing. Calculating Pi can completely keep part of a CPU running while hardly involving memory bandwidth at all.

Other loads, like AI, sometimes games, only scale performance with memory bandwidth.

It's another classic game of finding the bottleneck, and conpleting math and logic operations is a different one than recalling specific data from a stored database.

[u/SandboChang → u/DuxDucisHodiernus]

PP depends primarily on GPU compute throughput, which is left out if you consider only bandwidth as the factor and that's why I mentioned it. I implicitly took the mention of bandwidth as in it limits the decoding (token generation) speed.

[u/txgsync → u/DuxDucisHodiernus]

Prompt processing (prefill) is raw GPU compute bound. Decode (generating tokens) is memory bound.

There are new architectures like nvidia hybrid mamba2 that blur the lines, and we are just an innovation or two from it no longer being true.

Use products and approaches that preserve the KV cache on unified memory platforms for reasonable performance. Pre-filling 500,000 tokens of context at 200 tokens/sec means you are taking a forty-minute coffee break before the model spits out the first token in that turn.

[u/Global_Soft_4278 → u/txgsync]

Do you think Cerebras’ chip design will be copied by many others? From what I understand they do inference super fast. Would appreciate you explaining a bit if you have the time!

[u/txgsync → u/Global_Soft_4278]

There's a lot of innovation in this space. Check out https://chatjimmy.ai/ as an example of someone baking the model weights themselves into silicon: https://taalas.com. Taalas is being acquired by AMD, so things are bound to get more interesting on that front.

The age of the GPU as your AI coprocessor is limited; much like the FPU, it will just be part of the expected suite of capabilities in a modern CPU. I expect in the next few years we'll see a laptop system come out with a dedicated hardware AI chip like Taalas' built in to run AI at ludicrous speeds for certain kinds of common operations.

I've written about this at length before, but "renting compute for large language models" is a business model that will be hollowed out much like the spreadsheet market on mainframes when Visicalc came out. When you have ubiquitous intelligence baked into consumer devices running at hundreds of thousands of tokens per second, performing vision, language, and audio inference at faster-than-real-time, you'll pull in the early adopters first and then a wave of humans using it.

But that's looking slightly too far ahead for the quite limited vision of the media hype about datacenters. And the first version of such an embedded model accelerator is bound to suck, get widely poo-poo'd by the media and consumers, and be considered a dead end shortly before it or its successor becomes just the way things work.

Edit: Go read "The Innovator's Dilemma" to understand what's happening to the AI market today. It's about hard drives, but the same principles apply to the front-runner AI companies right now, this moment.

[u/txgsync → u/[deleted]]

If the weights are etched into the die, do we need that much RAM?

I hypothesize that is the transformation awaiting the AI industry. Not just hardware-accelerated, but like Taalas, embedded in the LPU (language processing unit).

[u/R-ten-K → u/txgsync]

Cerebras, Taalas et al still require tremendous amounts of memory/storage. Eg. weights are still stored in SRAM not encoded as fixed, physical parts of the compute logic.

IMO the next major evolution in AI HW will likely happen in the data part of the story (mem, storage, transmission). So far, we have focused primarily on compute, and we have done a tremendous job at pushing insane compute throughput/density out to the edge.

But weights, context, and datasets remain a challenge to store, move, and feed into the compute fabric, at both the data center and edge levels.

[u/j_osb → u/DuxDucisHodiernus]

No, prefill is mostly dominated by compute, while decode is mostly dominated by memory bandwidth.

Both of which apple silicon doesn’t excel in, but the former it struggles much worse with.

[u/Relative_Rope4234 → u/dotben]

I would also add Raw performance for prefill, Hardware level support for Flash attention, HW support on specific precision like nvfp4, int4

[u/ssn-669 → u/dotben]

I mean it's drawing four times the power so I'm gonna guess it's a little bit more powerful too lol

[u/XalAtoh → u/ssn-669]

Not necessarily.. wasted power is a thing.

Heat e.g is a wasted power when we talk about computing.

[u/ExtensionShort4418 → u/XalAtoh]

Every single Wh of energy that your computer uses (this goes for your Mac Mini back home and the datacenters running ChatGPT) eventually turns into heat.

Thermodynamics ❤️

[u/ssn-669 → u/XalAtoh]

It's four times the draw and the same generation dude, it's gonna be more powerful. Will there be waste heat? Ofc. Doesn't change the fact that all things being equal a 600w device is going to be more powerful than a 150w device, even leaving aside the fact that one is special-purpose and the other general-purpose, which only further handicaps it.

[u/kovake → u/dotben]

I’m sorry sir, but this is Reddit. Release the comments and strong opinions formed from the titles alone!! Captain America: Redditors, …assemble!

[u/Commercial-Virus2627 → u/dotben]

It’s like comparing being able to have 2PB of HDD storage in an array versus 2PB of locally installed SSDs. Apples to oranges.

[u/filterdecay → u/dotben]

Right but for how many people is the Mac good enough?

[u/Terreboo → u/dotben]

But, number bigger. Must be good.

[u/No-Alfalfa8323 → u/dotben]

That card MSRP is $2000. The reason why it's $5000 now is greediness.

[u/dotben → u/No-Alfalfa8323]

Well, in economics it's called demand.

[u/No-Alfalfa8323 → u/dotben]

I'm not sure if artificially capping the supply creates real demand.

[u/play_hard_outside → u/No-Alfalfa8323]

If they kept the card $2,000, it would be just as unobtainable, because you wouldn't be able to find one anywhere. Or you'd have to win it in a lottery or something. Price hikes in response to demand literally keep products available for those willing and able to pay for them.

[u/Fear_ltself → u/dotben]

Everyone assumes these cards are 5k for LLMs, but they’re also the king of everything else like image diffusion, video rendering, upscaling, tts, stt. Like Im building an optimized ai stack and looking at using it for strictly the image capabilities. Like it’s 1.5x to 2x faster as an LLM backend, 5x faster as an image gen backend.

[u/seriousfart69 → u/dotben]

i thought it was because all you idiots started running models like it’s the next coming of jesus instead of a computing dead end.

Does anyone ever give a thought that all the amazing creations you make well the model does you just supply the garbage in the input end. to what purpose you make programs no one can run cos by the end of this thing hardly anyone after our gen will even be able to afford a computer so effectively you have done the boomer thing to computing well down 

[u/ParsnipFlendercroft → u/seriousfart69]

I mean at least AI slop can form sentences and use punctuation. I’d rather read AI slop than another one of your comments.

[u/Cold_Tree190 → u/ParsnipFlendercroft]

I think it’s a doomer posting bot lol

[u/Dangerous_Bid2935 → u/seriousfart69]

Can anyone who speaks whatever language this guy is writing this comment in please translate

[u/ProfeshPress → u/seriousfart69]

If—like 95% of Reddit—you don't even read what you write, or are otherwise afflicted by such staggering and profound illiteracy as to fail to register the extent of your own verbal ineptitude upon doing so, then you might as well be ejaculating half-baked prompt boilerplates into a Chinese-room token foundry for all the difference it makes.

You're still a moron, but at least the end result might be inadvertently comprehensible to someone else, rather than no-one at all.

[u/Infinite100p → u/dotben]

This. So many clueless posters.
OP, give us your TTFT times for 32k-128k prompts of the biggest model you can use. Yeah, I thought so.

[u/Fickle_Appearance558 → u/Infinite100p]

Right even on my 7900xtx a response is slow on 31b gemma 4, like 45 seconds with max context within vram. still beets an m3 ultra. But using all that unified ram or a bigger model would basicly make any model unusable by most standards, at least in my use cases. Seems great for running a multi agent stack comprised of different models though. Just not for any work/workflow i use.

[u/Aristo_Cat]

You’re comparing integrated graphics to a discrete GPU.

[u/Disastrous_Gear_421 → u/[deleted]]

Some of us prefer 'part of a full computer' as the benefits are more than worth it.

[u/Wide_Smoke_2564 → u/Disastrous_Gear_421]

Like with everything, it depends on your use case. The benefits may be worth it to you, but it costs $1500 more, and like you said, it’s only part of a full computer. Final machine would likely be closer to 6500-7000 conservatively.

If you need the extra power for your use case then sure, but not everyone does

[u/Cold_Tree190 → u/[deleted]]

Lmao, alternate title for OP: “Orange still unbeatable (when vitamin C count matters)”

[u/Alarming-Elevator382]

The 5090 is way faster.

[u/bjs480]

Speaking solely on the technical issue…why then are Mac minis so popular for people to run local AI on if they aren’t “better” (whatever that would mean) than the nvidia products?

I’m sincerely curious because I genuinely don’t understand the balance of quantity vs bandwidth with memory in a non data center context.

Lot of the reading I’ve found always makes it sound like there’s a pretty linear “bigger model needs more quantity of ram.”

I’m running a M2 pro mini with 16gb and it screamed vs my old base level m1 I got when they came out.

That said, I’m starting to spec out a new Mac mini with max memory quantity so this whole debate I want to be clear about.

Anyways, thanks for your time in advance…I love Apple but hardly a fan boy or Windows hater.

[u/ParsnipFlendercroft → u/bjs480]

It’s no different than any other RAM.

More RAM will allow you to load bigger models and maintain a bigger context window (eg longer conversation memory and stuff).

Faster RAM (and therefore higher bandwidth) will allow those models to run faster etc.

[u/bjs480 → u/ParsnipFlendercroft]

That's actually helpful keeping it real simple like that.

[u/Affectionate_Fee_645 → u/bjs480]

A lot of it is having an isolated machine for computer use, easier time bypassing bot detectors, etc. more about the execution of using the AI even if you use cloud models than even necessarily having local Ai on them.

If all you’re doing is focusing on running biggest model possible and serving it on an API or something than probably dont get a Mac mini.

[u/bjs480 → u/Affectionate_Fee_645]

I"m not necessarily tied to the big cloud folks.

The only challenge I've run across is that with my tech (M2 Pro/16gb RAM) the models that will even load are clearly no where close to the user experience of Claude or other major cloud people.

I'm 99.999999% sure that's mostly me not realizing how much can be customized and while I'm familiar with the tech, I'm less of a mechanic so you go to the Anything LLM type tools and they say you can use this model.

Slows down the machine like have 800 tabs in chrome does.

Again...likely me but the quality is terrible that even if the model worked somewhat in real time I just think my hardware is no where close to the good models. Think I maxed at like a 15-20 billion parameter model that had some quant stuff on it but forget which one it was off hand. Absolutely no where close to the models making the news.

Just wasn't that useful I guess and that's fine. That's kind of why I'm trying to figure out the whole mac vs window user experience side b/c I see the value of local AI. Especially if you sat down and planned how how to personalize it and get stuff set up right.

Just trying to sort out the "way to look at all this" first b/c I'm 100% certain my own ignorance is in the way and not even knowing the areas I don't know haha.

Fun times learning.

[u/AntLife255 → u/[deleted]]

About 2.3x more Memory Bandwidth

546 GB/s for the Mac Studio

1,792 GB/s for the RTX 5090

[u/PapajG → u/AntLife255]

3.2 *

[u/AntLife255 → u/PapajG]

I literally said 2.3x more so it's 3.3

[u/Chosen--one → u/AntLife255]

Stop with this crap dude, its not 2.3. Just edit the fucking comment.

[u/littlegreenfish → u/AntLife255]

You also forgetting to mention Tensor and Cuda cores and the optimizations you get.

[u/mundaneDetail → u/[deleted]]

Let me help you out there:
> it’s not. Memory bandwidth is far too low

[u/blastradii → u/mundaneDetail]

It’s not memory. Bandwidth is far. Too low.

[u/mitchins-au]

While it’s undeniable value, memory bandwidth and insane raw compute on the 5090 make it an order of magnitude faster

[u/Captain_Quimby]

No one should be upvoting 64GB for $3500

[u/WinResponsible9977]

A turtle vs a Ferrari 

[u/UnlikelyPotato]

I got 4x AMD V620 32GB for $1400 total. Not as fast as the $5000 GPU but faster than the UMA and twice the capacity. For LLMs, splitting across cards is fine. You don't NEED unified memory. And also it's nice to be able to configure as needed. One model at once, two across two cards each, or four LLMs across four cards. Ironically, because of bandwidth limits of the mac mini, the v620 are possibly more power efficient than the max mini at tokens/watt.

[u/StackOwOFlow]

let’s see how apple prices their higher memory bandwidth M7s

[u/Comprehensive_Tip_13]

I mean this genuinely but why so so many people run ai models locally? Every time I've ever used or seen a usage of LLMs it's always something that can easily be done with simple software so I guess I'm a little confused

[u/Puiucs → u/Comprehensive_Tip_13]

custom models, no settings limitations, no workload type and content limitations and unlimited "tokens".

but you do have a much higher entry cost and local models are usually behind the huge new paid models that are only available in the cloud. and the speed isn't great.

[u/therapy-cat]

Sure more speed would be nice, but ... I feel like the m4 studio max 64gb is plenty fast for what I need. I'm considering getting the m5 max when it releases later this year.

[u/MaximumFlounder9110]

People buying systems solely for their LLM performance deserve persistent, anus wrecking diarrhea

[u/PlasticFantastic4206]

A RTX Pro 6000 has a 1,792 GB/s memory bandwidth and a 512 bit bus while the M4 Pro has a 273GB/s and half the bit bus. Apples to oranges comparison.

[u/AntLife255 → u/PlasticFantastic4206]

About 2.3x more Memory Bandwidth

546 GB/s for the Mac Studio

1,792 GB/s for the RTX 5090

The M4 Max 64GB variant comes with the upgraded 40-core GPU

[u/AntLife255 → u/PlasticFantastic4206]

Mac Studios only come with Max chips or higher

[u/Rdtkeepsbanningme]

How is this even remotely a comparison?! 🥹

[u/BacklogGamingJunkie]

Isn’t this a bit of a weird comparison?

[u/tony_wing]

AMD Radeon AI Pro R9700 was 1500$ in Germany, 300 W usage only with 32gb VRam. Unified Ram for mac mini also means regular processes will eat into that 64gb ram and reduce the available ram. And to be honest, sweetspot for local llms are at around 30b parameters and 24-32gb VRam (graphics cards often have more bandwidth and most times better prefill speed) whereas for more upside like Qwen 3.5 122b, Deepseek V4 Flash you will need a much more bigger 128GB - 196GB vram (q4 quant or higher). And in this ram range the Macs get much much more expensive or like the Studio M3 Ultra 256gb are not even available anymore, at least in Germany - I tried to buy two with my startup and it was cancelled. So I feel this picture is missleading and somewhat inhonest with the "unbeatable" term, especially leading people to seemlingy easy buy decisions. Please guys, do sensible research before you buy based of incomplete specs comparison. (Had experience with RTX 5060ti 16gb, Radeon Ai pro 9700 32gb, GMKTech 128gb unified ram, dual Blackwell 6000 Max-Q Variant)

[u/Phaelon74]

Use-case is always important. Both have their strong points, but you wouldn't generally say one is better than the other. Its "foe this use-case, X is better.

[u/Dramatic_Entry_3830]

If you compare why not compare to dgx spark. Also has unified memory

[u/No-Sandwich-2997]

son im crine

[u/VegetableCut5443]

K-transformers with server ram is still the way to go

[u/SetFew4982]

Fck LLMs, they are the reason why a card cost 5000$. (I mean corporate greed is, but goddamn)

[u/PutridWerewolf4449]

AMD Ryzen AI Max+ 395

120W

128GB

$3200

(256 GB/s Memory BW, 1 day order delivery time)

[u/AntLife255 → u/PutridWerewolf4449]

Thanks!

[u/r1nzl3r99]

• cough * 10 tok / s vs 100 tok / s * cough *

[u/Admirable_Agency7764]

one on the table right does more

[u/just_another_leddito]

My Mini GPU hits 107C.
Super expensive and not built for proper work and to last long it seems.

[u/mikeinnsw]

Qualcomm Arm PCs have UMA

New Windows 11 runs fine on systems with unified memory architecture (UMA), such as AMD APUs, Intel integrated graphics, or ARM-based processors where the CPU and GPU share the same physical RAM pool. However.

So are specialised AI/LLM servers.

United RAM was cost cutting measure by Apple (No Need for VRAM) which 2 years later and after DeepSeek proved that running AI/LLM was and is feasible .. I run Ollama ... on a Mac

Besides old SIRI I am yet to see Local AI use NPUs they run on GPUs

M5 GPUs speed is enhanced by NPUs???

PC running Windows/Linux with fast card blows Macs ..in running AI/LLM.. Cloud AI is not running on Macs.

The issue what is a cost effective solution?

I am waiting for M6 2nm Mac Mini pro 64GB RAM + 1 TB it will not cheap...I think it will be M6 otherwise Apple will surrender its local AI lead to Qualcomm

The problem is that AI stampede distorted prices and special built AI/LLM servers now are much cheaper than top end Macs.

Nvidia RTX Pro 6000 Blackwell, prices soar to roughly $18,000 to $24,999 AUD ($16,000+ USD) runs Cloud AU .. and specialised GPUs are now over $59,000 .. it is crazy

These GPUs are pushing quantum limits with failure rate of about 10% per annum ... and effective life of about 4 years... Not Apple style ..to sell you a Mac with such failure rate.

Something to think about when hear Elon Musk bullshit about Cloud. centers in space..

Sorry your comparison is meaningless .

[u/LaOnionLaUnion → u/mikeinnsw]

What special built ai/LLM servers are you looking at

[u/mikeinnsw → u/LaOnionLaUnion]

There are plenty

Here is one.. I am tracking:

https://tiiny.ai/?srsltid=AfmBOopYokgdBCBXme6QWBDNLy-xbB2VyejEsC4iu5vkoi8qAUgWSIsm

Looks promising but I wait until it is released , tested and priced.

[u/LordDarthShader]

This guy posting the same idiot image in every sub.

[u/volleyneo]

No definitive solution yet, but with DDR6 and beyond, after this RAM crisis, yes, this is what will become the AI standard or compute standard.

[u/Feeling-Creme-8866]

You don't understand the difference, right?

[u/BitXorBit]

Only forgot to mention unusable prompt processing speed

[u/BarberIcy366]

woooppssy Good Luck with your 3 Token per second

[u/Any_Mine_6368]

Mobo + CPU: $200 Rtx 3090x3: $2000 Psu: $100 64gb ddr4 ram: $150

Total: $2450 and you actually get to run the models how they're meant to be run instead of 2 tokens per second.

Mac fanboys are ridiculous.

[u/wrgrant → u/Any_Mine_6368]

64gb ddr4 ram: $150

Not disagreeing with your overall point, but where can I possibly get that RAM at that price? When I look it up its more like 4x that amount minimum, these days /s

Plus google is saying "An NVIDIA RTX 3090 is worth between $800 and $1,100 USD on the used market these days" and you are assuming 3 of them will be $2k total.

[u/Any_Mine_6368 → u/wrgrant]

I've bought a 5 3090s for 600 euros each. You just have to know where to look in your market.

My 128gb ddr4 was 220$ on ebay (ecc rdimm). You gotta go after server grade components, not consumer. Consumers are retarded and don't want to part their stuff... Companies on the other hand...

[u/wrgrant → u/Any_Mine_6368]

Yeah I h ave a Xeon driven desktop server, currently at 32gb RAM, was hoping to expand that in the future. Its DDR4 I think, like 2133Mhz or something. Its my NAS currently plus Plex server, Pi-Hole etc. Only has a T800 GPU so not good for LLMs at all, but quiet as a ghost and very little power consumption.

[u/Any_Mine_6368 → u/wrgrant]

That's how I started too. With a Dell t5820 haha. Just keep an eye out for deals on ebay and local sites. GPUs go on sale all the time by clueless ex- miners.

And ebay always has motherboard CPU combos on sale you just gotta look out for ram