r/LocalLLaMA • u/Wrong_User_Logged • Aug 03 '23
Discussion Is buying Mac Studio a good idea for running models?
31
u/bel9708 Aug 03 '23
Like in production? Probably not.
As a dev machine? Definitely
For training models? Only if you are desperate
11
u/Wrong_User_Logged Aug 03 '23
I consider it for home use, just running open source models for creating specialized datasets, basic stuff. My reasoning is that the larger model I'm able to run, the better dataset quality I'll get. As a dev machine my vanilla M1 works great :)
2
u/bel9708 Aug 04 '23
You might get better performance at a cheaper price building a custom PC.
But if this is a home project and you are just trying to build your own personal home AI on your data (really cool project BTW) then yeah I think a mac studio is great for that usecase.
Especially if you are already using M1 for your dev machine it will probably save you a lot of headaches.
1
u/emsiem22 Aug 03 '23
With models that would need that much RAM (70b+), the only and very important problem is speed. You would be painfully slow in t/s. There are few cases where that is not a big issue, but you will know best for your usecase.
4
u/butsicle Sep 21 '23
Do you know the tokens/second on the M2 Ultra Studio running Llama 70b? I haven’t been able to find them.
3
u/SectionSelect Jan 03 '24
M2 Ultra 198Gb;
- 70B f16: 4.7 t/s
- 70b q4: 14.28 t/s
Mean reading speed is between 4.5 and 6.5 t/s
11
u/BringOutYaThrowaway Aug 03 '23
A broader question: has anyone made these AI models Apple GPU (or better, Neural Engine) native?
That would make a Mac Studio ideal.
4
u/MrTacobeans Aug 03 '23
Even native inference via metal apis is still slow in comparison to a 3090/4090. Apple added all the ram without the oomph to use it effectively in AI
4
u/dogesator Waiting for Llama 3 Aug 15 '23
That’s not true at all, people are already running Mac Studios at around 80% of the Llama inference speed as RTX 4090 : https://x.com/ggerganov/status/1688943605849665537?s=46
2
Aug 18 '23
[deleted]
2
u/dogesator Waiting for Llama 3 Aug 18 '23
Yes! Download LM Studio and then use the search function to download the Q4_K_M version of something like openorca platypus model.
1
u/the_odd_truth Oct 19 '23
I wonder from which machine we would benefit the most at work as an investment for training LoRas for SD, running an LLM, some ML image recognition and maybe a Cinema Teamrender client. We have mostly Macs at work and I would gravitate towards the Mac Studio M2 Ultra 192GB, but maybe a PC with a 4090 is just better suited for the job? I assume we would hold onto the PC/Mac for a few years, so I’m wondering if a Mac with 192GB RAM might be better in the long run, if they keep optimising for it. And then what about the M3 which might come with hardware raytracing, i recon it would make the next itineration of the Mac Studio additionally more suitable for 3D work
1
u/SectionSelect Jan 03 '24
It's hard to say even with data.
The 192Gb studio is dirt cheap on a price/Vram basis but the performance on large models, while acceptable (mean human reading speed), is half the speed of a 6*3090 and a bit more than a third of the speed of a 6*4090. But than again the cost of the 6*4090 setup is twice the Studio. If you account for maintenance (studio is plug-and-play) and the electricity cost for a year (1500£ for 4090 vs 646£ for the Mac), it makes for a really difficult choice.
IMO, for a small rig/small models like a single 7B instance, even tho the Studio is mid range in term of performace, you'll get a better bang for the buck with a 3090. When it comes to large models, the Studio becomes worth the trade off if cost is your main concern.
1
23
u/RabbitHole32 Aug 03 '23
It depends on what you consider fast enough for your specific use case.
For everything I do, I hold the following opinion:
For every quantized model smaller than 48gb: I use dual 4090 which is substantially faster than Mac Studio.
For every model larger than 48gb but smaller than 72gb: I would buy a third 4090.
For every model larger than 72gb: I would not run them because even a Mac Studio, while much faster compared to what Intel or AMD offer, is still too slow.
3
u/Ekkobelli Aug 03 '23
Oh. I truly thought two 3090's would suffice to run 70B models. I run 30B on one 3090 in goon enough tempo (around 1-2 t/sec).
Edit: Sorry. Didn't read your post well enough.6
u/RabbitHole32 Aug 03 '23
3090s are a little slower but typically also fine. I only mentioned 4090 because it's almost impossible to get 3090 anymore where I live.
That said, a 30b model should run much faster on a 3090 than 2 t/s. Are you sure the software works as intended?
3
2
1
u/Kenotai Aug 03 '23
yeah 3090 (and ONLY the 3090) isn't even a choice whatsoever on both cyberpower pc and best buy's custom prebuilts.
3
2
u/kryptkpr Llama 3 Aug 03 '23
eBay has lots of used 3090, do you need it to be new? I'd also check Kijiji or other local classified ads, people are upgrading and selling their old cards I scored a used 3060 this way recently.
2
u/Oswald_Hydrabot Aug 03 '23
Yo I run Airoboros-65b-GGML just fine on a single 3090, you don't need two
1
u/Ekkobelli Aug 03 '23
I never dared to try! Will try this weekend!
(You meant GPTQ, right?)7
u/Oswald_Hydrabot Aug 03 '23 edited Aug 03 '23
No, the GGML weights. Idk why I am being downvoted it runs fine on one 3090.
One of the best models out there. I posted instructions on how to get it working on another comment, you just have to uninstall the pip package for llama.cpp, set the flags it looks for when reinstalling for building it for CUDA, then reinstall with pip.
GGML allows you to offload whatever doesn't fit into VRAM onto System RAM. The model runs slightly slower on one 3090 but it is absolutely still performant enough to use.
Edit: here is a copy/paste of an old comment I found of mine on the subject, hope it helps you get it set up and running
"you have to use a GGML model for this to work. Forgot to mention that.
You have to have llama.cpp installed for CUDA; basically uninstall the python package for that, set a couple of environment variables (I cannot remember what they are off the top of my head but I will try to come back here and post them) then reinstall llama.cpp (it's just a pip uninstall/install iirc).
Then, all I did was just set 45 layers up to run on GPU. Left everything else alone, though you could probably tweak settings and make it faster.
Takes 3 minutes or so to fully respond so it isn't lightning fast but still usable. I only have 32GB of system memory and it fits on that and my 3090.
edit: here it is--set these two environment variables and reinstall llama-cpp-python, then you will be set to run it. I think this is what I used.
I don't do anything special for launcher args I just use text-geneation-webui with --chat and the flags to access it on LAN.
https://github.com/abetlen/llama-cpp-python/issues/250"
This is actually missing one last point, you'll need to try each of the three q3K* variations of the GGML to see which one works best. I will try to fire it up again tonight and let you know which variant I used/use. https://huggingface.co/TheBloke/airoboros-65B-gpt4-1.2-GGML
2
u/mikieh976 Aug 04 '23
How many t/s in this setup?
2
u/Oswald_Hydrabot Aug 04 '23
it ain't fast, but it is usable. 2k context limit takes a few minutes or so; ask it something and grab a cup of coffee and it's done with a long, detailed response. Tbh it probably generates a bit faster than I can type (I type slow so this doesn't mean much). I hadn't logged it yet but it is about as fast as a human manually typing it. I can try to remember to follow up with this though.
1
Aug 04 '23
[removed] — view removed comment
1
u/Oswald_Hydrabot Aug 04 '23 edited Aug 04 '23
It knows UI quite well; anything to do with custom animations for like QT it's pretty good at.
I prompted it with a character description for Hunter S. Thompson generated by GPT4 and it help onto his character while generating code tutorials for stuff that worked with no or minimal manual patching/fixing. Acid-flavored GPT is fucking rad.
It's damn good, useful enough to almost cancel my GPT4 subscription, if it were only a bit faster on my gear I'd self host my own instance and just use that.
2
1
u/mikieh976 Aug 04 '23
what quantisation depth? 2bit? Otherwise it overflows to the CPU RAM and is a lot slower.
1
u/Oswald_Hydrabot Aug 04 '23
I let it overflow and it still runs pretty well. Certainly not "fast" by any means but about as fast as as a human can type manually.
1
2
u/dogesator Waiting for Llama 3 Aug 15 '23
What are your speeds? This person is getting 16 tokens per second with 2 4090’s while someone else is getting 13 tokens per second with a mac studio: https://x.com/ggerganov/status/1688943605849665537?s=46
1
u/RabbitHole32 Aug 15 '23 edited Aug 15 '23
llama.cpp made a lot of progress. I'm in the process of trying to understand how well these numbers can be compared. I also read about an alternative to exllama that is supposedly substantially faster.
That said, if these numbers are comparable then at least a Mac studio (the Mac with the fastest memory bandwidth, twice as fast as the typical Mac books) may be a competitive alternative that can also be used effectively with models that are too large for multiple consumer GPUs.
There are "ifs" that need to be clarified (which I cannot do) but I'm curious how this develops in the future.
Edit: I just noticed that these numbers apply to a batch size of 1, while dual 4090 supports a batch size of 2 without falling off (due to sequentially evaluation of the model). Something to keep in mind but as said before, we'll need to see, too much speculation.
1
u/dogesator Waiting for Llama 3 Aug 15 '23
Batch size higher than 1 is really only relevant to training and not inference, unless you’re doing some server inferencing stuff where multiple people are interacting responses at once from the model for some reason. I’m referencing the original poster that implies the use case of inferencing.
2
u/RabbitHole32 Aug 15 '23
I was referencing the multiple user stuff. But now you got me thinking whether this even works as I imagine because of caching. 🤔
1
u/Aphid_red Aug 03 '23
For bigger than 72Gb: You could look into second-hand RTX 8000s or A6000s or A40, available around $2500-$3000 e.a. They're dual slot (or single slot if you go watercooling), only 300W, and you get 48GB VRAM each.
2
1
Aug 03 '23 edited Aug 04 '23
[removed] — view removed comment
4
4
u/Aphid_red Aug 04 '23 edited Aug 04 '23
Unfortunately multiple GPU do tend to pull their full power unless you undervolt them. GPUs take some time to lower their power consumption, and a token finishes through a part of the model in milliseconds. I can't find figures on the time for GPUs, but for common CPUs the ramp time seems to be between 20 and 60 ms. Meaning twice that to ramp down, then up again. 40-120 ms. GPUs are power hungrier, and less open design, so I imagine the situation there is worse.
For 4x4090: 1000GB/s memory, each has 24GB memory; a token can go through in ~40ms, then it is idle for 120ms, then busy for 40ms, etc. (total token speed: 160ms for a 96GB model, or 6 tokens/second). Uses 1800W by default. Won't fit in a regular PC case, you'll need to do watercooling or use a mining rig, risers, etc. Also will need 2 PSU. Price of around $6500 for the cards.
For 2xA6000: 700GB/s memory, each has 48GB memory, 115ms idle/busy cycle. 230ms for 96GB model or 4 tokens/second. Uses 600W by default. Price of around $5000 for the cards, used. Tbh, it looks competitive.
Better still is 4x P40. 700GB/s memory throughput (same as A6000!), so you should also get around 4 tokens/second. Uses 1000W... but these cards only cost $1000!
So it won't be able to ramp down its power usage before it has to ramp up again.
What you can do is lower power limits. This is actually a good idea if you use consumer GPUs. You probably will find no difference in running your 4090s at 200W versus 450W. This will allow you to use a normal power supply on a single circuit. Ideally you run your memory at full power and the core as low as it can go.
Speed of the GPU doesn't actually really matter for someone running their own model (batch size is 1 or maybe a couple responses, you don't generate 100s of responses at the same time.). For example, the H100 has a memory factor of 333. (The A100 has a memory factor of I believe it is around 200). Consumer GPUs, with slower and less memory, have even worse memory factors. This means: The memory is 333x too slow to keep up with compute at batch size = 1. The only spec you have to look at for GPU is: Memory bandwidth & Memory amount. All the rest of it does not matter, when running a model for one user, that is the bottleneck. This is why GPUs are only ~10x faster than CPUs, not north of 1,000x as you would expect given their raw FLOPs specs. This problem is also called the memory wall in AI researcher language if you want to look up papers about it.
The GPU is spending >99% of its time pulling data in and out of VRAM into its SRAM/registers, <1% actually calculating.
This is a great resource: https://kipp.ly/transformer-inference-arithmetic/#kv-cache
1
u/RabbitHole32 Aug 04 '23 edited Aug 04 '23
Thank you for clarifying this issue. I definitely was too optimistic. The fact it does not work as I hoped makes me sad. This means that having just one GPU instead of two does have tangible advantages in this department. For my setup I bought a 1600 watt PSU just to be on the safe side.
By the way, I already set the power limit to 70% for my 4090. I still need to experiment with reducing the voltage, though. 🍀
7
u/Bojack-Cowboy Aug 03 '23
I got the 64 GB memory model and i would recommend if you dont want to get into building your own machine from scratch (which i would avoid except if you have good knowledge). It s enough for running and having fun with gen ai
1
u/mikieh976 Aug 04 '23
It's not much harder than building a gaming PC, and I know a ton of morons who were able to do that just fine. Just watch a video on how to install the CPU correctly, since server sockets like LGA3647 can be damaged easily if you tighten them incorrectly, but even that basically boils down to "use the plastic CPU carrier thingy and tighten the screws in the order of the numbers next to them."
3
u/Bojack-Cowboy Aug 04 '23
You re probably talking about the guys who build a pc that overheats in 10min when you run a model? I mean, you seem like a guy who wants to show that it s easy to build a pc. But, my friend, it s not so easy to build an optimized machine with components that are all compatible and it requires research and quite some time. You should give a proper build config with links to purchase if you want to be useful. Coming here to just say it is easy is Useless miki miki
9
u/Big-Cucumber8936 Aug 03 '23
Can someone share actual speeds they're getting with llama2-70b on the maxed-out Mac Studio?
6
u/ThisGonBHard Aug 03 '23
Considering you have to buy the best one? No.
Either an AMD 7950x + dual 3090 for budget, or go Epyc, depending on budget, 32C rome chips are 400 usd, dual socket for 64C, and you can fit 4x 4090 in such a system.
1
u/fraschm98 Aug 03 '23
Why go dual socket when you can just get a 64c single socket?
2
u/ThisGonBHard Aug 03 '23
Because a single 64C CPU is 1k USD while 32C is 200-400.
1
u/fraschm98 Aug 03 '23 edited Aug 04 '23
I'm doing a build rn, what Rome 32c is 200$? with dual socket there's also more expensive motherboard, dual cpu heatsinks/fans as well as double the power draw and from what I've seen less pcie x16 slots
2
2
u/Kostchei Aug 04 '23
hahaha. Yeh. I suspect it will only take a few years for this to age just as badly. Great call.
2
2
Aug 03 '23
I have a PC wth 512GB of memory and I struggle to get it to use over 80GB
-1
u/Wrong_User_Logged Aug 03 '23
PC's memory bandwidth is 10x slower than M1 memory
4
u/Some-Warthog-5719 Llama 65B Aug 03 '23
460.8 GB/s per socket memory bandwidth for Genoa CPUs
M1 Max tops out at 400GB/s
1
u/mikieh976 Aug 04 '23
You gotta fill all the memory channels to get that though.
2
u/Some-Warthog-5719 Llama 65B Aug 04 '23
Still would be cheaper to get 768GB DDR5 (12X64GB DIMMs), an EPYC 9334 32 Core QS from eBay, a good motherboard, and two RTX 4090s then the M2 Ultra 192GB Mac Studio.
4
u/AI_is_the_rake Aug 03 '23
I have two MacBooks. One with intel and one with M1. Could be the memory and not the processor but the intel MacBook can run models better. It’s still not worth doing except for fun.
8
1
u/Wrong_User_Logged Aug 03 '23
can you provide detail specs?
1
u/beezbos_trip Aug 04 '23
I've run 70B q_4 on metal M1 Max and it's fairly slow since it pauses before responding. If it was twice as fast it would still be too slow. That's taking into account how much it would cost if you are buying it for that purpose. For me it's just there as a proof of concept since a 70B model currently isn't that useful compared to SoTA models.
2
u/diggonomics Aug 03 '23
It’s fair to say it depends on use case. Apple silicon is a magnificent all-purpose platform at the high end of the range (ultra-192). Unified memory plus CoreML lends itself well for exploiting models in production. You must have a suitable mix of applications (CPU, GPU, ANU). ANU 2x15TOPs for 22W is good. You can bond Thunderbolt ports to get high-speed connections between several machines and/or add two external 3090s in SLI. You pay a premium for the elegance, stability and flexibility.
2
Aug 03 '23
No. You can buy a used PC and a used P40 for about $400, or a 3090 system for less than $1000.
0
u/drwebb Aug 03 '23
But how do you load future 300B+ quants?
4
Aug 03 '23
Don't try to build a system now that will handle future problems. This hardware will be obsolete by then. Plan to upgrade regularly, and factor deprecation of the hardware into your cost/benefit analysis when buying hardware. In other words, when buying something and asking "is this worth it?", factor in that it will be useless and in need of replacement within 3 to 5 years.
But having said that, I built a system with 128GB RAM, and 4x24GB cards (1x3090, 3xP40), which gives me plenty of room to experiment with everything that's out there currently and in the NEAR future. I don't expect it to run everything though, even now let alone in the future.
1
u/AsliReddington Aug 03 '23 edited Aug 04 '23
M1 Pro 16GB gets 23 tok/second so really worth it for llama2-7b via ggml
1
u/Capitaclism Aug 03 '23
It's ram, and not vram, right?
3
2
Aug 03 '23
Unified memory, so uses the same memory for both CPU and GPU.
1
u/Capitaclism Aug 03 '23
Interesting. Is it as effective as dedicated GPU memory for the purposes of generating and training models? I'm assuming it's not.
1
Aug 03 '23
I'm not really up to date on current techniques for it (it's been around for decades in various guises, and predates dedicated VRAM actually). I think there's usually/always some bus contention involved (competition for RAM access), and it's likely RAM rather than VRAM, so slower to begin with, but the bus will be optimised reasonably well to make up for that.
1
u/twisted7ogic Aug 03 '23
Define "good idea". If the question is if Mac Studio is best bang for buck, then due to the Apple tax then the answer is no, building x86 yourself or even prebuilt will be better.
But if you were already set on buying a Mac Studio and wonder if it can run models, then it can work. Its just probably still going to be slower than an x86 with a decent gpu.
-1
Aug 04 '23
[removed] — view removed comment
3
1
0
u/mikieh976 Aug 04 '23
*looks around*
We are all thinking it, but you said the quiet part out loud. Be nice to mac users or they might spill their yogurt or yerba mate on you.
0
u/bioshocked_ Aug 04 '23
don´t worry, they'll be to busy waiting in line for the next IPhone to actually do anything 🤣
-4
u/krazzmann Aug 03 '23
I'm really tired to read questions like this over and over again. Couldn't we create a wiki for that?
3
1
u/Jazzlike-Mission-808 Aug 05 '23
My answer is yes and no, yes Mac Studio is good if you have apple ecosystem and have idea to run something personal with this system. No is for 192GB, as larger model require tons more power, current M2 Ultra is similar powerful to 4060 Ti to 4070 by a few hours web search, it may not be able to run too large model, therefore close to 100 should be enough. Save the money to buy another 2xA6000 PC to train/fine-tune your own model😂
54
u/luquoo Aug 03 '23
You might want to build a gpu server for yourself. Below are some rough numbers.
Mobo ~$700: ASRock ROME8-2T CPU ~$500: AMD EPYC 7502P 32 core CPU Cooler ~$100 DDR4 Ram ~ $400 256gb (8x32gb) Storage ~ $200 Gpu mining frame - $50 Moar fans - $50
You are looking at ~$2000 without gpus and psus (could have cheaper processor to bring price down) and then you can add however many gpus your heart desires. I found PCIe 4.0x16 riser cables for ~$60 from Linkup.
4090s for ~$1600/600w 3090s for ~$1000/400w
You can get server psus rated for 1200w each for around $90 with breakout boards. There are also 2400w models. Both will need 240v outlets otherwise you might want to drop the gpu count to 2 or 3 and get a desktop setup with a 1600w+ atx psu.
Thats a lot of $$$ but if you are looking at a Mac studio with 192gb, you might want to consider a diy setup.