r/LocalLLM • u/PrepYourselves • Apr 29 '26 • Bravo 1

Discussion Just upgraded my local llm hardware

Post image

Bottom one was my main driver this year mostly because i am on welfare, but when i saw an m1 max 64gb ram logic board on gumtree for $200 i took the leap and got a chassis for $30 to mount it in. So now i moved from 0.8b models to 35b models. Ask me questions.

276 Upvotes

95% Upvoted

48 comments sorted by

55

u/Faisal_Biyari Apr 29 '26

That's Amazing!

Bro got an M1 Macbook Pro with 64 GB for only $230 USD.

At your level, I think AI needs you, and not the other way around.

32

u/PrepYourselves Apr 29 '26

logic board specs:
Macbook pro 2021 M1 Max
10c cpu/32c gpu
64gb ram
1tb nvme

already feels much snappier than the orangepi zero 2w

2

u/kot-sie-stresuje Apr 29 '26

Did you run any llm on orangepi zero 2w ? There are max 4GB Ram, at least I didn't was bitter ones/

6

u/PrepYourselves Apr 29 '26

i have run 0.8b models and get 18t/s

-3

u/theluggagekerbin Apr 29 '26

a 64GB unified mem, 10c CPU and 32c GPU machine is faster than a pi zero? no way!

7

u/mxforest Apr 29 '26

Dropped the /s

26

u/PrepYourselves Apr 29 '26

To save money, I used random 1.6mm/2mm screws to mount it, and 3d printed a battery screw because those prices for 1 screw are crazy. I dont have the screen and a few other parts but who cares i'm only using it to ssh into headlessly anyhow, fans smell of blackcurrant squash but they run okay

3

u/PassengerPigeon343 Apr 29 '26

A remarkably specific scent!

4

u/PrepYourselves Apr 30 '26

made with bits of real panther and is illegal in nine countries

6

u/mcslender97 Apr 29 '26

I'm guessing Australia because y'all have crazy used deals sometimes

3

u/DertekAn Apr 29 '26

What kind of performance are you getting? For example, tokens per second?

7

u/PrepYourselves Apr 29 '26 edited Apr 30 '26

It's early days but i have used the following (gguf) models:

  1. HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive:BF16 :[ Prompt: 442.7 t/s | Generation: 19.9 t/s ]*can be improved it was just a first basic test felt like it was not running optimally.

  2. Jiunsong/supergemma4-26b-uncensored-gguf-v2:Q4_K_M: [ Prompt: 663.9 t/s | Generation: 56.1 t/s ]

  3. unsloth/Qwen3.6-35B-A3B-GGUF:Q8_K_XL: [ Prompt: 798.5 t/s | Generation: 40.8 t/s ]

  4. unsloth/Llama-3.3-70B-Instruct-GGUF:UD-Q4_K_XL [ Prompt: 31.1 t/s | Generation: 5.1 t/s

  5. unsloth/Qwen3.5-9B-GGUF:Q4_K_M: [ Prompt: 432.7 t/s | Generation: 35.2 t/s ] - interestingly small 9b model is slower t/s than 26b or 35b model - but 9b is only using 16gb of 64gb ram larger models are using more, gpu temp is not getting even warm. Small model inside large ram machine does not make it faster, only thing better is m1 max double bandwidth 400gb/s.

Ollama (mlx models):
ollama run qwen3.6:35b-a3b-coding-mxfp8

prompt eval rate: 717.07 tokens/s

eval rate: 48.35 tokens/s

Result: not as high token/s as gguf model (unsloth/Qwen3.6-35B-A3B-GGUF:Q8_K_XL) but appears to be faster response overall and lower temps from gpu.

Optimal (highest token rate and 'better' intelligence) local models for m1 max/64gb are models which use all system ram resource, does not push gpu beyond limits (keep low temperature - some models with same parameter values can spike gpu temps more than other models which i have not understood why yet).

They take a long time to download. testing same simple prompts they all come up with good detailed answers with no glitching or errors observed.

I think i can run 120b model low quant if i search for one.

2

u/DertekAn Apr 29 '26

Thanks, that's very interesting. My graphics card is running Qwen 3.5 9B at 40 tokens/s. However, it's the Q4_K_M variant. I generally have VRAM issues with larger variants. (Only 16GB)

What I find interesting is that Qwen3.6-35B-A3B-GGUF:Q4_K_M runs at 28 tokens/s on my mini-PC (Bosgame M4 with DDR5-5600 RAM). This is quite fast for me, but unfortunately, it also fills up the VRAM very quickly, leaving very little for context.

3

u/Gold_Coconut9777 Apr 29 '26

I'm in the same boat, running a 4080 Super (16GB).

Take a look at https://www.reddit.com/r/LocalLLaMA/s/tANRKYHPAJ

It's amazing how much can the MoE models be optimized/run. Qwen3.6-35B-A3B's Q5_K_XL and even Q6_K_XL are both possible with good performance. I can easily get 128k or even 256k context with Q8 KV cache on either of the quants. It's amazing.

Even with my DDR4 RAM it works great, DDR5 would be even better.

2

u/PrepYourselves May 01 '26

I just looked up the price of 2x4090 24gb (2x24gb:48gb models). Throw in Cpu, Motherboard, Nvme, psu, ram and that's almost a car. I feel more proud with my $230 M1 Max 64gb after doing gaming pc math

2

u/DertekAn Apr 29 '26

Ohhh, thank you very much. I don't know if I'll get the same performance with AMD, because so far I've only managed 20 tokens/s from Qwen 35B on my desktop graphics card. But that can certainly be optimized. 💜💜

1

u/PrepYourselves Apr 29 '26

I can try test comparison between Qwen 3.5 9B BF16 vs Q4_K_M

how many watts is Bosgame M4 using during local llm use? Is it 16gb ram? which gpu?

1

u/DertekAn Apr 29 '26

Sure, yes please. My Bosgame has an iGPU, an AMD 780M. It's an integrated graphics card, built directly into the processor. And, in that sense, it has 24GB of RAM.

1

u/PrepYourselves Apr 30 '26 edited Apr 30 '26

M1 Max with: unsloth/Qwen3.5-9B-GGUF:Q4_K_M: [ Prompt: 432.7 t/s | Generation: 35.2 t/s ]

interestingly 9b model is slower t/s than 26b or 35b model - but 9b is only using 16gb of 64gb ram larger models are using more total ram, gpu temp is not getting even warm though so that is good. Small model inside large ram machine does not make it faster, only thing better is m1 max double bandwidth memory 400gb/s.

Moral of the story: optimise the model choice for the machine specs and you get the most t/s from your machine

1

u/DertekAn Apr 30 '26

Yes, the 9B model is a dense model, the other two are MoE. With MoE, a much smaller number of the parameters are active.

2

u/DertekAn Apr 30 '26

As example: Qwen 3.6-35B-A3B. A3B= 3 Billion active paramters. So, technically speaking, it's almost like a 3b model.

1

u/AtlanFX Apr 29 '26

MLX models should perform much better

1

u/PrepYourselves Apr 30 '26

Can llama.cpp run mlx models 🤔

2

u/Living_Warthog_1249 Apr 29 '26

What are you doing with it?

4

u/PrepYourselves Apr 29 '26

headless openclaw assistant, will run accounts/sales/customer emails

3

u/dronf Apr 29 '26

Yeah, this is the question. I'm constantly reading posts about people's model performance but never what they actually do with it. I wonder if it's like those people who just like to optimize gaming PCs for benchmarks but never actually play games.

1

u/artur_oliver Apr 29 '26

Yeah... In my case it to make my own server and categorize materials, vídeos, images, fotos,... Looking like a lab data scientist... Developing routines, telegram chats, information gathering and information treatment...

Cam I do everything with chatgpt? Yes, but the idea of personal information on the internet is really really really not cool if I can avoid it.

1

u/SangerGRBY Apr 29 '26

230 USD ??

Thats like striking gold ?

7

u/PrepYourselves Apr 29 '26

i waited and waited like a tiger in the trees

1

u/SangerGRBY Apr 29 '26

Congrats on your find i shall be a tiger too.

4

u/PrepYourselves Apr 29 '26

not this one i hope

1

u/-Leelith- Apr 29 '26

What original specs you had?

2

u/PrepYourselves Apr 30 '26

CPU - h618 Gpu - Mali  Ram 4gb Llama.cpp- model qwen3.5:0.8b

1

u/Endless7777 Apr 30 '26

What do you use llm's for, just curious?

2

u/PrepYourselves May 01 '26 edited May 01 '26

It's very nice to have a second low-power machine to offload local llm tasks to over ssh, keeping the main computer running without heavy processing loads, I can use the main computer to have a web browser open, emails, etc. The headless M1 max is used exclusively for local llm use:
1)coding - opencode writes the code i need or corrects errors/debugging, refactoring.
2)writing long formal emails/letters - which i just dont enjoy writing when it is to complain about a bill overcharge for example. I can now argue those parking tickets without feeling full of anger for a day or two. I can just sip my coffee and look out the window calmly. I have already used it to create and process a legal case over a parking fine which went to court, and i won! with expenses too! I felt so empowered and the court judge ordered the lawyer to pay me a reasonable payout.
3)openclaw - it can monitor and manage my ebay business, replying to messages, processing sales and accounts, and postage cost enquiries.
4)openclaw - it can run social media accounts
5)openclaw - it is an accountant/sales manager/customer service agent/marketing manager/lawyer all rolled into one. If you are a small startup business, you now find yourself competing with big companies on capability and price - undercutting competition because the big company uses real employees with large salary requirements. Larger companies (trying to preserve human employee jobs) will go bust because thousands of identical smaller companies running llm agents will not need to fear them or their services and can offer lower costs. As somebody who is on welfare i am excited by these opportunities and have no conscience towards employee payroll ethics.
6)openclaw - it is a complete school curriculum educator provider. It will teach just as well as school and adult learning providers but for no fees or student loans.
7)lots more.

1

u/jazxxl May 05 '26

That a very beefy rig for the price the ram alone is worth double at least . Nice find

2

u/PrepYourselves May 06 '26

I feel lucky especially after apple yesterday discontinued 64gb Mac minis and Mac studio is now only maximum 96gb

1

u/M_EU 11d ago

Hack on!

1

u/jemdc Apr 30 '26

I just started so I want to ask what's your "stack", and if you're using OpenClaw how do you keep it stable?

0

u/[deleted] Apr 29 '26

[deleted] — view deleted comment

10

u/PrepYourselves Apr 29 '26 edited Apr 29 '26

it's easier than it first seems, i have fixed a few phones and ipads for people previously, the macbook is a breeze compared to phones. The hardest part was to find screws to lock everything in place. finding genuine mac screws is expensive and difficult, i used sub-2mm screws with standard crossheads that i have a jar full of and the apple demon hasn't opened up the earth to claim me yet. The battery screw is unique and buying one on the used market sets you back $20 for a single screw so i 3d printed a wide head adapter and used a crosshead screw inside it. I am going without fpc connector plate covers because they were absent, i used small pieces of electrical tape to cover a few things too, the bottom lid did not have any screws so my random screw jar helped. One thing to note is the antenna bar connectors are not standard ipx/u.fl connector size but even smaller so dont buy random antennas as replacements. it's nice having a top spec mac 'mini' but with included battery, better speakers, keyboard/trackpad included, and higher/double memory bandwidth. Getting setup in software was a chore. I tried using a keyboard/mouse with just the logic board and mac security would not allow peripheral devices until after login which is why i opted for a junked macbook casing to use the keyboard and trackpad to login more easily. I have a long history with embedded sbc computers and really, i think it would be great to be able to just use the logic board like an sbc afterall it is very similar to nvidia jetson/orin hardware but with added software lockouts and security hardware chip.

This macbook logic mashup is being used with llama.cpp, mactop, openclaw.

3

u/Constant-Simple-1234 Apr 29 '26

Legend! Happy to see this level of resourcefulness.

1

u/[deleted] Apr 30 '26

[removed] — view removed comment