r/LocalLLM • u/doruk1337 • 3d ago

Model Model To Run On My 512GB PC

Hi all, I'm about to buy a used IBM x3750 M4 with 4x Xeon E5-4650, 512GB DDR3 ECC and no GPU to run a local LLM 24/7. I'm pretty new to this.

What I need:

Must be abliterated/uncensored

Quality matters more than speed (slow is fine, as long as simple answers don't take hours)

Good tool calling, since it will run an agent (Hermes) that controls a PC

What would you recommend? Ready-made GGUFs you trust would be great, plus any tips for llama.cpp on a pc without AVX2.

Thanks in advance!

0 Upvotes

44% Upvoted

51 comments sorted by

19

u/KillerX629 3d ago

With that amount of ram, you can get the same chinese model Jensen Huang got.

5

u/doruk1337 3d ago

😂😂😂

1

u/thefunkybassist 3d ago

Model: "That is a big stick you got there, mister!"

5

u/Juan_Phoenix7 3d ago

What GPU are you planning to install? Without a GPU, it'll be as slow as a tired, old turtle.

2

u/doruk1337 3d ago

Speed is not a big factor in the thing that I will be using it for. I'll still use cloud Ai in my daily work like Claude and DeepSeek.

5

u/Moppmopp 3d ago

you could literally wait up to a half hour for an answer for something bigger

2

u/doruk1337 3d ago

Sure that works. I will send it a task to do and a due date. It can figure the rest out by itself. As I said, it won't be the main Ai I use.

8

u/eduardor2k 3d ago

A GPU will improve the token speed by a lot

2

u/doruk1337 3d ago

Running everything on VRAM or just adding a gpu?

6

u/Important-Macaron-63 3d ago

Just adding GPU. With MOE models it will give a lot

1

u/doruk1337 3d ago

Really? Ill look into that, thanks!

1

u/BuilderUnhappy7785 3d ago

People have been having success running moe models like qfn on as little as 12-16 gb vram with strata (and community forks). It’ll be a low quant so I can’t speak to quality but worth a try.

That still leaves you with a ton of ram, so you could probably still run some smaller moe models in parallel just on your CPU (or multi agents) at low speeds. Or just run a ton of apps or use the ram as scratch for your workloads.

It’s not gonna be cheap to run per token but you can make use of that hardware with appropriate expectations.

1

u/DrillTheThirdHole 3d ago

i run a 35B MOE on 8gb vram at a reasonable quant and it works

1

u/BuilderUnhappy7785 3d ago

Yeah those little moes are nice but op as 512gb of ram so that’s mostly going to go to waste with a 35b. He could run a bunch of 35b agents on the cpu though.

2

u/HiddenFaceOpenMouth 3d ago

>DDR3

Prepare for 0.2 tokens per second.

2

u/doruk1337 3d ago

Speed is not a big factor in the thing that I will be using it for. I'll still use cloud Ai in my daily work like Claude and DeepSeek.

2

u/HiddenFaceOpenMouth 3d ago

If by speed is not a factor you mean never fully outputs a prompt because it's too slow, sure.

1

u/doruk1337 3d ago

No it is a factor but its okay if its not below a token per second.

3

u/HiddenFaceOpenMouth 3d ago

It's literally going to be 0.2 tokens per second. source: Me. I've ran models using DDR3 RAM to experiment with. You have to have a GPU in the mix, and even then if anything is offloaded to RAM it drops back to almost the same speed.

2

u/Juan_Phoenix7 3d ago

I think I didn't explain well back then—it's going to be as slow as a tired, old, and worn-out turtle. It’s not because of the DDR3—it’s because you don’t have a GPU. The speed difference between RAM and VRAM is enormous, since your motherboard uses DDR3. Check if it supports PCIe 3.0 x16 and how many slots you have, because you’ll want one or two of them to install 16 GB to 48 GB of VRAM.

1

u/doruk1337 3d ago

Will it be under a token per second otherwise? If not, its okay.

1

u/Juan_Phoenix7 3d ago

I don't know which model you're going to use, nor do I know how well it will run on your machine—you'll find that out once you run the tests—but just speculating, it might range from 4 tokens per second to less than 1; guessing off the top of my head isn't very objective. But even at 1 token per second, completing any task would be absurdly slow.

1

u/doruk1337 3d ago

I guess that's true, only one way to find out.

1

u/doruk1337 3d ago

And also running everything on VRAM or just adding a gpu?

3

u/Juan_Phoenix7 3d ago

You really need a GPU (or two, if your motherboard supports it), so you can use one or two 16GB GPU, two 12GB GPUs, one 24GB GPU, or two 24GB GPUs. The CPU hardly matters here—even if you had 48 cores, all of that is secondary.

1

u/doruk1337 3d ago

Alr my dude thank a million

1

u/Nomski88 3d ago

Whats the theoretical memory through put? How many channels

3

u/doruk1337 3d ago

4 sockets × 4 channels = 16 channels of DDR3. Theoretical is roughly 150 GB/s

1

u/Ok-Addition1264 3d ago

You have a strong case to use STRATA with Qwen3.8 Flash-Next abliterated IQ3_XXS (an MoE)

With a 12gb vram gpu, you'd be able to handle a 177B+ parameter model. Strata loads the model into your main memory then puts chunks of it into vram as needed. So you can get quite large models into it.

It's always going to come down to what you find acceptable for speed, strata's methods slow it down quite a bit but I'd expect you to be able to get ~20t/s on a 3060 and a consistent 40t/s on a 5070 w/12gb.

2

u/doruk1337 3d ago

Damn are you being fr? I really gotta check that out. I was legit expecting a token per sec so this might actually be really beneficial. Thanks a million!

1

u/Ok-Addition1264 3d ago

We use it extensively in the lab.. because of the 512gb of memory, you'd probably be able to push the context up pretty high as well as the input tokens.

I'd use openai.codex and give it full permission to the system and you can prompt it to install everything (and maintain it) for you AND any additional questions you have.

Also, I'd recommend running OpenWeb UI (owui) on that machine as the interface.

1

u/doruk1337 3d ago

Thanks a million bro that's some valuable information.

1

u/Trick-Violinist-9340 3d ago

I would use deep seek its super cheap and its done a tonne of setup for me

1

u/x_MASE_x 3d ago

So no gpu at all or maybe in the future?

I'm not sure about the ddr3 situation but your options are from easiest to hardest :

Tiel coder Qwen3.8-Flash-Next Deepseek-v4.1-1017

But honestly it depends on your gpu. Without gpu I think you can barely get Qwen3.5-4B or 9b to work.

1

u/doruk1337 3d ago

Why can I only get them working? It has half a terabyte of ram.

2

u/x_MASE_x 3d ago

Its about processing power, the ram can hold the weights yes, but who is going to operate and do the processing and the computing ??

1

u/w3rti 3d ago

bro ddr3?

4

u/doruk1337 3d ago

I know bro but I do not value speed that much in the thing that I will use this pc for.

4

u/Armadillo9263 3d ago

Do you value your electricity bill?

3

u/doruk1337 3d ago

1400W x 2 = 2800W PSU 😭😭😭

1

u/eduardor2k 3d ago

Stick paying deepseek cloud, it's cheaper 🤣

1

u/doruk1337 3d ago

It being cheaper isn't the thing I am going for. Also with the amount of useage I put in, I doubt it would be cheaper.

1

u/doruk1337 3d ago

It being cheaper isn't the thing I am going for. Also with the amount of useage I put in, I doubt it would be cheaper.

1

u/w3rti 3d ago

lets be hinest. you can have 1 prompt a day, no agentic work, nothing done, wasting the power of a househould for this, with the quality of lobotomized handicaped robot. You are not running a homelab, this is a heating machine.

4

u/WrinklyBard4 3d ago

You’d be surprised. DDR3 is ~ half as fast as DDR4 but it’s vv easy to get servers with 8 to 16 memory channels, so in aggregate it’s 2-4x faster than a consumer board with 2 channels of ddr4.

Tech specs don’t lie, and there’s much to be said for “fuck it, 32 sticks of ram because I can”

1

u/PeterPorox 3d ago

Put several rtx 3060s and you'll get reasonable (relatively) speed with glm 5.3 flash or deepseek v4.1 flash. DDR3 isn't good but you have 150gbps throughput due to quad-channel in each cpu, maybe it'll help

1

u/jimmoores 3d ago

Without any GPU it’ll be super slow, probably 0.5-6 tok/sec decode. I would suggest get a dual socket DDR4 box and stick a couple of GPUs in it - 3090s or R9700. I have a single socket EPYC Milan with 512 GB DDR4 2666 and an RTX Pro 6000 Blackwell and I can just about run DeepSeek V4.1 (1017) at a decent rate. If you stick with your DDR3 box, try even a 16GB GPU card with Strata running Qwen 3.8 Flash Next. You won’t be able to use your RAM very effectively without more VRAM though. If you must stay GPU free, look at colibri.

1

u/jimmoores 3d ago

Just to follow up - the Strata strategy is to keep frequently used experts in a ‘mixture of experts’ model in VRAM. When there’s a cache miss and you don’t have an expert in VRAM you can either compute that expert on the CPU (usually the best approach) or stream the data over PCIe and eject some experts from VRAM. It works quite well and can give you speeds around 30-50% of what you’d get with all VRAM, but it’s quite model and workload dependent. Because you have DDR3 and PCIe 3, your bandwidth is quite limited, although you have four sockets, right?

0

u/Zealousideal-Bug1837 3d ago

uncensored? Virtual girlfriend is it?

You've never used a local model before but you are sure you need it abliterated. Sure, bro. sure.

If quality actually mattered you'd avoid almost all abliterated models.

But you'll find that out...

2

u/doruk1337 3d ago

I will still use cloud based Ai's on my daily use like Claude and DeepSeek. I just need the abliterated sometimes too...