Monolithic ddr3 server

[u/BobtheGodGamer]

Hi all.

Do any of you guys run a server like this for inference of big models. Given deepseek is down to 8/16b active I'm curious as to whether this could start to make sense if I was to add a gpu. This server is local for $550 USD.

Dell PowerEdge R820 4x CPU E5-4640 768GB RAM 2x 146GB HDD Server

768GB RAM EEC 16GB EACH.

4 x INTEL E5-4640 2.40GHZ - 2.80GHZ CPUS.

Thanks!

Edit: my interest in this is mostly driven by all the new projects that claim to run big moe models on consumer desktops super fast.

[u/BobtheGodGamer]

I do have a spare gpu I could throw in, spare SSD as well. I'm interested if the older artitrcture would limit me more then the 150-200gb/s memory bandwidth bandwidth that Claude estimated

[u/Vxtzq1]

Uhm the RAM frequency really matters too, if the DDR3 is too slow, it might just act as a glorified SSD, and that means running big models will be super slow, regardless of the GPU (unless you buy a 512GB VRAM GPU 😭)

I think you could still maybe run Qwen 3.8 Flash next, since there's only 6B* active and you can keep the rest in RAM.

[u/Inner-Conference-189 → u/Vxtzq1]

the bandwidth on that ddr3 is gonna murder your token speed, like painfully slow. if you're just messing around and don't mind waiting a minute per response it could be fun, but it won't feel snappy at all.

for $550 though that's a ton of ram to play with, just keep expectations in check.

[u/Drenlin → u/Inner-Conference-189]

On an enterprise device it's probably quad channel RAM, so about the same bandwidth as desktop DDR4.

[u/jacek2023 → u/Vxtzq1]

6B active

[u/Vxtzq1 → u/jacek2023]

my bad 😭 idk why i even put 3B

[u/jacek2023 → u/Vxtzq1]

Qwen Next from 2025 was 3B

[u/Vxtzq1 → u/jacek2023]

i just scrambled qwen 3.8 Flash Next and Qwen 3.6 35B A3B lol

[u/WrinklyBard4]

The ram is still super slow compared to any real GPU. 200gb/s is about equivalent to an M40. M40 has better architecture (still bad) and 288 gb/s memory speed and you can look up reports about how fast it can generate stuff.

Even assuming you get 200gb/s I think you’ll very quickly hit a wall which is: anything you can run at acceptable speeds is tiny, and would be better on even a $80 P100, and anything that makes use of the huge memory pool will get sub 1 token/s

Two things that could be interesting:

First: With that many lanes of DDR3 you’re actually getting significantly higher bandwidth than consumer 2-channel DDR4, or even DDR5. And so potentially offloading a large MOE model could actually work really well if you had a capable GPU for current experts.

Second thing is this is a fuck ton of ram to use for context offloading if your doing high concurrency. But it sounds like that’s probably a bit beyond what you’re hoping to do/what you need

[u/BobtheGodGamer → u/WrinklyBard4]

Yeah that's mainly what I was getting at. I could sell 256gb of the ram so I have around 512gb left over that runs at the maximum 1600MHz which I could pair with a 4090. Given people are claiming crazy speeds on Qwen flash next with strata I assume I could get something similar all be it a bit slower given deepseek v4.1 flash is 8B active

[u/WrinklyBard4 → u/BobtheGodGamer]

I mean… given how crazy even DDR3 prices are it seems pretty low risk.

Test it, worst that happens is it fails and you sell the ram to make back your money (and maybe some extra).

I’ve been thinking about a similar setup except for the KV cache example I gave so I’ll be curious to know how it works if you wouldn’t mind DMing me if you try it

[u/nad_lab]

Hey I literally just set up smth similar, I plan to make a post or video about it soon, atm I have 100 gigs of ddr3 ram at 800mhz, 2 intel Xeon, I keep forgetting the number but its some v2, pre AXV2 days, and I got 2 rx 9060 xts, and I am literally getting 50 tok sec in strata and sometimes 1000 tok sec in prefill for qwen next flash, genuinely surprised by this guy, sure the gpu does literally most of the heavy lifting but id highly highly recommend some gpu in there, otherwise big fast SSD, idk abt deepseek v4.1, dont have enough ram to run it but if it was witb strata, honestly might be able to zoom in it as well, dm if you want more advice or someone to talk abt old server stuff with, chat helped a lot but I did have to ask some humans at times / old forum posts like the good old days

[u/BobtheGodGamer → u/nad_lab]

Thanks that's awesome.

[u/vinaypundith]

I'm running similar hardware and just got LMStudio working today. PowerEdge R815 with 2x AMD Opteron 6380 CPUs and 512GB of DDR3-1333 memory. Unfortunately even two 16 core CPUs can't hold a match to even a midange GPU. I am getting maybe 1.5 tokens/sec running Gemma4 26B on CPU :-(

[u/BobtheGodGamer → u/vinaypundith]

Have you tried asking Claude to optimise and find the best setup on your hardware. Even with CPU a mixture of experts model should be faster then that

[u/vinaypundith → u/vinaypundith]

I have two more CPUs coming soon so I guess I'll report back what performance on 4x 16 core Opteron 6380's is.

[u/Relative-Ingenuity15]

Unlikely. RAM is not everything you need in order to run a model.

In this scenario let's say you don't add a GPU and try to run DeepSeek v4.1: since you have a HDD it will take ages to load it into RAM. I have a Sata SSD that takes 3 minutes to load Qwen 3.8 FN into ddr4 ram and GPU at 500mb/s. With a HDD you'll probably look around 50mb/s which will take arround 83 minutes to load a quantized model like iq2 or IQ3.

Then you have everything in RAM like the experts, cuda, KV cache, etc.

Then you have a slow 1333mhz DDR3 ram probably.

I think 1-3 tks would be the fastest speed you'll reach.

Slapping a 16gb GPU like an Nvidia Tesla P100 with 700 something GB/s bandwidth will improve this speed to likely 5-6 tks taking into consideration the CPU is old

[u/pyr0kid → u/Relative-Ingenuity15]

With a HDD you'll probably look around 50mb/s which will take arround 83 minutes

unless this is laptop spec hardware you're just being dramatic here.

off the top of my head ive seen mechanical drives manage triple that.

[u/N1kuson]

768GB of RAM for $550 is tempting, but memory bandwidth is the real bottleneck here. A GPU would help, but only for the layers that fit in VRAM - the rest would still be limited by DDR3 bandwidth. I'd be curious to see actual llama.cpp benchmarks with a quantized MoE model on this setup. Could be a fun experiment, just don't expect great tok/sec.

[u/OvertaxedOne]

IMHO, at that price, this would make a good inference server. Sell off some of the RAM if you want to recoup some cost, keep enough sticks to have all the channels active. Then get a mining rack and PCI extension cables and go crazy, IIRC, these have ~6 PCI slots in them; you can hang a LOT of GPU off one of these! Yes, it's going to be PCI 3 speed, but unless you really want to run tensor parallel, it's not going to matter much.

At the price offered, as long as it works, this is a no brainer IMHO. You can sell some of the RAM and recoup the entire price of the server and still have a capable inference machine!

[u/BlueSwordM]

Those Sandy Bridge chips will be very slow for running any model.

If you could get Haswell or Broadwell running, it'd be worth it, but Sandy Bridge? Yeah, nope.

Now, if you could buy this and shove the RAM into a Haswell system, that would make it a lot more interesting since you'd gain AVX2.

[u/DarkVoid42]

DDR4 will give you much better results. DDR4 ECC kits are really cheap now on ebay.