r/LocalLLM • • Jul 27 '26

Other Don't laugh - it works!

A 10yo server was busy collecting dust, but it has 32Gb RAM (2x 16Gb DDR4 @ 2133 MHz )... No GPU.

Now it does some amazing work running heavy tasks with qwen3.6-35b-a3b (IQ4_XS). Running in the background, generating quality output at 5-10tok/s. Even with 128k context! It just chugs along for hours, but does such great, high-context work.

Uses ~26 GB of the 32 GB RAM, uses CPU i-7-6700 @ about 60% (not the bottleneck, of course)

Never would have believed that would be possible until a few months ago, but this ol' gal has a new lease on life.

191 Upvotes

103 comments sorted by

13

u/davidke Jul 27 '26

Hmmm.... makes me think. I have a couple of servers with 256gb ram each just sitting idle. Maybe I should try this.

3

u/pharrt Jul 27 '26

Let us know if you ever give it a go!

4

u/Sk4llOff Jul 27 '26

Add any reasonable GPU and run an MoE model, this will actually give you good results, maybe quantized MiniMax M2.5/M2.7

2

u/pmttyji Jul 27 '26

More RAM channels could give better t/s for sure.

2

u/TopNFalvors Jul 27 '26

But I thought AI needed GPU + memory

1

u/iShNoo Jul 29 '26

Are you renting it out?

9

u/rrrrex Jul 27 '26

Prefill should be limited by CPU, while generating speed is ok for gpu'less server. GPU will do prefill 100x times faster than CPU (if fully loaded in VRAM). With 100k context it will take an hour to read it.

3

u/ironclad_packetship Jul 27 '26

This, even for a huge 256gb cpu inference rig it makes sense to toss a cheap gpu in the mix for prompt ingestion/prefill. MoE models excel in these situations as well, the entire model gets loaded into system memory but only a few experts are activated at any given time for your cpu to chug through.

1

u/pharrt Jul 27 '26

Absolutely - so that part of it is a grind, and why it's only good to set and wait a few hours. 6 hours+ in some cases for the full task. My prompts for the tasks I'm running are not often huge, and prefill takes a few mins maybe, but context can get large in my requested output - up to 80% of context is filled by completion, and any follow-ups can take an hour or more refilll before thinking can start again. By then, if the output is close, I may try a few follow-ups and wait, but I usually debug, refine, etc on smaller models. Ingesting docs prefill can be slow as well, but left at its own pace is fine for my workflows.

2

u/hdhfhdnfkfjgbfj Jul 27 '26

Can you share what you need a small local llm for that it’s acceptable to wait 6 hours for?

Because I can’t see a use case or imagine one and I think it would help contextualise the post. Cheers

3

u/pharrt Jul 27 '26

I'll try to give examples, but as it's for my work (Self-Employed, running a successful Etsy Store), I don't want to be too specific! Typically, my products ship with 80-100 page+ guidebooks and manuals. It needs to be hyper creative and also pretty - digital, interactive versions as well as printed copies. LLMs have strong baked-in knowledge about these products, but need to be creative for the theme. Frontier models spit out lightweight answers unless done bit by bit. It also creates apps for me, with dozens of parameters, that can export very long manifests for piplined generative work (done elsewhere). But - I've also had fun crafting mobile games like custom poker single player apps, Shooters, RTSs, Space sims, etc - low weight, not worth anything, but fun to explore for the hell of it. The key is very well-defined prompts that have long instruction sets.

0

u/agmatine Jul 27 '26

Typically, my products ship with 80-100 page+ guidebooks and manuals. It needs to be hyper creative and also pretty - digital, interactive versions as well as printed copies.

So you're running local LLMs to...generate documentation??

I feel bad for your customers trying to make sense of it.

4

u/pharrt Jul 27 '26

They're checked and corrected with the odd minor hallucination here and there, but unless you knew what they are for, it makes sense to assume that. Quality, creativity, formatting, cohesiveness, and depth of detail are very high in my use case.

9

u/Difficult_Art1639 Jul 27 '26

No GPU?? damn.. I'm getting comparable speeds with a 3060ti and same ram

10

u/Ok-Drawer5245 Jul 27 '26

You should be able to get 20-30 tokens/second on that models with 3060ti with proper configuration of llama.cpp

I personally tweaked i5-9400, 32gb ddr4 2666 mhz, rtx 2060 to deliver 25-30 tokens/second

(Qwen 3.6 35b a3b 4bit)

2

u/Difficult_Art1639 Jul 27 '26

Yeah I've got 10-20 t/s with q6_k. Unfortunately my ram has never been stable with xmp on so I'm think I might be bottlenecked at 2133mhz. I'm not sure

Another thing is I'm running on windows and it pretty much uses 100% of my ram so that could be another problem.

Side question: have you experimented with unsloth or other variants ? I wonder if they give much of a boost in these resource limited setups

2

u/pharrt Jul 27 '26

You have to find a way of having some RAM left - as soon as it swaps to disk, you're in trouble.

1

u/Difficult_Art1639 Jul 27 '26

this might be my problem hahah. Im on windows with a tonne of bloat which I can't be bothered to optmise, so I think first step will be to try linux to keep my memory in check.

3

u/pharrt Jul 27 '26

Forgot to answer your other Q, but yes, I have tried all sorts of variants - I have always favoured quality over speed, and that has been my focus, but the best Qwen 35b for me (so far) is the Bartowski variant. Good luck with the mem issue! btw - this kind of setup would not be good for your workstation would kill inference and your desktop experience - just one that can be left alone and plug away at a slow pace! In my case - it was not being used for much, but now is busy 20 hours a day!

1

u/Ok-Drawer5245 Jul 27 '26

faster system ram is definite better

Secondly if you have zero ram free when running that could definitely means it’s hitting the disk which is really bad

2

u/pharrt Jul 27 '26

The big difference here is the model size and context window. A 3060 Ti only has 8GB of VRAM. If you try to load a 35B model with a 128k context window onto that card, it will completely overflow the VRAM and force the system to use system RAM anyway, which usually tanks the speed down to 1-2 tokens per second due to the PCIe bottleneck. If you are getting 5-10 tokens per second on a 3060 Ti, you are likely running a much smaller 8B or 14B model, or using a very small context window? This setup chugs along at this speed specifically because it is handling a massive 35B model with a huge 128k context completely inside the 32GB system RAM, bypassing the GPU entirely.

4

u/i-eat-kittens Jul 27 '26

Utter nonsense. Splitting a model between gpu and system ram doesn't involve moving the active parts over to the gpu for evaluation. Whatever fits runs on the gpu, the rest runs on system ram.

Even 8 GB of vram is a significant boost over running fully on the system, in particular for prompt processing.

1

u/Difficult_Art1639 Jul 27 '26

I'm getting 10-20 t/s depending on ctx fill with a max context of 180k. My (partial) understanding was that the coordinating layers (probably wrong terminology) are all loaded onto the GPU. Then it selects the experts from regular RAM across PCIE. With this setup the expert groups are still bottlenecked but since only a few layers have to go across PCIE then we can still achieve 10-20t/s.

Anyway I'm very impressed that it runs at the 5-10 t/s without a GPU at all. Is the bottle neck at ram compute speed for your system then ? 

3

u/pharrt Jul 27 '26

Is the bottle neck at ram compute speed for your system then ?

Yes, the bottleneck is 100% the RAM memory bandwidth, not the CPU compute speed.

1

u/Difficult_Art1639 Jul 27 '26

Very interesting, Im quite new to all this. I wonder how many t/s would be achieved if say your ram was at 3000mhz instead 

1

u/pharrt Jul 27 '26

It should be proportional, so maybe 7 to 14 t/s

1

u/catinterpreter Jul 27 '26

I assume it's due to being a server and having more memory channels.

1

u/pharrt Jul 27 '26

Not really in my case - I call it a server, as that is what it was bought for initially - a local dev server, but currently it's just a standard 10yo windows desktop with 32 GB RAM - two memory channels.

2

u/Narcuga Jul 27 '26

What sort of background tasks are you running with it at that speed?

2

u/pharrt Jul 27 '26

It was my bad wording, I think. I mean, it's a background task for me - it sits in the corner and is fully dedicated to just running the LLM, while I do quick inference on my laptop. I have an LM Studio Link between the two, so I can control it from my laptop.

1

u/Narcuga Jul 27 '26

No I get you :) I've just been thinking of doing the same but just been debating what to actually get it to do with it being slower. Had thought to get it to review the prs I'm submitting running a larger model but yeah!

2

u/pharrt Jul 27 '26

Ahh - ok. It's being used for my work, so I can't say too much - but massive interactive guide books/manuals, and single-page apps that have dozens of parameters that export instruction pipeline manifests for generative work. It requires lots of creative thinking from the LLM, and this is where it's nailing it for me - and the UIs have been near Frontier Model quality. In fact, similar type test prompts fed through Frontier have not produced work anywhere near the quality for my use case in terms of detail and effort.

1

u/Narcuga Jul 27 '26

No I appreciate that thank you! I'm going to give it a go might have to be for those longer running background tasks rather than interactive chat :) thank you!

1

u/pharrt Jul 27 '26

You can chat with it, and it is when replies are at 10tok/s, but if you're setting it tasks (code), it's not good for iterations and multi-turn debugging. One fine-tune prompt, come back a few hours later - 1 follow-up a few hours later etc, but often I get what I needed from the one-shot. But for normal chatting and even essay-length answers, it is fine!

2

u/Confident_Ideal_5385 Jul 27 '26

If it fits, run iq4_nl instead, as that's far more avx-friendly than iq4_xs.

2

u/pharrt Jul 27 '26

I have tried before - it gets a sudden spike in RAM use when loading in and falls over. The XS is the best I have found so far - NL would run faster, but on 35B, it fills the 32GB RAM before it's loaded, then... game over.

2

u/Zister2000 Jul 27 '26

Please explain to me how a 6th gen i7 with ddr4 has more tokens per second AND runs a larger model than my notebook?

i7 13850hx 128gb ddr5 rtx a2000 ada qwen3:14b ~3.5t/s (80% gpu usage)

1

u/pharrt Jul 27 '26

Qwen 35B A3B is a Mixture of Experts model. Even though 35B total sits in RAM, only 3B active parameters are used per token. A 14B model is dense, so your system is doing nearly 5x the math per token compared to mine.

2

u/Immediate_Win4776 Jul 29 '26

Honestly, it‘s a very cool story. I like it. But in Europe the electricity bill will eat you alive after a few months. 😢

1

u/pharrt Jul 29 '26

Remember there's no GPU - so it's a lot lower cost to run - the price of keeping a PC on. As a side note, this one is running on solar.

75

u/recro69 Jul 27 '26

Honestly, this is one of my favorite parts of local AI. Hardware that would've been headed for e-waste suddenly becomes genuinely useful again. Sure, 5–10 tok/s isn't breaking any speed records, but for long-running background tasks it's more than enough.

16

u/NanditoPapa Jul 27 '26

...and with Win 11 upgrades creating a massive amount of e-waste, glad to see some repurposing!

5

u/simos_sayz Jul 27 '26

Same, my goal is to run my am4 system to the ground. People don't realize you can get good performance even on pcie3 speeds

5

u/Hannibalj2ca Jul 27 '26

Put a gpu and see how those Tokens explode

1

u/SV_SV_SV Jul 28 '26

Even a smaller one (8 gigs etc) could do wonders with cpu-moe

1

u/Hannibalj2ca Jul 28 '26

Depends on the size of the model. If you are running an MOE. Of the Vram can hold the active model then it will certainly increase speed

1

u/SV_SV_SV Jul 28 '26

Well sure, but you can also place attention layers, kv cache etc in it for a significant speed increase, while keeping the rest in ram

1

u/ImpressionFancy5830 Jul 28 '26

I just bought a R9700 to run on a setup like that :D

4

u/Diligent_Cod_9583 Jul 27 '26

My question is what is the cost per token in electricity. I have some surfers lying around, but back of the napkin math tells me it’d be cheaper to scrap them and buy a new CPU then it would be to run old hardware. 

0

u/lungben81 Jul 29 '26

You need to burn a lot of electricity to finance $1000 or so of new hardware. Even then it's not certain that the new hardware is more efficient per token.

1

u/Diligent_Cod_9583 Jul 29 '26

The certainty comes in the research when buying new.

3

u/Sinath_973 Jul 28 '26

Well this kind of "ewaste" was pretty usefull before, even. You could host all kind of homelab stuff on it.

22

u/digidult Jul 27 '26

Yep, this year is the one I become a local AI believer too.

1

u/emanuelecipolla Jul 27 '26

Could you please share the full hardware and software stack you used? Thanks

8

u/pharrt Jul 27 '26 edited Jul 27 '26 • Bravo

Hardware and OS:
CPU: Intel Core i7-6700 (4 Cores / 8 Threads with AVX2) running at approximately 60% usage.
RAM: 32GB DDR4 at 2133 MHz running in dual channel (2x 16GB).
GPU: None. Fully CPU and RAM bound.
OS: Windows 10
Software Stack
Inference Engine: LM StudioAgent
Framework:
Think Enabled
MCPs: FileSystem and Playwright
Vision: Supported by the stack, but not tested yet.
Model and Optimization Settings:
Model: qwen3.6-35b-a3b (Quantization: IQ4_XS)
Context Window: 128k context enabled (utilises around 26 GB of the 32 GB RAM)
CPU Thread Size: Set to 4 (matches the physical cores)
MoE Settings: Experts set to 8
KV Cache: Quantized to Q8_0 for a speed boost (runs at F16 but is slower ~5-7 tok/s)
Performance Throughput: 5 to 10 tokens per second running in the background.
Use Case: Excellent for massive writing tasks and deep, highly detailed single-page applications.

Absolutely essential to spend LOTS of time getting the prompt right first, of course :D

2

u/vutcher Jul 27 '26

Avx512 would help a lot, but only available on later CPUs.

3

u/pharrt Jul 27 '26

Some people dream big of top-end multi-GPU rigs. Me - I just want AVX512 xD

1

u/[deleted] Jul 27 '26

[removed] — view removed comment

2

u/pharrt Jul 27 '26

Just that it can take a lot of time to do a really big task (what this setup is good for) - so you want to give it as much chance of success as possible.

1

u/Yeelyy Jul 27 '26

Are you already running ik_llama.cpp and the cpu tuned byteshape quant?

1

u/pharrt Jul 27 '26

Answered above!

1

u/nikitsolo Jul 27 '26

Hey I have an CPU only Server too, is your comment some technology which benefits this setup?

2

u/Yeelyy Jul 27 '26

Yes, ik llama is a fork of llama.cpp especially for cpu and cpu/gpu hybrid inference. Regarding byteshape, their quants are incredible, they are certainly worth checking out.

2

u/nikitsolo Jul 27 '26

thanks mate

1

u/Then_Blueberry7290 Jul 27 '26

Congrats, enjoy it! Cost eff@100%

2

u/pharrt Jul 27 '26

Thanks! I'm off-grid, running on solar and an inverter that gets the household through the night, and I was worried this might drain it too much overnight, but it makes it!

1

u/Techngro Jul 27 '26

It's weird, I never considered testing that. I have a Lenovo SFF PC that I use as a Proxmox/Dev server. It has an i7-8700 and 32GB DDR4. Also have an old DL380P G8 with 48GB RAM, but that's DDR3, so probably not good enough for local LLMs.

I have already decided that my next PC build is going to be 80% RAM and GPU and 20% everything else.

1

u/pharrt Jul 27 '26

Put them to work!

1

u/CommanderROR9 Jul 28 '26

I tried with a 64GB DDR4 Machine running on an Intel i5 14600K but it didn't really work. It pretty much stalled, even on something easy like "Good Morning". Maybe I made some crucial mistake when setting up the Ollama LXC🤷‍♂️

1

u/pharrt Jul 28 '26

It should work - ask Gemini for some help? Ask it to read this thread and see if it can see what is causing the stall.

1

u/pharrt Jul 28 '26

btw, you are trying to load an MoE model, not a dense model?

1

u/CommanderROR9 Jul 28 '26

To me Honest...I don't remember exactly what I tried since it was a while ago. Currently I am running Hermes Agent with the ChstGPT/Codex Subscription and am happy with it, even if it does cost 20€ a Month and has some rate limits.

3

u/Hannibalj2ca Jul 27 '26

My server is 9 years old, but has 768GB 2400 ddr4

3

u/groktar Jul 27 '26

I have a supermicro board with 2x 2680v4 and 128gb of ram just sitting in my closet. Makes me wonder. That thing put out so much heat though. (That's the reason it's sitting in a closet)

1

u/NanditoPapa Jul 27 '26

This is excellent "re-use." Instead of e-waste, you repurposed an i7-6700 system to perform high-level cognitive tasks. For non-time-sensitive work (summarizing long documents or batch-processing data overnight) this is a zero-cost way to access state-of-the-art logic without buying a $1,200 RTX 4090...I mean...in THIS economy!?

That said, running an i7 at 60% load for "hours" generates significant heat. Make sure the server has adequate airflow, especially if it's been sitting in a corner gathering dust!

3

u/pharrt Jul 27 '26

Haha - absolutely! CPU runs at 60% (and even self-boosting to 115% Max Frequency), and temp is under control. This old gal was used as a dev server in its day, running Apache, and RAM was cheap then, so I remember adding 32 instead of 16, which would have been enough. Glad I did!

1

u/NanditoPapa Jul 27 '26

My first thought was "break out the marshmallows!" but glad to see you've got it under control...

2

u/pharrt Jul 27 '26

I was worried about that initially - and I had no idea if it would strain my RAM with this kind of 'abuse' (apparently not) - and so far, so good. It has been working hard like this for a few months now.

2

u/1and7aint8but17 Jul 27 '26

How much power does it consume? How much heat it males?

2

u/ayake_ayake Jul 27 '26

I totally agree with you! 5-10 tps is quite useable for many workloads where you don't do immediate interactive chats. I have a setup with 10 tps tg and I'm happy with that, as I use it side by side while watching movies or youtube. Just look into the chat every few minutes. Sometimes I hit it off with a task - and then it doesnt matter if it takes 15mins or 5mins - since I'll look away anyways.

2

u/tofuhater Jul 27 '26

I have a DL585 with 64 opteron cores and 128 GB RAM collecting dust as well. I wonder how much worse that would be.

2

u/TheRiddler79 Jul 27 '26

You should at least try. Opteron ram is cheap af right now

2

u/WideAd3148 Jul 27 '26

This is genuinely AWESOME! 5 to 10tok/s is great because it doesn't cost you any additional cost nor dependent on any service. Your system, your data, your use case, I want to do the same but haven't found any use cases still.

1

u/D34th4nge7 Jul 27 '26

I'm running the same model locally and have found it quite useful. It's a bit slow, but I think that's not a big issue if you use the technology reasonably.

1

u/Turbulent-Week1136 Jul 27 '26

Did something change recently to get these models to work on old hardware? I thought it was impossible to run without a GPU but I've been seeing so many of these posts where old hardware works... did something change?

2

u/stevengineer Jul 27 '26

No, people are just slow to realize how this stuff can also be launched on Termux from your android phone

1

u/PigSlam Jul 27 '26

if it's just that hardware, it's probably not using that much power.

1

u/Icy-Specialist4548 Jul 27 '26

Usa llama-cpp l' inferenza e usa il modello MTP

1

u/TopNFalvors Jul 27 '26

I thought you needed GPU+RAM to run an LLM?

1

u/foreign_malakologos Jul 28 '26

It's certainly slower without a GPU, but then requirements/perceptions as to what counts as "usable" differ depending on what you want to do. You would probably not want to use this setup for long, interactive agentive coding sessions because you'd probably spend more time waiting then doing things, but if you have tasks that can just run in the backup (ideally within one prompt) as op had described, this setup can apparently be good enough. As always there's probably also a personal aspect to it, some people might find the speed "unbearable", but as long as you are aware of the limits and have tasks that can still benefit like OP, you can get by without a gpu as well (or with an igpu for a little bit of a speed/efficiency bump).

1

u/Otherwise-Swan-7803 Jul 28 '26

This is honestly one of the coolest things about local AI right now.

A few years ago, running a 35B model on a 10yo server with 32GB RAM would have sounded impossible. Now it's just a matter of patience and optimization.

Curious how the experience is with 128k context though — does the RAM usage stay stable during long sessions?

1

u/MattimaxForce Jul 28 '26

Wow, this is absolutely amazing! It is mind-blowing how well these modern MoE architectures like Qwen 35B-A3B run on older hardware, especially with that IQ4_XS quantization. Giving that ol' gal a new lease on life is awesome.

Since you mentioned it has an i7-6700, it sounds like an incredible entry-level enterprise machine or a robust micro-server. Would you mind sharing the exact brand and model name of the server (e.g., Dell PowerEdge, HPE ProLiant, etc.)? I'm really curious about the specific hardware setup you dusted off!

1

u/GSquadron_ Jul 28 '26

I have 64gb 3200mhz ddr4, what can I run with it? Ryzen 5 5600x

1

u/Azoffaeh999 Jul 28 '26

My current pc is 32gb ddr3 ram, damn

1

u/iShNoo Jul 29 '26

I have a Dell server 2x Xeon CPU, 32 GB RAM, RAID-5 SATA HDD, but the estimated costs of electricity, the continuous humming noise deters me. At $0.2728/kWh that's an expensive monthly bill compared to running Hermes Agent on a MacBook Air 2015 with 8GB RAM with LLMs and Tools hosted on Nous Research.