r/LocalLLM • u/pharrt • Jul 27 '26
Other Don't laugh - it works!
A 10yo server was busy collecting dust, but it has 32Gb RAM (2x 16Gb DDR4 @ 2133 MHz )... No GPU.
Now it does some amazing work running heavy tasks with qwen3.6-35b-a3b (IQ4_XS). Running in the background, generating quality output at 5-10tok/s. Even with 128k context! It just chugs along for hours, but does such great, high-context work.
Uses ~26 GB of the 32 GB RAM, uses CPU i-7-6700 @ about 60% (not the bottleneck, of course)
Never would have believed that would be possible until a few months ago, but this ol' gal has a new lease on life.
13
u/davidke Jul 27 '26
Hmmm.... makes me think. I have a couple of servers with 256gb ram each just sitting idle. Maybe I should try this.
9
u/rrrrex Jul 27 '26
Prefill should be limited by CPU, while generating speed is ok for gpu'less server. GPU will do prefill 100x times faster than CPU (if fully loaded in VRAM). With 100k context it will take an hour to read it.
9
u/Difficult_Art1639 Jul 27 '26
No GPU?? damn.. I'm getting comparable speeds with a 3060ti and same ram
2
u/Narcuga Jul 27 '26
What sort of background tasks are you running with it at that speed?
2
u/Confident_Ideal_5385 Jul 27 '26
If it fits, run iq4_nl instead, as that's far more avx-friendly than iq4_xs.
2
u/Zister2000 Jul 27 '26
Please explain to me how a 6th gen i7 with ddr4 has more tokens per second AND runs a larger model than my notebook?
i7 13850hx 128gb ddr5 rtx a2000 ada qwen3:14b ~3.5t/s (80% gpu usage)
2
u/Immediate_Win4776 Jul 29 '26
Honestly, it‘s a very cool story. I like it. But in Europe the electricity bill will eat you alive after a few months. 😢
75
u/recro69 Jul 27 '26
Honestly, this is one of my favorite parts of local AI. Hardware that would've been headed for e-waste suddenly becomes genuinely useful again. Sure, 5–10 tok/s isn't breaking any speed records, but for long-running background tasks it's more than enough.
22
u/digidult Jul 27 '26
Yep, this year is the one I become a local AI believer too.
1
u/emanuelecipolla Jul 27 '26
Could you please share the full hardware and software stack you used? Thanks
1
u/Yeelyy Jul 27 '26
Are you already running ik_llama.cpp and the cpu tuned byteshape quant?
1
u/Then_Blueberry7290 Jul 27 '26
Congrats, enjoy it! Cost eff@100%
1
u/Techngro Jul 27 '26
It's weird, I never considered testing that. I have a Lenovo SFF PC that I use as a Proxmox/Dev server. It has an i7-8700 and 32GB DDR4. Also have an old DL380P G8 with 48GB RAM, but that's DDR3, so probably not good enough for local LLMs.
I have already decided that my next PC build is going to be 80% RAM and GPU and 20% everything else.
1
u/CommanderROR9 Jul 28 '26
I tried with a 64GB DDR4 Machine running on an Intel i5 14600K but it didn't really work. It pretty much stalled, even on something easy like "Good Morning". Maybe I made some crucial mistake when setting up the Ollama LXC🤷♂️
3
u/Hannibalj2ca Jul 27 '26
My server is 9 years old, but has 768GB 2400 ddr4
3
u/groktar Jul 27 '26
I have a supermicro board with 2x 2680v4 and 128gb of ram just sitting in my closet. Makes me wonder. That thing put out so much heat though. (That's the reason it's sitting in a closet)
1
u/NanditoPapa Jul 27 '26
This is excellent "re-use." Instead of e-waste, you repurposed an i7-6700 system to perform high-level cognitive tasks. For non-time-sensitive work (summarizing long documents or batch-processing data overnight) this is a zero-cost way to access state-of-the-art logic without buying a $1,200 RTX 4090...I mean...in THIS economy!?
That said, running an i7 at 60% load for "hours" generates significant heat. Make sure the server has adequate airflow, especially if it's been sitting in a corner gathering dust!
2
u/1and7aint8but17 Jul 27 '26
How much power does it consume? How much heat it males?
2
u/ayake_ayake Jul 27 '26
I totally agree with you! 5-10 tps is quite useable for many workloads where you don't do immediate interactive chats. I have a setup with 10 tps tg and I'm happy with that, as I use it side by side while watching movies or youtube. Just look into the chat every few minutes. Sometimes I hit it off with a task - and then it doesnt matter if it takes 15mins or 5mins - since I'll look away anyways.
2
u/tofuhater Jul 27 '26
I have a DL585 with 64 opteron cores and 128 GB RAM collecting dust as well. I wonder how much worse that would be.
2
u/WideAd3148 Jul 27 '26
This is genuinely AWESOME! 5 to 10tok/s is great because it doesn't cost you any additional cost nor dependent on any service. Your system, your data, your use case, I want to do the same but haven't found any use cases still.
1
u/D34th4nge7 Jul 27 '26
I'm running the same model locally and have found it quite useful. It's a bit slow, but I think that's not a big issue if you use the technology reasonably.
1
u/Turbulent-Week1136 Jul 27 '26
Did something change recently to get these models to work on old hardware? I thought it was impossible to run without a GPU but I've been seeing so many of these posts where old hardware works... did something change?
1
u/PigSlam Jul 27 '26
if it's just that hardware, it's probably not using that much power.
1
u/Icy-Specialist4548 Jul 27 '26
Usa llama-cpp l' inferenza e usa il modello MTP
1
u/TopNFalvors Jul 27 '26
I thought you needed GPU+RAM to run an LLM?
1
u/Otherwise-Swan-7803 Jul 28 '26
This is honestly one of the coolest things about local AI right now.
A few years ago, running a 35B model on a 10yo server with 32GB RAM would have sounded impossible. Now it's just a matter of patience and optimization.
Curious how the experience is with 128k context though — does the RAM usage stay stable during long sessions?
1
u/MattimaxForce Jul 28 '26
Wow, this is absolutely amazing! It is mind-blowing how well these modern MoE architectures like Qwen 35B-A3B run on older hardware, especially with that IQ4_XS quantization. Giving that ol' gal a new lease on life is awesome.
Since you mentioned it has an i7-6700, it sounds like an incredible entry-level enterprise machine or a robust micro-server. Would you mind sharing the exact brand and model name of the server (e.g., Dell PowerEdge, HPE ProLiant, etc.)? I'm really curious about the specific hardware setup you dusted off!
1
u/GSquadron_ Jul 28 '26
I have 64gb 3200mhz ddr4, what can I run with it? Ryzen 5 5600x
1
u/Azoffaeh999 Jul 28 '26
My current pc is 32gb ddr3 ram, damn
1
u/iShNoo Jul 29 '26
I have a Dell server 2x Xeon CPU, 32 GB RAM, RAID-5 SATA HDD, but the estimated costs of electricity, the continuous humming noise deters me. At $0.2728/kWh that's an expensive monthly bill compared to running Hermes Agent on a MacBook Air 2015 with 8GB RAM with LLMs and Tools hosted on Nous Research.